2024-04-25T01:03:37.613 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-25T01:03:37.613 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672309 --description fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/alternate-pool} --owner scheduled_teuthology@teuthology --verbose --name teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi -- /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672309/orig.config.yaml 2024-04-25T01:03:37.666 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T01:03:37.762 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672309 branch: squid description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/alternate-pool} email: ceph-qa@ceph.com first_in_suite: false job_id: '7672309' kernel: client: branch: testing kdb: 1 sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - bad backtrace - object missing on disk - error reading table object - error reading sessionmap - unmatched fragstat - unmatched rstat - was unreadable, recreating it now - Scrub error on inode - Metadata damage detected - MDS_FAILED - MDS_DAMAGE sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 8528 sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 sleep_before_teardown: 0 subset: 122/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi090.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHhKKIWdeKmaEvW2s6Px2CB2wu5+nf+EUnjoYA/bxEDczaUNHNtgxqJdt4l2JR7Wl3nhZNluZNuKdwPXdezSumc= smithi103.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEA6BtYIgnGNys70WvzPevHdJk4Kw8F56Dgwf+M3Il5OcQdD9/MBCQPcBUO8YG/5tH6Ao817SNhuxBOyPkv6YYo= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_recovery_pool teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-24_21:24:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-25T01:03:37.762 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa; will attempt to use it 2024-04-25T01:03:37.768 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa/tasks 2024-04-25T01:03:37.768 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-25T01:03:37.770 INFO:teuthology.task.internal:Checking packages... 2024-04-25T01:03:37.797 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd2c45d977142e024a59b68a8f1dce8f0b9840551' 2024-04-25T01:03:37.797 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-25T01:03:37.797 INFO:teuthology.packaging:ref: None 2024-04-25T01:03:37.797 INFO:teuthology.packaging:tag: None 2024-04-25T01:03:37.797 INFO:teuthology.packaging:branch: squid 2024-04-25T01:03:37.797 INFO:teuthology.packaging:sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:03:37.798 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-25T01:03:38.099 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2445.gd2c45d97 2024-04-25T01:03:38.101 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-25T01:03:38.113 INFO:teuthology.task.internal:no buildpackages task found 2024-04-25T01:03:38.113 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-25T01:03:38.128 INFO:teuthology.task.internal:Saving configuration 2024-04-25T01:03:38.151 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-25T01:03:38.165 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T01:03:38.201 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672309', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 00:57:59.959668', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHhKKIWdeKmaEvW2s6Px2CB2wu5+nf+EUnjoYA/bxEDczaUNHNtgxqJdt4l2JR7Wl3nhZNluZNuKdwPXdezSumc='} 2024-04-25T01:03:38.245 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672309', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 00:57:59.961108', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEA6BtYIgnGNys70WvzPevHdJk4Kw8F56Dgwf+M3Il5OcQdD9/MBCQPcBUO8YG/5tH6Ao817SNhuxBOyPkv6YYo='} 2024-04-25T01:03:38.245 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-25T01:03:38.259 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-04-25T01:03:38.259 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-04-25T01:03:38.260 INFO:teuthology.run_tasks:Running task console_log... 2024-04-25T01:03:38.582 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f49117f20d0>, signals=[15]) 2024-04-25T01:03:38.582 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-25T01:03:38.593 INFO:teuthology.task.internal:Opening connections... 2024-04-25T01:03:38.593 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi090.front.sepia.ceph.com 2024-04-25T01:03:38.595 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:03:38.673 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi103.front.sepia.ceph.com 2024-04-25T01:03:38.674 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:03:38.754 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-25T01:03:38.760 DEBUG:teuthology.orchestra.run.smithi090:> uname -m 2024-04-25T01:03:38.779 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64 2024-04-25T01:03:38.780 DEBUG:teuthology.orchestra.run.smithi090:> cat /etc/os-release 2024-04-25T01:03:38.839 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="CentOS Stream" 2024-04-25T01:03:38.839 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="9" 2024-04-25T01:03:38.839 INFO:teuthology.orchestra.run.smithi090.stdout:ID="centos" 2024-04-25T01:03:38.839 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE="rhel fedora" 2024-04-25T01:03:38.840 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="9" 2024-04-25T01:03:38.840 INFO:teuthology.orchestra.run.smithi090.stdout:PLATFORM_ID="platform:el9" 2024-04-25T01:03:38.840 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T01:03:38.840 INFO:teuthology.orchestra.run.smithi090.stdout:ANSI_COLOR="0;31" 2024-04-25T01:03:38.840 INFO:teuthology.orchestra.run.smithi090.stdout:LOGO="fedora-logo-icon" 2024-04-25T01:03:38.840 INFO:teuthology.orchestra.run.smithi090.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T01:03:38.840 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://centos.org/" 2024-04-25T01:03:38.840 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T01:03:38.840 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T01:03:38.840 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T01:03:38.841 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2024-04-25T01:03:38.866 DEBUG:teuthology.orchestra.run.smithi103:> uname -m 2024-04-25T01:03:38.884 INFO:teuthology.orchestra.run.smithi103.stdout:x86_64 2024-04-25T01:03:38.885 DEBUG:teuthology.orchestra.run.smithi103:> cat /etc/os-release 2024-04-25T01:03:38.943 INFO:teuthology.orchestra.run.smithi103.stdout:NAME="CentOS Stream" 2024-04-25T01:03:38.943 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION="9" 2024-04-25T01:03:38.943 INFO:teuthology.orchestra.run.smithi103.stdout:ID="centos" 2024-04-25T01:03:38.943 INFO:teuthology.orchestra.run.smithi103.stdout:ID_LIKE="rhel fedora" 2024-04-25T01:03:38.943 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_ID="9" 2024-04-25T01:03:38.943 INFO:teuthology.orchestra.run.smithi103.stdout:PLATFORM_ID="platform:el9" 2024-04-25T01:03:38.943 INFO:teuthology.orchestra.run.smithi103.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T01:03:38.943 INFO:teuthology.orchestra.run.smithi103.stdout:ANSI_COLOR="0;31" 2024-04-25T01:03:38.943 INFO:teuthology.orchestra.run.smithi103.stdout:LOGO="fedora-logo-icon" 2024-04-25T01:03:38.944 INFO:teuthology.orchestra.run.smithi103.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T01:03:38.944 INFO:teuthology.orchestra.run.smithi103.stdout:HOME_URL="https://centos.org/" 2024-04-25T01:03:38.944 INFO:teuthology.orchestra.run.smithi103.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T01:03:38.944 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T01:03:38.944 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T01:03:38.944 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server 2024-04-25T01:03:38.966 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-25T01:03:38.986 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-25T01:03:39.013 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-25T01:03:39.013 DEBUG:teuthology.orchestra.run.smithi090:> test '!' -e /home/ubuntu/cephtest 2024-04-25T01:03:39.016 DEBUG:teuthology.orchestra.run.smithi103:> test '!' -e /home/ubuntu/cephtest 2024-04-25T01:03:39.033 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-25T01:03:39.128 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-25T01:03:39.129 DEBUG:teuthology.orchestra.run.smithi090:> test -z $(ls -A /var/lib/ceph) 2024-04-25T01:03:39.131 DEBUG:teuthology.orchestra.run.smithi103:> test -z $(ls -A /var/lib/ceph) 2024-04-25T01:03:39.160 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-25T01:03:39.409 INFO:teuthology.run_tasks:Running task kernel... 2024-04-25T01:03:39.452 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'} 2024-04-25T01:03:39.452 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}, 'client.2': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}, 'client.3': {'branch': 'testing'}} 2024-04-25T01:03:39.452 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}}, timeout 300 2024-04-25T01:03:39.452 DEBUG:teuthology.orchestra.run.smithi090:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T01:03:39.453 DEBUG:teuthology.orchestra.run.smithi103:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T01:03:39.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:03:39.472 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-25T01:03:39.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:03:39.479 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-25T01:03:39.653 DEBUG:teuthology.task.kernel:sha1 for client.1 is 806ec36b21129f113ce356c007dd59694d340b37 2024-04-25T01:03:39.653 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-04-25T01:03:39.653 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-04-25T01:03:39.671 INFO:teuthology.orchestra.run.smithi103.stdout:5.14.0-437.el9.x86_64 2024-04-25T01:03:39.671 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-04-25T01:03:39.671 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-25T01:03:39.672 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum upgrade -y linux-firmware 2024-04-25T01:03:39.732 DEBUG:teuthology.task.kernel:sha1 for client.0 is 806ec36b21129f113ce356c007dd59694d340b37 2024-04-25T01:03:39.733 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-04-25T01:03:39.733 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2024-04-25T01:03:39.751 INFO:teuthology.orchestra.run.smithi090.stdout:5.14.0-437.el9.x86_64 2024-04-25T01:03:39.751 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-04-25T01:03:39.751 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-25T01:03:39.752 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum upgrade -y linux-firmware 2024-04-25T01:03:42.445 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-04-25T01:03:44.130 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - AppStream 31 MB/s | 19 MB 00:00 2024-04-25T01:03:46.452 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 1.6 MB/s | 8.0 MB 00:05 2024-04-25T01:03:49.589 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - CRB 3.2 MB/s | 6.2 MB 00:01 2024-04-25T01:03:50.095 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 7.4 MB/s | 19 MB 00:02 2024-04-25T01:03:52.198 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 16 kB 00:01 2024-04-25T01:03:54.879 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 4.9 MB/s | 6.2 MB 00:01 2024-04-25T01:03:56.259 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 5.3 MB/s | 21 MB 00:03 2024-04-25T01:03:57.579 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-04-25T01:04:01.243 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-25T01:04:03.234 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-25T01:04:03.235 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-04-25T01:04:03.235 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-25T01:04:03.329 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-25T01:04:03.329 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-25T01:04:03.485 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ 2024-04-25T01:04:03.486 DEBUG:teuthology.orchestra.run.smithi090:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-25T01:04:03.488 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi090.front.sepia.ceph.com to complete... 2024-04-25T01:04:15.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25 01:04:15 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [357367792/357367792] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-25T01:04:15.628 INFO:teuthology.orchestra.run.smithi090.stderr:FINISHED --2024-04-25 01:04:15-- 2024-04-25T01:04:15.628 INFO:teuthology.orchestra.run.smithi090.stderr:Total wall clock time: 12s 2024-04-25T01:04:15.628 INFO:teuthology.orchestra.run.smithi090.stderr:Downloaded: 1 files, 341M in 12s (28.8 MB/s) 2024-04-25T01:04:15.631 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-25T01:04:15.631 DEBUG:teuthology.orchestra.run.smithi090:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-25T01:04:17.729 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying... ######################################## 2024-04-25T01:04:17.831 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing... ######################################## 2024-04-25T01:04:17.949 INFO:teuthology.orchestra.run.smithi090.stdout:Updating / installing... 2024-04-25T01:04:19.482 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 985 kB/s | 21 MB 00:21 2024-04-25T01:04:24.527 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-25T01:04:25.914 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-04-25T01:04:26.514 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-25T01:04:26.515 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-04-25T01:04:26.516 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-25T01:04:26.622 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.1... 2024-04-25T01:04:26.623 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-25T01:04:26.773 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ 2024-04-25T01:04:26.773 DEBUG:teuthology.orchestra.run.smithi103:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-25T01:04:26.775 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi103.front.sepia.ceph.com to complete... 2024-04-25T01:04:38.932 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25 01:04:38 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [357367792/357367792] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-25T01:04:38.932 INFO:teuthology.orchestra.run.smithi103.stderr:FINISHED --2024-04-25 01:04:38-- 2024-04-25T01:04:38.932 INFO:teuthology.orchestra.run.smithi103.stderr:Total wall clock time: 12s 2024-04-25T01:04:38.932 INFO:teuthology.orchestra.run.smithi103.stderr:Downloaded: 1 files, 341M in 12s (28.8 MB/s) 2024-04-25T01:04:38.935 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.1... 2024-04-25T01:04:38.935 DEBUG:teuthology.orchestra.run.smithi103:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-25T01:04:41.078 INFO:teuthology.orchestra.run.smithi103.stdout:Verifying... ######################################## 2024-04-25T01:04:41.201 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing... ######################################## 2024-04-25T01:04:41.324 INFO:teuthology.orchestra.run.smithi103.stdout:Updating / installing... 2024-04-25T01:04:49.238 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-04-25T01:05:10.868 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi090.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-25T01:05:10.868 DEBUG:teuthology.orchestra.run.smithi090:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-25T01:05:10.964 INFO:teuthology.orchestra.run.smithi090.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-04-25T01:05:10.964 INFO:teuthology.orchestra.run.smithi090.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-04-25T01:05:10.965 INFO:teuthology.orchestra.run.smithi090.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-04-25T01:05:10.965 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-04-25T01:05:10.965 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-04-25T01:05:10.965 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-04-25T01:05:10.965 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-04-25T01:05:10.965 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-04-25T01:05:10.965 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-04-25T01:05:10.965 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-25T01:05:10.965 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-25T01:05:10.965 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-25T01:05:10.965 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-25T01:05:10.965 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-25T01:05:10.965 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-25T01:05:10.965 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-25T01:05:10.966 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-25T01:05:10.966 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-25T01:05:10.966 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-25T01:05:10.966 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-25T01:05:10.966 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-25T01:05:10.966 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-25T01:05:10.966 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-25T01:05:10.966 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-25T01:05:10.966 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-25T01:05:10.966 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-25T01:05:10.966 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-25T01:05:10.966 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-25T01:05:10.966 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-25T01:05:10.966 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-04-25T01:05:10.966 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-04-25T01:05:10.967 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-04-25T01:05:10.967 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-25T01:05:10.967 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-04-25T01:05:10.967 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-25T01:05:10.967 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-25T01:05:10.967 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-04-25T01:05:10.967 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-04-25T01:05:10.967 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-04-25T01:05:10.967 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-04-25T01:05:10.967 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-04-25T01:05:10.967 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-04-25T01:05:10.967 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-04-25T01:05:10.967 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-04-25T01:05:10.967 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-25T01:05:10.968 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-04-25T01:05:10.968 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-04-25T01:05:10.968 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-04-25T01:05:10.968 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-04-25T01:05:10.968 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-04-25T01:05:10.968 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-04-25T01:05:10.968 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-04-25T01:05:10.968 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-04-25T01:05:10.968 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-04-25T01:05:10.968 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-04-25T01:05:10.968 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-04-25T01:05:10.968 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-04-25T01:05:10.968 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-04-25T01:05:10.968 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-04-25T01:05:10.968 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-04-25T01:05:10.968 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-04-25T01:05:10.969 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-04-25T01:05:10.969 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-04-25T01:05:10.969 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-04-25T01:05:10.969 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-04-25T01:05:10.969 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-04-25T01:05:10.969 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-04-25T01:05:10.969 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-04-25T01:05:10.969 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-04-25T01:05:10.969 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-04-25T01:05:10.969 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-04-25T01:05:10.969 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-04-25T01:05:10.969 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-04-25T01:05:10.969 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-04-25T01:05:10.969 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-04-25T01:05:10.969 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-04-25T01:05:10.969 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-04-25T01:05:10.970 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-04-25T01:05:10.970 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-04-25T01:05:10.970 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-04-25T01:05:10.970 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-04-25T01:05:10.970 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-04-25T01:05:10.970 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-04-25T01:05:10.970 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-04-25T01:05:10.970 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-04-25T01:05:10.970 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-04-25T01:05:10.970 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-04-25T01:05:10.970 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-04-25T01:05:10.970 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-04-25T01:05:10.970 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-25T01:05:10.970 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-04-25T01:05:10.971 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-04-25T01:05:10.971 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-04-25T01:05:10.971 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-04-25T01:05:10.971 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-04-25T01:05:10.971 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-04-25T01:05:10.971 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-04-25T01:05:10.971 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-04-25T01:05:10.971 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-04-25T01:05:10.971 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-04-25T01:05:10.971 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-04-25T01:05:10.971 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-04-25T01:05:10.971 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-04-25T01:05:10.971 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-04-25T01:05:10.971 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-04-25T01:05:10.972 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-04-25T01:05:10.972 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-04-25T01:05:10.972 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-04-25T01:05:10.972 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-04-25T01:05:10.972 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-04-25T01:05:10.972 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-04-25T01:05:10.972 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-04-25T01:05:10.972 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-04-25T01:05:10.972 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-25T01:05:10.972 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-25T01:05:10.972 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-04-25T01:05:10.972 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-04-25T01:05:10.972 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-04-25T01:05:10.972 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-04-25T01:05:10.972 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-04-25T01:05:10.972 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-04-25T01:05:10.973 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-04-25T01:05:10.973 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-04-25T01:05:10.973 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-04-25T01:05:10.973 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-04-25T01:05:10.973 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-04-25T01:05:10.973 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-04-25T01:05:10.973 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-04-25T01:05:10.973 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-25T01:05:10.973 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-25T01:05:10.973 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-04-25T01:05:10.973 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-04-25T01:05:10.973 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-04-25T01:05:10.973 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-04-25T01:05:10.973 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-04-25T01:05:10.974 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-04-25T01:05:10.974 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-04-25T01:05:10.974 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-04-25T01:05:10.974 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-04-25T01:05:10.974 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-04-25T01:05:10.974 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-04-25T01:05:10.974 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-04-25T01:05:10.974 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-04-25T01:05:10.974 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-04-25T01:05:10.974 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-04-25T01:05:10.974 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-04-25T01:05:10.974 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-04-25T01:05:10.974 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-04-25T01:05:10.974 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-04-25T01:05:10.975 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-04-25T01:05:10.975 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-04-25T01:05:10.975 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-04-25T01:05:10.975 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-04-25T01:05:10.975 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-04-25T01:05:10.975 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-04-25T01:05:10.975 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-04-25T01:05:10.975 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-04-25T01:05:10.975 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-25T01:05:10.975 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-25T01:05:10.975 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-04-25T01:05:10.975 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-04-25T01:05:10.975 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-04-25T01:05:10.975 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-04-25T01:05:10.975 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-04-25T01:05:10.976 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-04-25T01:05:10.976 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-04-25T01:05:10.976 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-04-25T01:05:10.976 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-04-25T01:05:10.976 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-04-25T01:05:10.976 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-04-25T01:05:10.976 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-25T01:05:10.976 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-04-25T01:05:10.976 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-04-25T01:05:10.976 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-04-25T01:05:10.976 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-04-25T01:05:10.976 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-04-25T01:05:10.976 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-04-25T01:05:10.976 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-04-25T01:05:10.976 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-25T01:05:10.977 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-04-25T01:05:10.977 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-04-25T01:05:10.977 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-04-25T01:05:10.977 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-04-25T01:05:10.977 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-04-25T01:05:10.977 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-04-25T01:05:10.977 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-25T01:05:10.977 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-25T01:05:10.977 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-25T01:05:10.977 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-04-25T01:05:10.977 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-25T01:05:10.977 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-04-25T01:05:10.977 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-25T01:05:10.977 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-25T01:05:10.978 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-25T01:05:10.978 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-25T01:05:10.978 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-25T01:05:10.978 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-25T01:05:10.978 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-04-25T01:05:10.978 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-04-25T01:05:10.978 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-04-25T01:05:10.978 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-04-25T01:05:10.978 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-04-25T01:05:10.978 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-04-25T01:05:10.978 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-04-25T01:05:10.978 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-25T01:05:10.978 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-25T01:05:10.978 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-25T01:05:10.979 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-25T01:05:10.979 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-25T01:05:10.979 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-04-25T01:05:10.979 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-04-25T01:05:10.979 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-04-25T01:05:10.979 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-25T01:05:10.979 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-25T01:05:10.979 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-25T01:05:10.979 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-25T01:05:10.979 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-25T01:05:10.979 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-25T01:05:10.979 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-04-25T01:05:10.979 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-04-25T01:05:10.979 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-25T01:05:10.980 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-04-25T01:05:10.980 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-04-25T01:05:10.980 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-04-25T01:05:10.980 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-04-25T01:05:10.980 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-25T01:05:10.980 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-25T01:05:10.980 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-25T01:05:10.980 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-04-25T01:05:10.980 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-04-25T01:05:10.980 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-04-25T01:05:10.980 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-25T01:05:10.980 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-04-25T01:05:10.980 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-04-25T01:05:10.980 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-04-25T01:05:10.980 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-25T01:05:10.981 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-04-25T01:05:10.981 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-04-25T01:05:10.981 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-25T01:05:10.981 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-04-25T01:05:10.981 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-25T01:05:10.981 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-04-25T01:05:10.981 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-04-25T01:05:10.981 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-04-25T01:05:10.981 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-04-25T01:05:10.981 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-04-25T01:05:10.981 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-04-25T01:05:10.981 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-04-25T01:05:10.981 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-04-25T01:05:10.981 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-04-25T01:05:10.982 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-04-25T01:05:10.982 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-04-25T01:05:10.982 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-04-25T01:05:10.982 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-04-25T01:05:10.982 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-04-25T01:05:10.982 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-04-25T01:05:10.982 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-04-25T01:05:10.982 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-25T01:05:10.982 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-04-25T01:05:10.982 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-04-25T01:05:10.982 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-25T01:05:10.982 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-04-25T01:05:10.982 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-04-25T01:05:10.982 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-04-25T01:05:10.983 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-04-25T01:05:10.983 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-04-25T01:05:10.983 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-04-25T01:05:10.983 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-04-25T01:05:10.983 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-04-25T01:05:10.983 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-04-25T01:05:10.983 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-04-25T01:05:10.983 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-04-25T01:05:10.983 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-25T01:05:10.983 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-25T01:05:10.983 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-04-25T01:05:10.983 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-04-25T01:05:10.983 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-04-25T01:05:10.983 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-04-25T01:05:10.984 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-04-25T01:05:10.984 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-04-25T01:05:10.984 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-04-25T01:05:10.984 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-04-25T01:05:10.984 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-25T01:05:10.984 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-25T01:05:10.984 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-04-25T01:05:10.984 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-04-25T01:05:10.984 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-04-25T01:05:10.984 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-04-25T01:05:10.984 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-04-25T01:05:10.984 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-04-25T01:05:10.984 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-04-25T01:05:10.984 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-04-25T01:05:10.984 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-04-25T01:05:10.985 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-25T01:05:10.985 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-04-25T01:05:10.985 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-25T01:05:10.985 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-25T01:05:10.985 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-25T01:05:10.985 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-25T01:05:10.985 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-25T01:05:10.985 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-25T01:05:10.985 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-25T01:05:10.985 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-25T01:05:10.985 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-25T01:05:10.985 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-04-25T01:05:10.985 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-04-25T01:05:10.985 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-04-25T01:05:10.985 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-25T01:05:10.986 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-04-25T01:05:10.986 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-04-25T01:05:10.986 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-04-25T01:05:10.986 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-04-25T01:05:10.986 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-04-25T01:05:10.986 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-25T01:05:10.986 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-04-25T01:05:10.986 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-04-25T01:05:10.986 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-04-25T01:05:10.986 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-04-25T01:05:10.986 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-04-25T01:05:10.986 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-04-25T01:05:10.986 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-04-25T01:05:10.986 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-04-25T01:05:10.987 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-04-25T01:05:10.987 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-04-25T01:05:10.987 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-04-25T01:05:10.987 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-04-25T01:05:10.987 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-04-25T01:05:10.987 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-04-25T01:05:10.987 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-04-25T01:05:10.987 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-04-25T01:05:10.987 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-04-25T01:05:10.987 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-04-25T01:05:10.987 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-04-25T01:05:10.987 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-25T01:05:10.987 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-04-25T01:05:10.987 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-04-25T01:05:10.988 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-04-25T01:05:10.988 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-04-25T01:05:10.988 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-04-25T01:05:10.988 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-04-25T01:05:10.988 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-04-25T01:05:10.988 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-04-25T01:05:10.988 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-04-25T01:05:10.988 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-04-25T01:05:10.988 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-04-25T01:05:10.988 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-04-25T01:05:10.988 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-04-25T01:05:10.988 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-04-25T01:05:10.988 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-04-25T01:05:10.988 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-04-25T01:05:10.988 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-04-25T01:05:10.989 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-04-25T01:05:10.989 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-04-25T01:05:10.989 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-25T01:05:10.989 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-04-25T01:05:10.989 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-04-25T01:05:10.989 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-25T01:05:10.989 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-04-25T01:05:10.989 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-04-25T01:05:10.989 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-04-25T01:05:10.989 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-04-25T01:05:10.989 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-04-25T01:05:10.989 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-04-25T01:05:10.989 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-04-25T01:05:10.989 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-04-25T01:05:10.990 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-04-25T01:05:10.990 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-04-25T01:05:10.990 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-04-25T01:05:10.990 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-04-25T01:05:10.990 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-04-25T01:05:10.990 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-04-25T01:05:10.990 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-25T01:05:10.990 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-04-25T01:05:10.990 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-04-25T01:05:10.990 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-04-25T01:05:10.990 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-04-25T01:05:10.990 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-04-25T01:05:10.990 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-04-25T01:05:10.990 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-04-25T01:05:10.991 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-04-25T01:05:10.991 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-04-25T01:05:10.991 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-04-25T01:05:10.991 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-04-25T01:05:10.991 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-04-25T01:05:10.991 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-04-25T01:05:10.991 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-04-25T01:05:10.991 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-04-25T01:05:10.991 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-04-25T01:05:10.991 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-04-25T01:05:10.991 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-04-25T01:05:10.991 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-04-25T01:05:10.991 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-04-25T01:05:10.991 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-04-25T01:05:10.991 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-04-25T01:05:10.992 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-04-25T01:05:10.992 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-04-25T01:05:10.992 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-04-25T01:05:10.992 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-04-25T01:05:10.992 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-04-25T01:05:10.992 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-04-25T01:05:10.992 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-04-25T01:05:10.992 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-04-25T01:05:10.992 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-04-25T01:05:10.992 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-04-25T01:05:10.992 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-04-25T01:05:10.992 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-04-25T01:05:10.992 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-04-25T01:05:10.992 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-04-25T01:05:10.993 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-04-25T01:05:10.993 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-04-25T01:05:10.993 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-04-25T01:05:10.993 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-04-25T01:05:10.993 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-04-25T01:05:10.993 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-04-25T01:05:10.993 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-04-25T01:05:10.993 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-04-25T01:05:10.993 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-04-25T01:05:10.993 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-04-25T01:05:10.993 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-04-25T01:05:10.993 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-04-25T01:05:10.993 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-25T01:05:10.993 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-25T01:05:10.993 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-25T01:05:10.994 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-25T01:05:10.994 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-25T01:05:10.994 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-25T01:05:10.994 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-25T01:05:10.994 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-25T01:05:10.994 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-25T01:05:10.994 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-25T01:05:10.994 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-25T01:05:10.994 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-25T01:05:10.994 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-04-25T01:05:10.994 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-04-25T01:05:10.994 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-04-25T01:05:10.994 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-25T01:05:10.994 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-04-25T01:05:10.995 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-04-25T01:05:10.995 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-04-25T01:05:10.995 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-04-25T01:05:10.995 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-04-25T01:05:10.995 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-04-25T01:05:10.995 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-04-25T01:05:10.995 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-04-25T01:05:10.995 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-04-25T01:05:10.995 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-04-25T01:05:10.995 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-04-25T01:05:10.995 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-04-25T01:05:10.995 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-04-25T01:05:10.995 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-04-25T01:05:10.995 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-04-25T01:05:10.996 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-04-25T01:05:10.996 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-04-25T01:05:10.996 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-04-25T01:05:10.996 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-04-25T01:05:10.996 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-04-25T01:05:10.996 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-04-25T01:05:10.996 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-04-25T01:05:10.996 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-04-25T01:05:10.996 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-04-25T01:05:10.996 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-04-25T01:05:10.996 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-04-25T01:05:10.996 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-04-25T01:05:10.996 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-25T01:05:10.996 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-25T01:05:10.996 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-04-25T01:05:10.997 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-25T01:05:10.997 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-25T01:05:10.997 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-25T01:05:10.997 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-25T01:05:10.997 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-25T01:05:10.997 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-25T01:05:10.997 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-25T01:05:10.997 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-25T01:05:10.997 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-25T01:05:10.997 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-25T01:05:10.997 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-25T01:05:10.997 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-25T01:05:10.997 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-25T01:05:10.998 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-25T01:05:10.998 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-25T01:05:10.998 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-25T01:05:10.998 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-25T01:05:10.998 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-25T01:05:10.998 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-25T01:05:10.998 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-25T01:05:10.998 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-25T01:05:10.998 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-25T01:05:10.998 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-25T01:05:10.998 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-04-25T01:05:10.998 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-04-25T01:05:10.998 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-25T01:05:10.998 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-04-25T01:05:10.999 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-04-25T01:05:10.999 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-04-25T01:05:10.999 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-04-25T01:05:10.999 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-25T01:05:10.999 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-04-25T01:05:10.999 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-25T01:05:10.999 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-25T01:05:10.999 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-25T01:05:10.999 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-25T01:05:10.999 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-25T01:05:10.999 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-04-25T01:05:10.999 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-04-25T01:05:10.999 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-25T01:05:10.999 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-04-25T01:05:10.999 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-25T01:05:11.000 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-04-25T01:05:11.000 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-25T01:05:11.000 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-04-25T01:05:11.000 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-25T01:05:11.000 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-04-25T01:05:11.000 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-04-25T01:05:11.000 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-25T01:05:11.000 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-04-25T01:05:11.000 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-25T01:05:11.000 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-04-25T01:05:11.000 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-25T01:05:11.000 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-04-25T01:05:11.000 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-25T01:05:11.000 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-04-25T01:05:11.001 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-25T01:05:11.001 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-04-25T01:05:11.001 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-04-25T01:05:11.001 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-25T01:05:11.001 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-25T01:05:11.001 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-04-25T01:05:11.001 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-04-25T01:05:11.001 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-04-25T01:05:11.001 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-25T01:05:11.001 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-04-25T01:05:11.001 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-04-25T01:05:11.001 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-04-25T01:05:11.001 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-04-25T01:05:11.001 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-25T01:05:11.002 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-04-25T01:05:11.002 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-25T01:05:11.002 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-25T01:05:11.002 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-25T01:05:11.002 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-04-25T01:05:11.002 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-25T01:05:11.002 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-04-25T01:05:11.002 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-04-25T01:05:11.002 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-04-25T01:05:11.002 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-25T01:05:11.002 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-04-25T01:05:11.002 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-04-25T01:05:11.002 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-04-25T01:05:11.002 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-04-25T01:05:11.003 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-04-25T01:05:11.003 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-04-25T01:05:11.003 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-04-25T01:05:11.003 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-04-25T01:05:11.003 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-25T01:05:11.003 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-25T01:05:11.003 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-25T01:05:11.003 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-25T01:05:11.003 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-25T01:05:11.003 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-25T01:05:11.003 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-25T01:05:11.003 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-25T01:05:11.003 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-25T01:05:11.003 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-04-25T01:05:11.003 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-04-25T01:05:11.004 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-04-25T01:05:11.004 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-04-25T01:05:11.004 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-04-25T01:05:11.004 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-04-25T01:05:11.004 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-04-25T01:05:11.004 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-04-25T01:05:11.004 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-04-25T01:05:11.004 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-04-25T01:05:11.004 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-04-25T01:05:11.004 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-04-25T01:05:11.004 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-04-25T01:05:11.004 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-04-25T01:05:11.004 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-04-25T01:05:11.004 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-04-25T01:05:11.005 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-04-25T01:05:11.005 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-04-25T01:05:11.005 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-04-25T01:05:11.005 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-25T01:05:11.005 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-04-25T01:05:11.005 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-04-25T01:05:11.005 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-04-25T01:05:11.005 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-04-25T01:05:11.005 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-04-25T01:05:11.005 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-04-25T01:05:11.005 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-04-25T01:05:11.005 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-04-25T01:05:11.005 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-04-25T01:05:11.006 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-04-25T01:05:11.006 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-04-25T01:05:11.006 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-04-25T01:05:11.006 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-04-25T01:05:11.006 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-04-25T01:05:11.006 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-04-25T01:05:11.006 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-04-25T01:05:11.006 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-04-25T01:05:11.006 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-04-25T01:05:11.006 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-04-25T01:05:11.006 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-04-25T01:05:11.006 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-04-25T01:05:11.006 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-04-25T01:05:11.006 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-04-25T01:05:11.007 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-25T01:05:11.007 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-04-25T01:05:11.007 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-04-25T01:05:11.007 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-04-25T01:05:11.007 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-04-25T01:05:11.007 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-04-25T01:05:11.007 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-04-25T01:05:11.007 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-04-25T01:05:11.007 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-25T01:05:11.007 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-04-25T01:05:11.007 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-04-25T01:05:11.007 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-04-25T01:05:11.007 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-04-25T01:05:11.007 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-04-25T01:05:11.007 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-04-25T01:05:11.008 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-25T01:05:11.008 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-25T01:05:11.008 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-04-25T01:05:11.008 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-04-25T01:05:11.008 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-04-25T01:05:11.008 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-04-25T01:05:11.008 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-04-25T01:05:11.008 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-25T01:05:11.008 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-04-25T01:05:11.008 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-25T01:05:11.008 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-25T01:05:11.008 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-25T01:05:11.008 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-04-25T01:05:11.008 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-04-25T01:05:11.009 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-04-25T01:05:11.009 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-04-25T01:05:11.009 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-04-25T01:05:11.009 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-25T01:05:11.009 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-04-25T01:05:11.009 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-04-25T01:05:11.009 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-04-25T01:05:11.009 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-04-25T01:05:11.009 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-04-25T01:05:11.009 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-25T01:05:11.009 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-25T01:05:11.009 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-25T01:05:11.009 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-04-25T01:05:11.009 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-25T01:05:11.009 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-04-25T01:05:11.010 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-04-25T01:05:11.010 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-04-25T01:05:11.010 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-04-25T01:05:11.010 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-25T01:05:11.010 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-25T01:05:11.010 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-25T01:05:11.010 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-25T01:05:11.010 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-04-25T01:05:11.010 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-25T01:05:11.010 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-25T01:05:11.010 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-25T01:05:11.010 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-25T01:05:11.010 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-04-25T01:05:11.011 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-25T01:05:11.011 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-04-25T01:05:11.011 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-25T01:05:11.011 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-04-25T01:05:11.011 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-25T01:05:11.011 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-04-25T01:05:11.011 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-25T01:05:11.011 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-04-25T01:05:11.011 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-25T01:05:11.011 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-25T01:05:11.011 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-25T01:05:11.011 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-04-25T01:05:11.011 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-04-25T01:05:11.011 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-25T01:05:11.012 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-25T01:05:11.012 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-25T01:05:11.012 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-25T01:05:11.012 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-25T01:05:11.012 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-04-25T01:05:11.012 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-25T01:05:11.012 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-25T01:05:11.012 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-04-25T01:05:11.012 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-25T01:05:11.012 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-25T01:05:11.012 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-25T01:05:11.012 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-25T01:05:11.012 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-25T01:05:11.012 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-04-25T01:05:11.012 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-25T01:05:11.013 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-25T01:05:11.013 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-04-25T01:05:11.013 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-04-25T01:05:11.013 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-25T01:05:11.013 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-04-25T01:05:11.013 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-25T01:05:11.013 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-25T01:05:11.013 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-25T01:05:11.013 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-25T01:05:11.013 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-25T01:05:11.013 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-25T01:05:11.013 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-25T01:05:11.013 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-25T01:05:11.014 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-25T01:05:11.014 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-25T01:05:11.014 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-04-25T01:05:11.014 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-04-25T01:05:11.014 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-25T01:05:11.014 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-04-25T01:05:11.014 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-04-25T01:05:11.014 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-25T01:05:11.014 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-25T01:05:11.014 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-25T01:05:11.014 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-25T01:05:11.014 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-25T01:05:11.014 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-25T01:05:11.014 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-04-25T01:05:11.015 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-25T01:05:11.015 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-25T01:05:11.015 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-04-25T01:05:11.015 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-04-25T01:05:11.015 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-04-25T01:05:11.015 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-25T01:05:11.015 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-04-25T01:05:11.015 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-04-25T01:05:11.015 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-04-25T01:05:11.015 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-25T01:05:11.015 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-04-25T01:05:11.015 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-25T01:05:11.015 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-04-25T01:05:11.015 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-25T01:05:11.016 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-04-25T01:05:11.016 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-25T01:05:11.016 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-04-25T01:05:11.016 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-25T01:05:11.016 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-04-25T01:05:11.016 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-25T01:05:11.016 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-04-25T01:05:11.016 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-25T01:05:11.016 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-04-25T01:05:11.016 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-25T01:05:11.016 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-25T01:05:11.016 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-25T01:05:11.016 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-25T01:05:11.016 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-25T01:05:11.017 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-04-25T01:05:11.017 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-04-25T01:05:11.017 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-25T01:05:11.017 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-25T01:05:11.017 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-25T01:05:11.017 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-04-25T01:05:11.017 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-25T01:05:11.017 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-25T01:05:11.017 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-25T01:05:11.017 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-25T01:05:11.017 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-25T01:05:11.017 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-25T01:05:11.017 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-04-25T01:05:11.017 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-25T01:05:11.018 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-04-25T01:05:11.018 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-25T01:05:11.018 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-25T01:05:11.018 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-04-25T01:05:11.018 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-25T01:05:11.018 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-25T01:05:11.018 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-04-25T01:05:11.018 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-25T01:05:11.018 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-04-25T01:05:11.018 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-25T01:05:11.018 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-04-25T01:05:11.018 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-25T01:05:11.018 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-25T01:05:11.018 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-04-25T01:05:11.019 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-25T01:05:11.019 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-25T01:05:11.019 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-25T01:05:11.019 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-25T01:05:11.019 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-25T01:05:11.019 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-04-25T01:05:11.019 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-25T01:05:11.019 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-04-25T01:05:11.019 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-25T01:05:11.019 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-25T01:05:11.019 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-25T01:05:11.019 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-25T01:05:11.019 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-04-25T01:05:11.019 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-25T01:05:11.020 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-04-25T01:05:11.020 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-25T01:05:11.020 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-04-25T01:05:11.020 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-25T01:05:11.020 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-04-25T01:05:11.020 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-25T01:05:11.020 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-25T01:05:11.020 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-04-25T01:05:11.020 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-25T01:05:11.020 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-25T01:05:11.020 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-25T01:05:11.020 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-25T01:05:11.020 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-04-25T01:05:11.020 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-25T01:05:11.021 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-25T01:05:11.021 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-25T01:05:11.021 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-04-25T01:05:11.021 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-25T01:05:11.021 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-25T01:05:11.021 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-25T01:05:11.021 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-25T01:05:11.021 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-04-25T01:05:11.021 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-25T01:05:11.021 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-04-25T01:05:11.021 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-25T01:05:11.021 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-04-25T01:05:11.021 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-25T01:05:11.021 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-25T01:05:11.022 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-04-25T01:05:11.022 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-25T01:05:11.022 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-25T01:05:11.022 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-04-25T01:05:11.022 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-25T01:05:11.022 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-04-25T01:05:11.022 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-04-25T01:05:11.022 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-25T01:05:11.022 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-04-25T01:05:11.022 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-25T01:05:11.022 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-04-25T01:05:11.022 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-04-25T01:05:11.022 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-25T01:05:11.022 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-04-25T01:05:11.022 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-04-25T01:05:11.023 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-04-25T01:05:11.023 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-04-25T01:05:11.023 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-25T01:05:11.023 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-25T01:05:11.023 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-25T01:05:11.023 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-04-25T01:05:11.023 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-04-25T01:05:11.023 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-04-25T01:05:11.023 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-04-25T01:05:11.023 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-04-25T01:05:11.023 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-25T01:05:11.023 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-25T01:05:11.023 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-04-25T01:05:11.023 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-04-25T01:05:11.024 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-25T01:05:11.024 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-25T01:05:11.024 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-25T01:05:11.024 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-04-25T01:05:11.024 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-04-25T01:05:11.024 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-04-25T01:05:11.024 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-04-25T01:05:11.024 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-04-25T01:05:11.024 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-04-25T01:05:11.024 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-04-25T01:05:11.024 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-04-25T01:05:11.024 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-04-25T01:05:11.024 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-04-25T01:05:11.024 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-04-25T01:05:11.025 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-04-25T01:05:11.025 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-04-25T01:05:11.025 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-04-25T01:05:11.025 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-04-25T01:05:11.025 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-04-25T01:05:11.025 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-04-25T01:05:11.025 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-04-25T01:05:11.025 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-25T01:05:11.025 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-04-25T01:05:11.025 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-25T01:05:11.025 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-25T01:05:11.025 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-25T01:05:11.025 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-25T01:05:11.025 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-04-25T01:05:11.026 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-04-25T01:05:11.026 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-04-25T01:05:11.026 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-04-25T01:05:11.026 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-04-25T01:05:11.026 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-04-25T01:05:11.026 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-04-25T01:05:11.026 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-04-25T01:05:11.026 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-04-25T01:05:11.026 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-04-25T01:05:11.026 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-25T01:05:11.026 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-25T01:05:11.026 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-25T01:05:11.026 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-04-25T01:05:11.026 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-25T01:05:11.027 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-04-25T01:05:11.027 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-04-25T01:05:11.027 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-04-25T01:05:11.027 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-04-25T01:05:11.027 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-25T01:05:11.027 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-04-25T01:05:11.027 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-04-25T01:05:11.027 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-04-25T01:05:11.027 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-25T01:05:11.027 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-04-25T01:05:11.027 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-04-25T01:05:11.027 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-04-25T01:05:11.027 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-04-25T01:05:11.027 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-04-25T01:05:11.028 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-04-25T01:05:11.028 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-04-25T01:05:11.028 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-04-25T01:05:11.028 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-04-25T01:05:11.028 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-04-25T01:05:11.028 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-04-25T01:05:11.028 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-04-25T01:05:11.028 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-04-25T01:05:11.028 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-04-25T01:05:11.028 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-25T01:05:11.028 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-04-25T01:05:11.028 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-25T01:05:11.028 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-04-25T01:05:11.028 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-04-25T01:05:11.029 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-04-25T01:05:11.029 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-04-25T01:05:11.029 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-25T01:05:11.029 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-04-25T01:05:11.029 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-04-25T01:05:11.029 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-25T01:05:11.029 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-04-25T01:05:11.029 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-04-25T01:05:11.029 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-25T01:05:11.029 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-04-25T01:05:11.029 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-25T01:05:11.029 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-25T01:05:11.029 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-25T01:05:11.029 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-25T01:05:11.029 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-25T01:05:11.030 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-25T01:05:11.030 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-25T01:05:11.030 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-25T01:05:11.030 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-25T01:05:11.030 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-04-25T01:05:11.030 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-04-25T01:05:11.030 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-25T01:05:11.030 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-04-25T01:05:11.030 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-25T01:05:11.030 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-25T01:05:11.030 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-25T01:05:11.030 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-25T01:05:11.030 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-25T01:05:11.030 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-25T01:05:11.031 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-25T01:05:11.031 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-25T01:05:11.031 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-25T01:05:11.031 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-04-25T01:05:11.031 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-04-25T01:05:11.031 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-25T01:05:11.031 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-25T01:05:11.031 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-04-25T01:05:11.031 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-04-25T01:05:11.031 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-25T01:05:11.031 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-25T01:05:11.031 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-04-25T01:05:11.031 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-04-25T01:05:11.031 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-04-25T01:05:11.031 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-04-25T01:05:11.032 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-04-25T01:05:11.032 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-04-25T01:05:11.032 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-04-25T01:05:11.032 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-04-25T01:05:11.032 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-25T01:05:11.032 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-04-25T01:05:11.032 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-04-25T01:05:11.032 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-25T01:05:11.032 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-25T01:05:11.032 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-25T01:05:11.032 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-25T01:05:11.032 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-25T01:05:11.032 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-25T01:05:11.032 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-25T01:05:11.033 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-25T01:05:11.033 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-25T01:05:11.033 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-25T01:05:11.033 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-04-25T01:05:11.033 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-25T01:05:11.033 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-25T01:05:11.033 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-25T01:05:11.033 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-25T01:05:11.033 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-25T01:05:11.033 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-25T01:05:11.033 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-04-25T01:05:11.033 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-25T01:05:11.033 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-25T01:05:11.033 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-25T01:05:11.033 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-25T01:05:11.034 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-25T01:05:11.034 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-25T01:05:11.034 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-25T01:05:11.034 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-25T01:05:11.034 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-25T01:05:11.034 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-25T01:05:11.034 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-25T01:05:11.034 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-25T01:05:11.034 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-04-25T01:05:11.034 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-04-25T01:05:11.034 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-04-25T01:05:11.034 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-04-25T01:05:11.034 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-04-25T01:05:11.035 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-04-25T01:05:11.035 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-04-25T01:05:11.035 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-04-25T01:05:11.035 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-25T01:05:11.035 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-04-25T01:05:11.035 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-04-25T01:05:11.035 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-25T01:05:11.035 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-25T01:05:11.035 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-04-25T01:05:11.035 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-04-25T01:05:11.035 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-25T01:05:11.035 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-04-25T01:05:11.035 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-04-25T01:05:11.035 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-25T01:05:11.036 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-04-25T01:05:11.036 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-04-25T01:05:11.036 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-04-25T01:05:11.036 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-25T01:05:11.036 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-04-25T01:05:11.036 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-25T01:05:11.036 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-04-25T01:05:11.036 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-04-25T01:05:11.036 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-25T01:05:11.036 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-04-25T01:05:11.036 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-25T01:05:11.036 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-25T01:05:11.036 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-25T01:05:11.037 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-25T01:05:11.037 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-25T01:05:11.037 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-04-25T01:05:11.037 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-04-25T01:05:11.037 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-25T01:05:11.037 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-25T01:05:11.037 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-25T01:05:11.037 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-25T01:05:11.037 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-04-25T01:05:11.037 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-04-25T01:05:11.037 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-25T01:05:11.037 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-04-25T01:05:11.037 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-04-25T01:05:11.037 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-25T01:05:11.038 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-25T01:05:11.038 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-04-25T01:05:11.038 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-04-25T01:05:11.038 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-04-25T01:05:11.038 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-04-25T01:05:11.038 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-04-25T01:05:11.038 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-04-25T01:05:11.038 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-04-25T01:05:11.038 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-04-25T01:05:11.038 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-04-25T01:05:11.038 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-04-25T01:05:11.038 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-25T01:05:11.038 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-25T01:05:11.038 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-04-25T01:05:11.039 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-04-25T01:05:11.039 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-04-25T01:05:11.039 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-25T01:05:11.039 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-04-25T01:05:11.039 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-04-25T01:05:11.039 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-04-25T01:05:11.039 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-25T01:05:11.039 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-04-25T01:05:11.039 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-04-25T01:05:11.039 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-25T01:05:11.039 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-25T01:05:11.039 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-25T01:05:11.039 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-04-25T01:05:11.039 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-25T01:05:11.039 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-04-25T01:05:11.040 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-25T01:05:11.040 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-04-25T01:05:11.040 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-04-25T01:05:11.040 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-25T01:05:11.040 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-04-25T01:05:11.040 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-04-25T01:05:11.040 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-04-25T01:05:11.040 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-25T01:05:11.040 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-25T01:05:11.040 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-04-25T01:05:11.040 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-25T01:05:11.040 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-04-25T01:05:11.040 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-04-25T01:05:11.040 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-25T01:05:11.040 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-25T01:05:11.041 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-25T01:05:11.041 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-25T01:05:11.041 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-25T01:05:11.041 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-04-25T01:05:11.041 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-04-25T01:05:11.041 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-04-25T01:05:11.041 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-04-25T01:05:11.041 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-25T01:05:11.041 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-04-25T01:05:11.041 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-25T01:05:11.041 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-04-25T01:05:11.041 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-04-25T01:05:11.041 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-04-25T01:05:11.041 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-04-25T01:05:11.041 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-04-25T01:05:11.042 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-25T01:05:11.042 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-04-25T01:05:11.042 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-25T01:05:11.042 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-04-25T01:05:11.042 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-04-25T01:05:11.042 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-04-25T01:05:11.042 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-04-25T01:05:11.042 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-04-25T01:05:11.042 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-04-25T01:05:11.042 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-25T01:05:11.042 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-04-25T01:05:11.042 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-04-25T01:05:11.042 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-04-25T01:05:11.042 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-25T01:05:11.043 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-25T01:05:11.043 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-25T01:05:11.043 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-25T01:05:11.043 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-25T01:05:11.043 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-25T01:05:11.043 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-25T01:05:11.045 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-25T01:05:11.045 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-25T01:05:11.046 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-25T01:05:11.046 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-25T01:05:11.046 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-25T01:05:11.046 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-25T01:05:11.046 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-25T01:05:11.046 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-25T01:05:11.046 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-25T01:05:11.046 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-25T01:05:11.046 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-25T01:05:11.046 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-04-25T01:05:11.046 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-04-25T01:05:11.046 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-25T01:05:11.046 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-04-25T01:05:11.047 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-04-25T01:05:11.047 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-04-25T01:05:11.047 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-04-25T01:05:11.047 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-04-25T01:05:11.047 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-25T01:05:11.047 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-04-25T01:05:11.047 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-04-25T01:05:11.047 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-25T01:05:11.047 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-04-25T01:05:11.047 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-04-25T01:05:11.047 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-25T01:05:11.047 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-04-25T01:05:11.047 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-25T01:05:11.047 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-04-25T01:05:11.047 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-04-25T01:05:11.048 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-04-25T01:05:11.048 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-04-25T01:05:11.048 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-04-25T01:05:11.048 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-04-25T01:05:11.048 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-04-25T01:05:11.048 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-04-25T01:05:11.048 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-04-25T01:05:11.048 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-25T01:05:11.048 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-04-25T01:05:11.048 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-04-25T01:05:11.048 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-04-25T01:05:11.048 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-25T01:05:11.048 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-04-25T01:05:11.048 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-04-25T01:05:11.048 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-04-25T01:05:11.049 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-04-25T01:05:11.049 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-04-25T01:05:11.049 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-04-25T01:05:11.049 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-04-25T01:05:11.049 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-04-25T01:05:11.049 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-04-25T01:05:11.049 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-04-25T01:05:11.049 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-04-25T01:05:11.049 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-25T01:05:11.049 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-04-25T01:05:11.049 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-04-25T01:05:11.049 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-04-25T01:05:11.049 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-25T01:05:11.049 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-25T01:05:11.049 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-04-25T01:05:11.050 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-04-25T01:05:11.050 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-25T01:05:11.050 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-04-25T01:05:11.050 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-04-25T01:05:11.050 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-04-25T01:05:11.050 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-04-25T01:05:11.050 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-04-25T01:05:11.050 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-04-25T01:05:11.050 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-04-25T01:05:11.050 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-25T01:05:11.050 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-04-25T01:05:11.050 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-04-25T01:05:11.050 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-04-25T01:05:11.050 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-25T01:05:11.051 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-25T01:05:11.051 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-04-25T01:05:11.051 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-25T01:05:11.051 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-25T01:05:11.051 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-25T01:05:11.051 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-04-25T01:05:11.051 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-04-25T01:05:11.051 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-04-25T01:05:11.051 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-04-25T01:05:11.051 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-25T01:05:11.051 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-04-25T01:05:11.051 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-25T01:05:11.051 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-25T01:05:11.051 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-25T01:05:11.051 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-04-25T01:05:11.052 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-04-25T01:05:11.052 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-04-25T01:05:11.052 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-04-25T01:05:11.052 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-04-25T01:05:11.052 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-04-25T01:05:11.052 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-04-25T01:05:11.052 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-04-25T01:05:11.052 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-04-25T01:05:11.052 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-04-25T01:05:11.052 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-04-25T01:05:11.052 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-04-25T01:05:11.052 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-04-25T01:05:11.052 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-04-25T01:05:11.052 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-25T01:05:11.053 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-04-25T01:05:11.053 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-04-25T01:05:11.053 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-25T01:05:11.053 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-25T01:05:11.053 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-04-25T01:05:11.053 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-25T01:05:11.053 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-04-25T01:05:11.053 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-04-25T01:05:11.053 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-25T01:05:11.053 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-04-25T01:05:11.053 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-04-25T01:05:11.053 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-25T01:05:11.053 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-25T01:05:11.053 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-25T01:05:11.053 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-25T01:05:11.054 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-25T01:05:11.054 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-25T01:05:11.054 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-25T01:05:11.054 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-04-25T01:05:11.054 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-25T01:05:11.054 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-25T01:05:11.054 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-25T01:05:11.054 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-25T01:05:11.054 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-25T01:05:11.054 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-04-25T01:05:11.054 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-04-25T01:05:11.054 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-25T01:05:11.054 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-25T01:05:11.055 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-25T01:05:11.055 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-04-25T01:05:11.055 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-04-25T01:05:11.055 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-25T01:05:11.055 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-25T01:05:11.055 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-04-25T01:05:11.055 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-25T01:05:11.055 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-04-25T01:05:11.055 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-04-25T01:05:11.055 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-25T01:05:11.055 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-04-25T01:05:11.055 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-04-25T01:05:11.055 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-04-25T01:05:11.055 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-04-25T01:05:11.056 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-04-25T01:05:11.056 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-04-25T01:05:11.056 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-25T01:05:11.056 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-04-25T01:05:11.056 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-04-25T01:05:11.056 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-04-25T01:05:11.056 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-04-25T01:05:11.056 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-04-25T01:05:11.056 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-25T01:05:11.056 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-25T01:05:11.056 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-04-25T01:05:11.056 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-04-25T01:05:11.056 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-04-25T01:05:11.056 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-04-25T01:05:11.056 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-04-25T01:05:11.057 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-04-25T01:05:11.057 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-25T01:05:11.057 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-25T01:05:11.057 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-25T01:05:11.057 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-25T01:05:11.057 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-25T01:05:11.057 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-25T01:05:11.057 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-25T01:05:11.057 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-25T01:05:11.057 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-25T01:05:11.057 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-25T01:05:11.057 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-25T01:05:11.057 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-04-25T01:05:11.057 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-04-25T01:05:11.058 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-25T01:05:11.058 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-25T01:05:11.058 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-25T01:05:11.058 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-04-25T01:05:11.058 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-04-25T01:05:11.058 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-25T01:05:11.058 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-25T01:05:11.058 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-04-25T01:05:11.058 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-04-25T01:05:11.058 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-25T01:05:11.058 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-25T01:05:11.058 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-04-25T01:05:11.058 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-04-25T01:05:11.058 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-04-25T01:05:11.059 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-25T01:05:11.059 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-25T01:05:11.059 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-25T01:05:11.059 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-25T01:05:11.059 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-04-25T01:05:11.059 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-25T01:05:11.059 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-04-25T01:05:11.059 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-25T01:05:11.059 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-25T01:05:11.059 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-25T01:05:11.059 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-04-25T01:05:11.059 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-04-25T01:05:11.059 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-04-25T01:05:11.059 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-25T01:05:11.060 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-04-25T01:05:11.060 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-04-25T01:05:11.060 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-04-25T01:05:11.060 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-04-25T01:05:11.060 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-25T01:05:11.060 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-04-25T01:05:11.060 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-04-25T01:05:11.060 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-25T01:05:11.060 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-04-25T01:05:11.060 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-04-25T01:05:11.060 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-04-25T01:05:11.060 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-04-25T01:05:11.060 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-04-25T01:05:11.060 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-04-25T01:05:11.061 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-04-25T01:05:11.061 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-04-25T01:05:11.061 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-04-25T01:05:11.061 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-04-25T01:05:11.061 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-04-25T01:05:11.061 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-04-25T01:05:11.061 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-04-25T01:05:11.061 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-04-25T01:05:11.061 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-04-25T01:05:11.061 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-04-25T01:05:11.061 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-04-25T01:05:11.061 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-04-25T01:05:11.061 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-04-25T01:05:11.061 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-04-25T01:05:11.062 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-04-25T01:05:11.062 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-25T01:05:11.062 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-04-25T01:05:11.062 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-04-25T01:05:11.062 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-04-25T01:05:11.062 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-04-25T01:05:11.062 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-04-25T01:05:11.062 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-04-25T01:05:11.062 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-04-25T01:05:11.062 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-04-25T01:05:11.062 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-04-25T01:05:11.062 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-04-25T01:05:11.062 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-04-25T01:05:11.063 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-04-25T01:05:11.063 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-04-25T01:05:11.063 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-04-25T01:05:11.063 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-04-25T01:05:11.063 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-04-25T01:05:11.063 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-04-25T01:05:11.063 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-04-25T01:05:11.063 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-04-25T01:05:11.063 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-04-25T01:05:11.063 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-04-25T01:05:11.063 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-04-25T01:05:11.063 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-04-25T01:05:11.063 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-04-25T01:05:11.063 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-04-25T01:05:11.064 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-04-25T01:05:11.064 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-04-25T01:05:11.064 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-25T01:05:11.064 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-04-25T01:05:11.064 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-25T01:05:11.064 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-04-25T01:05:11.064 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-25T01:05:11.064 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-04-25T01:05:11.064 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-04-25T01:05:11.064 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-04-25T01:05:11.064 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-04-25T01:05:11.064 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-04-25T01:05:11.064 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-04-25T01:05:11.064 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-04-25T01:05:11.064 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-04-25T01:05:11.065 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-04-25T01:05:11.065 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-04-25T01:05:11.065 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-04-25T01:05:11.065 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-04-25T01:05:11.065 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-04-25T01:05:11.065 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-04-25T01:05:11.065 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-04-25T01:05:11.065 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-04-25T01:05:11.065 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-04-25T01:05:11.065 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-04-25T01:05:11.065 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-04-25T01:05:11.065 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-04-25T01:05:11.065 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-04-25T01:05:11.065 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-04-25T01:05:11.066 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-04-25T01:05:11.066 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-04-25T01:05:11.066 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-04-25T01:05:11.066 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-04-25T01:05:11.066 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-04-25T01:05:11.066 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-04-25T01:05:11.066 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-04-25T01:05:11.066 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-04-25T01:05:11.066 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-04-25T01:05:11.066 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-04-25T01:05:11.066 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-04-25T01:05:11.066 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-04-25T01:05:11.066 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-04-25T01:05:11.066 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-04-25T01:05:11.067 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-04-25T01:05:11.067 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-04-25T01:05:11.067 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-04-25T01:05:11.067 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-04-25T01:05:11.067 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-04-25T01:05:11.067 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-04-25T01:05:11.067 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-04-25T01:05:11.067 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-04-25T01:05:11.067 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-04-25T01:05:11.067 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-04-25T01:05:11.067 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-04-25T01:05:11.067 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-04-25T01:05:11.067 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-04-25T01:05:11.067 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-04-25T01:05:11.068 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-04-25T01:05:11.068 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-04-25T01:05:11.068 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-04-25T01:05:11.068 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-04-25T01:05:11.068 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-04-25T01:05:11.068 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-04-25T01:05:11.068 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-04-25T01:05:11.068 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-04-25T01:05:11.068 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-04-25T01:05:11.068 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-25T01:05:11.068 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-04-25T01:05:11.068 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-04-25T01:05:11.068 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-04-25T01:05:11.068 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-25T01:05:11.069 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-04-25T01:05:11.069 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-25T01:05:11.069 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-04-25T01:05:11.069 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-25T01:05:11.069 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-04-25T01:05:11.069 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-25T01:05:11.069 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-25T01:05:11.069 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-25T01:05:11.069 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-04-25T01:05:11.069 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-04-25T01:05:11.069 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-04-25T01:05:11.069 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-04-25T01:05:11.069 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-04-25T01:05:11.069 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-04-25T01:05:11.070 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-04-25T01:05:11.070 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-04-25T01:05:11.070 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-04-25T01:05:11.070 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-04-25T01:05:11.070 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-04-25T01:05:11.070 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-04-25T01:05:11.070 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-04-25T01:05:11.070 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-04-25T01:05:11.070 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-04-25T01:05:11.070 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-04-25T01:05:11.070 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-04-25T01:05:11.070 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-04-25T01:05:11.070 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-04-25T01:05:11.071 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-04-25T01:05:11.071 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-04-25T01:05:11.071 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-04-25T01:05:11.071 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-04-25T01:05:11.071 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-04-25T01:05:11.071 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-04-25T01:05:11.071 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-04-25T01:05:11.071 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-04-25T01:05:11.071 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-04-25T01:05:11.071 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-04-25T01:05:11.071 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-25T01:05:11.071 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-04-25T01:05:11.071 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-04-25T01:05:11.071 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-04-25T01:05:11.072 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-04-25T01:05:11.072 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-04-25T01:05:11.072 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-04-25T01:05:11.072 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-04-25T01:05:11.072 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-04-25T01:05:11.072 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-04-25T01:05:11.072 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-04-25T01:05:11.072 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-04-25T01:05:11.072 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-04-25T01:05:11.072 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-04-25T01:05:11.072 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-04-25T01:05:11.072 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-04-25T01:05:11.072 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-04-25T01:05:11.073 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-04-25T01:05:11.073 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-25T01:05:11.073 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-04-25T01:05:11.073 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-04-25T01:05:11.073 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-04-25T01:05:11.073 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-04-25T01:05:11.073 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-04-25T01:05:11.073 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-04-25T01:05:11.073 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-04-25T01:05:11.073 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-04-25T01:05:11.073 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-04-25T01:05:11.073 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-04-25T01:05:11.073 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-04-25T01:05:11.073 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-04-25T01:05:11.073 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-04-25T01:05:11.073 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-25T01:05:11.074 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-04-25T01:05:11.074 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-04-25T01:05:11.074 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-04-25T01:05:11.074 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-04-25T01:05:11.074 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-04-25T01:05:11.074 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-04-25T01:05:11.074 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-04-25T01:05:11.074 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-04-25T01:05:11.074 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-04-25T01:05:11.074 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-04-25T01:05:11.074 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-04-25T01:05:11.074 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-04-25T01:05:11.074 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-04-25T01:05:11.074 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-04-25T01:05:11.075 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-04-25T01:05:11.075 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-04-25T01:05:11.075 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-04-25T01:05:11.075 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-04-25T01:05:11.075 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-04-25T01:05:11.075 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-04-25T01:05:11.075 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-04-25T01:05:11.075 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-04-25T01:05:11.075 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-25T01:05:11.075 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-25T01:05:11.075 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-25T01:05:11.075 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-25T01:05:11.075 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-25T01:05:11.075 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-25T01:05:11.076 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-25T01:05:11.076 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-25T01:05:11.076 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-25T01:05:11.076 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-25T01:05:11.076 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-25T01:05:11.076 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-25T01:05:11.076 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-25T01:05:11.076 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-25T01:05:11.076 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-25T01:05:11.076 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-25T01:05:11.076 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-25T01:05:11.076 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-25T01:05:11.076 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-25T01:05:11.076 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-25T01:05:11.077 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-04-25T01:05:11.077 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-25T01:05:11.077 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-25T01:05:11.077 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-25T01:05:11.077 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-04-25T01:05:11.077 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-04-25T01:05:11.077 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-04-25T01:05:11.077 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-04-25T01:05:11.077 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-04-25T01:05:11.077 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-04-25T01:05:11.077 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-04-25T01:05:11.077 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-04-25T01:05:11.077 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-04-25T01:05:11.077 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-04-25T01:05:11.077 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-04-25T01:05:11.078 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-04-25T01:05:11.078 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-04-25T01:05:11.078 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-04-25T01:05:11.078 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-04-25T01:05:11.078 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-04-25T01:05:11.078 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-04-25T01:05:11.078 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-04-25T01:05:11.078 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-04-25T01:05:11.078 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-04-25T01:05:11.078 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-25T01:05:11.078 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-04-25T01:05:11.078 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-25T01:05:11.078 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-04-25T01:05:11.078 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-04-25T01:05:11.079 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-04-25T01:05:11.079 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-04-25T01:05:11.079 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-04-25T01:05:11.079 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-04-25T01:05:11.079 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-04-25T01:05:11.079 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-04-25T01:05:11.079 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-04-25T01:05:11.079 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-04-25T01:05:11.079 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-04-25T01:05:11.079 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-04-25T01:05:11.079 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-25T01:05:11.079 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-25T01:05:11.079 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-25T01:05:11.079 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-25T01:05:11.079 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-25T01:05:11.079 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-25T01:05:11.080 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-25T01:05:11.080 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-25T01:05:11.080 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-25T01:05:11.080 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-25T01:05:11.080 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-25T01:05:11.080 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-25T01:05:11.080 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-25T01:05:11.080 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-25T01:05:11.080 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-25T01:05:11.080 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-25T01:05:11.080 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-25T01:05:11.080 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-25T01:05:11.080 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-25T01:05:11.080 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-25T01:05:11.080 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-25T01:05:11.081 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-04-25T01:05:11.081 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-04-25T01:05:11.081 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-04-25T01:05:11.081 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-04-25T01:05:11.081 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-04-25T01:05:11.081 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-04-25T01:05:11.081 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-04-25T01:05:11.081 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-04-25T01:05:11.081 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-04-25T01:05:11.081 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-04-25T01:05:11.081 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-04-25T01:05:11.081 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-04-25T01:05:11.081 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-04-25T01:05:11.081 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-04-25T01:05:11.082 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-04-25T01:05:11.082 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-04-25T01:05:11.082 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-25T01:05:11.082 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-04-25T01:05:11.082 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-04-25T01:05:11.082 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-04-25T01:05:11.082 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-04-25T01:05:11.082 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-04-25T01:05:11.082 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-25T01:05:11.082 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-04-25T01:05:11.082 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-04-25T01:05:11.082 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-04-25T01:05:11.082 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-25T01:05:11.082 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-25T01:05:11.083 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-25T01:05:11.083 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-25T01:05:11.083 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-25T01:05:11.083 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-25T01:05:11.083 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-25T01:05:11.083 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-25T01:05:11.083 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-25T01:05:11.083 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-25T01:05:11.083 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-25T01:05:11.083 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-25T01:05:11.083 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-25T01:05:11.083 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-25T01:05:11.083 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-25T01:05:11.083 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-25T01:05:11.083 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-25T01:05:11.084 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-25T01:05:11.084 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-25T01:05:11.084 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-25T01:05:11.084 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-25T01:05:11.084 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-25T01:05:11.084 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-25T01:05:11.084 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-25T01:05:11.084 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-04-25T01:05:11.084 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-04-25T01:05:11.084 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-25T01:05:11.084 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-04-25T01:05:11.084 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-04-25T01:05:11.084 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-04-25T01:05:11.084 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-04-25T01:05:11.084 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-25T01:05:11.085 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-04-25T01:05:11.085 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-04-25T01:05:11.085 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-04-25T01:05:11.085 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-04-25T01:05:11.085 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-04-25T01:05:11.085 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-04-25T01:05:11.085 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-04-25T01:05:11.085 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-04-25T01:05:11.085 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-04-25T01:05:11.085 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-04-25T01:05:11.085 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-04-25T01:05:11.085 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-04-25T01:05:11.085 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-04-25T01:05:11.085 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-04-25T01:05:11.085 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-25T01:05:11.085 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-25T01:05:11.086 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-25T01:05:11.086 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-25T01:05:11.086 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-25T01:05:11.086 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-25T01:05:11.086 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-25T01:05:11.086 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-25T01:05:11.086 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-25T01:05:11.086 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-25T01:05:11.086 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-25T01:05:11.086 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-25T01:05:11.086 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-25T01:05:11.086 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-25T01:05:11.086 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-25T01:05:11.087 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-04-25T01:05:11.087 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-25T01:05:11.087 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-25T01:05:11.087 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-25T01:05:11.087 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-25T01:05:11.087 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-25T01:05:11.087 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-25T01:05:11.087 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-25T01:05:11.087 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-25T01:05:11.087 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-25T01:05:11.087 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-25T01:05:11.087 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-25T01:05:11.087 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-25T01:05:11.087 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-25T01:05:11.087 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-25T01:05:11.088 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-04-25T01:05:11.088 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-04-25T01:05:11.088 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-25T01:05:11.088 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-25T01:05:11.088 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-25T01:05:11.088 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-25T01:05:11.088 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-25T01:05:11.088 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-25T01:05:11.088 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-25T01:05:11.088 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-25T01:05:11.088 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-25T01:05:11.088 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-25T01:05:11.088 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-25T01:05:11.088 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-25T01:05:11.089 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-25T01:05:11.089 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-04-25T01:05:11.089 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-25T01:05:11.089 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-04-25T01:05:11.089 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-04-25T01:05:11.089 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-25T01:05:11.089 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-25T01:05:11.089 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-04-25T01:05:11.089 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-04-25T01:05:11.089 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-25T01:05:11.089 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-25T01:05:11.089 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-04-25T01:05:11.089 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-04-25T01:05:11.089 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-25T01:05:11.090 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-04-25T01:05:11.090 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-25T01:05:11.090 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-25T01:05:11.090 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-04-25T01:05:11.090 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-04-25T01:05:11.090 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-04-25T01:05:11.090 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-25T01:05:11.090 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-04-25T01:05:11.090 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-25T01:05:11.090 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-04-25T01:05:11.090 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-04-25T01:05:11.090 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-04-25T01:05:11.090 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-04-25T01:05:11.090 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-04-25T01:05:11.091 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-04-25T01:05:11.091 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-04-25T01:05:11.091 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-04-25T01:05:11.091 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-04-25T01:05:11.091 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-04-25T01:05:11.091 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-04-25T01:05:11.091 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-04-25T01:05:11.091 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-04-25T01:05:11.091 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-04-25T01:05:11.091 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-25T01:05:11.091 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-25T01:05:11.091 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-25T01:05:11.091 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-04-25T01:05:11.091 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-04-25T01:05:11.092 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-04-25T01:05:11.092 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-04-25T01:05:11.092 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-25T01:05:11.092 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-04-25T01:05:11.092 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-04-25T01:05:11.092 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-25T01:05:11.092 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-04-25T01:05:11.092 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-04-25T01:05:11.092 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-25T01:05:11.092 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-04-25T01:05:11.092 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-25T01:05:11.092 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-04-25T01:05:11.092 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-25T01:05:11.092 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-25T01:05:11.092 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-04-25T01:05:11.093 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-04-25T01:05:11.093 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-04-25T01:05:11.093 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-04-25T01:05:11.093 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-04-25T01:05:11.093 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-04-25T01:05:11.093 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-04-25T01:05:11.093 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-04-25T01:05:11.093 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-04-25T01:05:11.093 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-04-25T01:05:11.093 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-04-25T01:05:11.093 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-04-25T01:05:11.093 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-04-25T01:05:11.093 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-04-25T01:05:11.093 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-04-25T01:05:11.094 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-04-25T01:05:11.094 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-04-25T01:05:11.094 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-04-25T01:05:11.094 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-04-25T01:05:11.094 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-04-25T01:05:11.094 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-04-25T01:05:11.094 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-04-25T01:05:11.094 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-04-25T01:05:11.094 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-04-25T01:05:11.094 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-04-25T01:05:11.094 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-04-25T01:05:11.094 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-04-25T01:05:11.094 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-04-25T01:05:11.094 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-04-25T01:05:11.095 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-04-25T01:05:11.095 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-04-25T01:05:11.095 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-04-25T01:05:11.095 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-04-25T01:05:11.095 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-04-25T01:05:11.095 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-04-25T01:05:11.095 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-04-25T01:05:11.095 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-04-25T01:05:11.095 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-04-25T01:05:11.095 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-04-25T01:05:11.095 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-04-25T01:05:11.095 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-04-25T01:05:11.095 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-04-25T01:05:11.095 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-04-25T01:05:11.096 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-04-25T01:05:11.096 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-04-25T01:05:11.096 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-04-25T01:05:11.096 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-04-25T01:05:11.096 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-04-25T01:05:11.096 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-04-25T01:05:11.096 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-04-25T01:05:11.096 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-04-25T01:05:11.096 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-04-25T01:05:11.096 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-04-25T01:05:11.096 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-04-25T01:05:11.096 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-04-25T01:05:11.096 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-04-25T01:05:11.096 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-04-25T01:05:11.097 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-04-25T01:05:11.097 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-04-25T01:05:11.097 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-04-25T01:05:11.097 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-04-25T01:05:11.097 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-04-25T01:05:11.097 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-04-25T01:05:11.097 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-04-25T01:05:11.097 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-04-25T01:05:11.097 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-04-25T01:05:11.097 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-04-25T01:05:11.097 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-04-25T01:05:11.097 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-04-25T01:05:11.097 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-04-25T01:05:11.097 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-04-25T01:05:11.098 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-04-25T01:05:11.098 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-04-25T01:05:11.098 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-04-25T01:05:11.098 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-04-25T01:05:11.098 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-04-25T01:05:11.098 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-04-25T01:05:11.098 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-04-25T01:05:11.098 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-04-25T01:05:11.098 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-04-25T01:05:11.098 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-04-25T01:05:11.098 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-04-25T01:05:11.098 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-04-25T01:05:11.098 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-04-25T01:05:11.098 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-04-25T01:05:11.099 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-04-25T01:05:11.099 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-04-25T01:05:11.099 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-04-25T01:05:11.099 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-04-25T01:05:11.099 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-04-25T01:05:11.099 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-04-25T01:05:11.099 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-04-25T01:05:11.099 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-04-25T01:05:11.099 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-04-25T01:05:11.099 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-04-25T01:05:11.099 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-04-25T01:05:11.099 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-04-25T01:05:11.099 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-04-25T01:05:11.099 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-04-25T01:05:11.099 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-04-25T01:05:11.100 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-04-25T01:05:11.100 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-04-25T01:05:11.100 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-04-25T01:05:11.100 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-04-25T01:05:11.100 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-04-25T01:05:11.100 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-04-25T01:05:11.100 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-04-25T01:05:11.100 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-04-25T01:05:11.100 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-04-25T01:05:11.100 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-04-25T01:05:11.100 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-04-25T01:05:11.100 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-04-25T01:05:11.100 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-04-25T01:05:11.100 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-04-25T01:05:11.101 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-04-25T01:05:11.101 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-25T01:05:11.101 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-25T01:05:11.101 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-04-25T01:05:11.101 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-04-25T01:05:11.101 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-25T01:05:11.101 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-04-25T01:05:11.101 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-04-25T01:05:11.101 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-04-25T01:05:11.101 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-04-25T01:05:11.101 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-04-25T01:05:11.101 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-04-25T01:05:11.101 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-25T01:05:11.101 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-25T01:05:11.102 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-04-25T01:05:11.102 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-25T01:05:11.102 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-25T01:05:11.102 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-04-25T01:05:11.102 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-25T01:05:11.102 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-04-25T01:05:11.102 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-04-25T01:05:11.102 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-04-25T01:05:11.102 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-25T01:05:11.102 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-04-25T01:05:11.102 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-04-25T01:05:11.102 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-04-25T01:05:11.102 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-04-25T01:05:11.102 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-04-25T01:05:11.103 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-04-25T01:05:11.103 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-04-25T01:05:11.103 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-04-25T01:05:11.103 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-04-25T01:05:11.103 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-04-25T01:05:11.103 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-04-25T01:05:11.103 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-04-25T01:05:11.103 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-04-25T01:05:11.103 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-04-25T01:05:11.103 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-04-25T01:05:11.103 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-25T01:05:11.107 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-04-25T01:05:11.107 DEBUG:teuthology.orchestra.run.smithi090:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-25T01:05:11.131 INFO:teuthology.orchestra.run.smithi090.stdout:postinstall scriptlet (using /bin/sh): 2024-04-25T01:05:11.132 INFO:teuthology.orchestra.run.smithi090.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-25T01:05:11.132 INFO:teuthology.orchestra.run.smithi090.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-25T01:05:11.132 INFO:teuthology.orchestra.run.smithi090.stdout:fi 2024-04-25T01:05:11.132 INFO:teuthology.orchestra.run.smithi090.stdout:for file in vmlinuz System.map config; do 2024-04-25T01:05:11.132 INFO:teuthology.orchestra.run.smithi090.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-04-25T01:05:11.132 INFO:teuthology.orchestra.run.smithi090.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-04-25T01:05:11.132 INFO:teuthology.orchestra.run.smithi090.stdout: fi 2024-04-25T01:05:11.132 INFO:teuthology.orchestra.run.smithi090.stdout:done 2024-04-25T01:05:11.132 INFO:teuthology.orchestra.run.smithi090.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-04-25T01:05:11.132 INFO:teuthology.orchestra.run.smithi090.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-04-25T01:05:11.132 INFO:teuthology.orchestra.run.smithi090.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-25T01:05:11.132 INFO:teuthology.orchestra.run.smithi090.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-25T01:05:11.132 INFO:teuthology.orchestra.run.smithi090.stdout:fi 2024-04-25T01:05:11.132 INFO:teuthology.orchestra.run.smithi090.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-04-25T01:05:11.133 INFO:teuthology.orchestra.run.smithi090.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-04-25T01:05:11.133 INFO:teuthology.orchestra.run.smithi090.stdout:fi 2024-04-25T01:05:11.133 INFO:teuthology.orchestra.run.smithi090.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-25T01:05:11.133 INFO:teuthology.orchestra.run.smithi090.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-25T01:05:11.133 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-04-25T01:05:11.133 INFO:teuthology.orchestra.run.smithi090.stdout:fi 2024-04-25T01:05:11.134 DEBUG:teuthology.orchestra.run.smithi090:> sudo rpm -qi grub2-tools 2024-04-25T01:05:11.208 INFO:teuthology.orchestra.run.smithi090.stdout:Name : grub2-tools 2024-04-25T01:05:11.208 INFO:teuthology.orchestra.run.smithi090.stdout:Epoch : 1 2024-04-25T01:05:11.208 INFO:teuthology.orchestra.run.smithi090.stdout:Version : 2.06 2024-04-25T01:05:11.209 INFO:teuthology.orchestra.run.smithi090.stdout:Release : 68.el9 2024-04-25T01:05:11.209 INFO:teuthology.orchestra.run.smithi090.stdout:Architecture: x86_64 2024-04-25T01:05:11.209 INFO:teuthology.orchestra.run.smithi090.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-04-25T01:05:11.209 INFO:teuthology.orchestra.run.smithi090.stdout:Group : Unspecified 2024-04-25T01:05:11.209 INFO:teuthology.orchestra.run.smithi090.stdout:Size : 8264254 2024-04-25T01:05:11.209 INFO:teuthology.orchestra.run.smithi090.stdout:License : GPLv3+ 2024-04-25T01:05:11.209 INFO:teuthology.orchestra.run.smithi090.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-25T01:05:11.209 INFO:teuthology.orchestra.run.smithi090.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-25T01:05:11.209 INFO:teuthology.orchestra.run.smithi090.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-25T01:05:11.209 INFO:teuthology.orchestra.run.smithi090.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-25T01:05:11.209 INFO:teuthology.orchestra.run.smithi090.stdout:Packager : builder@centos.org 2024-04-25T01:05:11.209 INFO:teuthology.orchestra.run.smithi090.stdout:Vendor : CentOS 2024-04-25T01:05:11.209 INFO:teuthology.orchestra.run.smithi090.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-25T01:05:11.209 INFO:teuthology.orchestra.run.smithi090.stdout:Summary : Support tools for GRUB. 2024-04-25T01:05:11.209 INFO:teuthology.orchestra.run.smithi090.stdout:Description : 2024-04-25T01:05:11.210 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:05:11.210 INFO:teuthology.orchestra.run.smithi090.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-25T01:05:11.210 INFO:teuthology.orchestra.run.smithi090.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-25T01:05:11.210 INFO:teuthology.orchestra.run.smithi090.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-25T01:05:11.210 INFO:teuthology.orchestra.run.smithi090.stdout:hardware devices. 2024-04-25T01:05:11.210 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:05:11.210 INFO:teuthology.orchestra.run.smithi090.stdout:This subpackage provides tools for support of all platforms. 2024-04-25T01:05:11.211 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-25T01:05:11.211 INFO:teuthology.task.kernel:Updating grub on smithi090 to boot 6.9.0-rc3-g806ec36b2112 2024-04-25T01:05:11.211 DEBUG:teuthology.orchestra.run.smithi090:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-25T01:05:11.956 INFO:teuthology.orchestra.run.smithi090.stderr:Generating grub configuration file ... 2024-04-25T01:05:13.730 INFO:teuthology.orchestra.run.smithi090.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-25T01:05:13.754 INFO:teuthology.orchestra.run.smithi090.stderr:done 2024-04-25T01:05:13.756 DEBUG:teuthology.orchestra.run.smithi090:> mktemp 2024-04-25T01:05:13.815 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/tmp.XKcWQqiWfr 2024-04-25T01:05:13.816 DEBUG:teuthology.orchestra.run.smithi090:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.XKcWQqiWfr 2024-04-25T01:05:13.859 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0666 /tmp/tmp.XKcWQqiWfr 2024-04-25T01:05:14.008 DEBUG:teuthology.orchestra.remote:smithi090:/tmp/tmp.XKcWQqiWfr is 6KB 2024-04-25T01:05:14.056 DEBUG:teuthology.orchestra.run.smithi090:> rm -fr /tmp/tmp.XKcWQqiWfr 2024-04-25T01:05:14.070 DEBUG:teuthology.orchestra.run.smithi090:> sudo /bin/ls /boot/loader/entries || true 2024-04-25T01:05:14.138 INFO:teuthology.orchestra.run.smithi090.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-04-25T01:05:14.138 INFO:teuthology.orchestra.run.smithi090.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-04-25T01:05:14.138 INFO:teuthology.orchestra.run.smithi090.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-04-25T01:05:14.138 INFO:teuthology.orchestra.run.smithi090.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112.conf 2024-04-25T01:05:14.140 DEBUG:teuthology.orchestra.run.smithi090:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112 2024-04-25T01:05:14.293 DEBUG:teuthology.orchestra.run.smithi090:> sudo shutdown -r now 2024-04-25T01:05:29.054 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi103.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-25T01:05:29.054 DEBUG:teuthology.orchestra.run.smithi103:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-25T01:05:29.137 INFO:teuthology.orchestra.run.smithi103.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-04-25T01:05:29.138 INFO:teuthology.orchestra.run.smithi103.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-04-25T01:05:29.138 INFO:teuthology.orchestra.run.smithi103.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-04-25T01:05:29.138 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-04-25T01:05:29.138 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-04-25T01:05:29.138 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-04-25T01:05:29.138 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-04-25T01:05:29.138 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-04-25T01:05:29.138 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-04-25T01:05:29.138 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-25T01:05:29.138 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-25T01:05:29.138 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-25T01:05:29.138 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-25T01:05:29.139 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-25T01:05:29.139 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-25T01:05:29.139 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-25T01:05:29.139 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-25T01:05:29.139 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-25T01:05:29.139 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-25T01:05:29.139 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-25T01:05:29.139 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-25T01:05:29.139 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-25T01:05:29.139 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-25T01:05:29.139 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-25T01:05:29.139 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-25T01:05:29.139 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-25T01:05:29.139 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-25T01:05:29.139 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-25T01:05:29.140 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-25T01:05:29.140 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-04-25T01:05:29.140 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-04-25T01:05:29.140 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-04-25T01:05:29.140 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-25T01:05:29.140 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-04-25T01:05:29.140 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-25T01:05:29.140 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-25T01:05:29.140 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-04-25T01:05:29.140 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-04-25T01:05:29.140 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-04-25T01:05:29.140 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-04-25T01:05:29.140 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-04-25T01:05:29.140 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-04-25T01:05:29.141 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-04-25T01:05:29.141 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-04-25T01:05:29.141 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-25T01:05:29.141 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-04-25T01:05:29.141 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-04-25T01:05:29.141 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-04-25T01:05:29.141 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-04-25T01:05:29.141 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-04-25T01:05:29.141 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-04-25T01:05:29.141 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-04-25T01:05:29.141 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-04-25T01:05:29.141 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-04-25T01:05:29.141 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-04-25T01:05:29.141 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-04-25T01:05:29.141 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-04-25T01:05:29.142 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-04-25T01:05:29.142 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-04-25T01:05:29.142 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-04-25T01:05:29.142 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-04-25T01:05:29.142 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-04-25T01:05:29.142 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-04-25T01:05:29.142 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-04-25T01:05:29.142 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-04-25T01:05:29.142 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-04-25T01:05:29.142 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-04-25T01:05:29.142 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-04-25T01:05:29.142 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-04-25T01:05:29.142 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-04-25T01:05:29.142 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-04-25T01:05:29.143 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-04-25T01:05:29.143 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-04-25T01:05:29.143 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-04-25T01:05:29.143 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-04-25T01:05:29.143 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-04-25T01:05:29.143 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-04-25T01:05:29.143 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-04-25T01:05:29.143 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-04-25T01:05:29.143 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-04-25T01:05:29.143 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-04-25T01:05:29.143 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-04-25T01:05:29.143 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-04-25T01:05:29.143 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-04-25T01:05:29.143 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-04-25T01:05:29.144 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-04-25T01:05:29.144 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-04-25T01:05:29.144 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-04-25T01:05:29.144 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-04-25T01:05:29.144 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-25T01:05:29.144 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-04-25T01:05:29.144 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-04-25T01:05:29.144 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-04-25T01:05:29.144 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-04-25T01:05:29.144 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-04-25T01:05:29.144 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-04-25T01:05:29.144 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-04-25T01:05:29.144 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-04-25T01:05:29.144 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-04-25T01:05:29.144 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-04-25T01:05:29.145 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-04-25T01:05:29.145 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-04-25T01:05:29.145 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-04-25T01:05:29.145 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-04-25T01:05:29.145 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-04-25T01:05:29.145 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-04-25T01:05:29.145 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-04-25T01:05:29.145 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-04-25T01:05:29.145 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-04-25T01:05:29.145 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-04-25T01:05:29.145 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-04-25T01:05:29.145 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-04-25T01:05:29.145 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-04-25T01:05:29.145 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-25T01:05:29.145 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-25T01:05:29.146 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-04-25T01:05:29.146 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-04-25T01:05:29.146 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-04-25T01:05:29.146 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-04-25T01:05:29.146 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-04-25T01:05:29.146 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-04-25T01:05:29.146 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-04-25T01:05:29.146 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-04-25T01:05:29.146 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-04-25T01:05:29.146 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-04-25T01:05:29.146 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-04-25T01:05:29.146 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-04-25T01:05:29.146 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-04-25T01:05:29.146 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-25T01:05:29.147 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-25T01:05:29.147 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-04-25T01:05:29.147 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-04-25T01:05:29.147 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-04-25T01:05:29.147 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-04-25T01:05:29.147 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-04-25T01:05:29.147 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-04-25T01:05:29.147 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-04-25T01:05:29.147 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-04-25T01:05:29.147 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-04-25T01:05:29.147 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-04-25T01:05:29.147 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-04-25T01:05:29.147 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-04-25T01:05:29.147 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-04-25T01:05:29.148 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-04-25T01:05:29.148 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-04-25T01:05:29.148 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-04-25T01:05:29.148 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-04-25T01:05:29.148 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-04-25T01:05:29.148 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-04-25T01:05:29.148 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-04-25T01:05:29.148 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-04-25T01:05:29.148 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-04-25T01:05:29.148 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-04-25T01:05:29.148 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-04-25T01:05:29.148 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-04-25T01:05:29.148 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-04-25T01:05:29.148 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-04-25T01:05:29.148 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-25T01:05:29.149 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-25T01:05:29.149 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-04-25T01:05:29.149 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-04-25T01:05:29.149 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-04-25T01:05:29.149 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-04-25T01:05:29.149 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-04-25T01:05:29.149 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-04-25T01:05:29.149 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-04-25T01:05:29.149 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-04-25T01:05:29.149 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-04-25T01:05:29.149 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-04-25T01:05:29.149 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-04-25T01:05:29.149 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-25T01:05:29.149 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-04-25T01:05:29.150 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-04-25T01:05:29.150 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-04-25T01:05:29.150 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-04-25T01:05:29.150 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-04-25T01:05:29.150 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-04-25T01:05:29.150 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-04-25T01:05:29.150 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-25T01:05:29.150 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-04-25T01:05:29.150 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-04-25T01:05:29.150 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-04-25T01:05:29.150 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-04-25T01:05:29.150 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-04-25T01:05:29.150 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-04-25T01:05:29.150 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-25T01:05:29.151 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-25T01:05:29.151 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-25T01:05:29.151 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-04-25T01:05:29.151 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-25T01:05:29.151 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-04-25T01:05:29.151 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-25T01:05:29.151 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-25T01:05:29.151 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-25T01:05:29.151 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-25T01:05:29.151 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-25T01:05:29.151 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-25T01:05:29.151 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-04-25T01:05:29.151 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-04-25T01:05:29.151 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-04-25T01:05:29.151 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-04-25T01:05:29.152 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-04-25T01:05:29.152 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-04-25T01:05:29.152 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-04-25T01:05:29.152 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-25T01:05:29.152 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-25T01:05:29.152 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-25T01:05:29.152 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-25T01:05:29.152 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-25T01:05:29.152 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-04-25T01:05:29.152 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-04-25T01:05:29.152 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-04-25T01:05:29.152 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-25T01:05:29.152 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-25T01:05:29.152 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-25T01:05:29.152 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-25T01:05:29.153 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-25T01:05:29.153 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-25T01:05:29.153 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-04-25T01:05:29.153 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-04-25T01:05:29.153 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-25T01:05:29.153 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-04-25T01:05:29.153 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-04-25T01:05:29.153 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-04-25T01:05:29.153 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-04-25T01:05:29.153 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-25T01:05:29.153 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-25T01:05:29.153 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-25T01:05:29.153 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-04-25T01:05:29.153 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-04-25T01:05:29.154 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-04-25T01:05:29.154 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-25T01:05:29.154 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-04-25T01:05:29.154 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-04-25T01:05:29.154 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-04-25T01:05:29.154 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-25T01:05:29.154 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-04-25T01:05:29.154 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-04-25T01:05:29.154 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-25T01:05:29.154 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-04-25T01:05:29.154 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-25T01:05:29.154 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-04-25T01:05:29.154 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-04-25T01:05:29.154 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-04-25T01:05:29.155 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-04-25T01:05:29.155 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-04-25T01:05:29.155 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-04-25T01:05:29.155 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-04-25T01:05:29.155 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-04-25T01:05:29.155 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-04-25T01:05:29.155 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-04-25T01:05:29.155 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-04-25T01:05:29.155 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-04-25T01:05:29.155 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-04-25T01:05:29.155 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-04-25T01:05:29.155 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-04-25T01:05:29.155 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-04-25T01:05:29.155 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-25T01:05:29.155 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-04-25T01:05:29.156 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-04-25T01:05:29.156 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-25T01:05:29.156 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-04-25T01:05:29.156 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-04-25T01:05:29.156 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-04-25T01:05:29.156 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-04-25T01:05:29.156 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-04-25T01:05:29.156 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-04-25T01:05:29.156 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-04-25T01:05:29.156 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-04-25T01:05:29.156 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-04-25T01:05:29.156 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-04-25T01:05:29.156 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-04-25T01:05:29.156 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-25T01:05:29.156 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-25T01:05:29.157 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-04-25T01:05:29.157 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-04-25T01:05:29.157 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-04-25T01:05:29.157 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-04-25T01:05:29.157 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-04-25T01:05:29.157 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-04-25T01:05:29.157 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-04-25T01:05:29.157 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-04-25T01:05:29.157 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-25T01:05:29.157 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-25T01:05:29.157 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-04-25T01:05:29.157 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-04-25T01:05:29.157 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-04-25T01:05:29.157 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-04-25T01:05:29.158 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-04-25T01:05:29.158 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-04-25T01:05:29.158 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-04-25T01:05:29.158 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-04-25T01:05:29.158 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-04-25T01:05:29.158 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-25T01:05:29.158 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-04-25T01:05:29.158 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-25T01:05:29.158 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-25T01:05:29.158 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-25T01:05:29.158 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-25T01:05:29.158 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-25T01:05:29.158 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-25T01:05:29.158 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-25T01:05:29.159 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-25T01:05:29.159 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-25T01:05:29.159 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-04-25T01:05:29.159 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-04-25T01:05:29.159 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-04-25T01:05:29.159 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-25T01:05:29.159 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-04-25T01:05:29.159 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-04-25T01:05:29.159 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-04-25T01:05:29.159 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-04-25T01:05:29.159 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-04-25T01:05:29.159 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-25T01:05:29.159 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-04-25T01:05:29.159 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-04-25T01:05:29.159 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-04-25T01:05:29.160 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-04-25T01:05:29.160 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-04-25T01:05:29.160 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-04-25T01:05:29.160 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-04-25T01:05:29.160 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-04-25T01:05:29.160 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-04-25T01:05:29.160 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-04-25T01:05:29.160 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-04-25T01:05:29.160 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-04-25T01:05:29.160 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-04-25T01:05:29.160 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-04-25T01:05:29.160 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-04-25T01:05:29.160 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-04-25T01:05:29.160 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-04-25T01:05:29.161 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-04-25T01:05:29.161 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-04-25T01:05:29.161 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-25T01:05:29.161 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-04-25T01:05:29.161 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-04-25T01:05:29.161 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-04-25T01:05:29.161 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-04-25T01:05:29.161 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-04-25T01:05:29.161 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-04-25T01:05:29.161 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-04-25T01:05:29.161 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-04-25T01:05:29.161 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-04-25T01:05:29.161 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-04-25T01:05:29.162 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-04-25T01:05:29.162 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-04-25T01:05:29.162 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-04-25T01:05:29.162 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-04-25T01:05:29.162 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-04-25T01:05:29.162 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-04-25T01:05:29.162 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-04-25T01:05:29.162 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-04-25T01:05:29.162 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-04-25T01:05:29.162 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-25T01:05:29.162 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-04-25T01:05:29.162 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-04-25T01:05:29.162 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-25T01:05:29.162 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-04-25T01:05:29.163 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-04-25T01:05:29.163 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-04-25T01:05:29.163 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-04-25T01:05:29.163 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-04-25T01:05:29.163 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-04-25T01:05:29.163 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-04-25T01:05:29.163 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-04-25T01:05:29.163 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-04-25T01:05:29.163 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-04-25T01:05:29.163 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-04-25T01:05:29.163 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-04-25T01:05:29.163 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-04-25T01:05:29.163 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-04-25T01:05:29.163 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-25T01:05:29.164 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-04-25T01:05:29.164 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-04-25T01:05:29.164 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-04-25T01:05:29.164 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-04-25T01:05:29.164 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-04-25T01:05:29.164 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-04-25T01:05:29.164 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-04-25T01:05:29.164 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-04-25T01:05:29.164 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-04-25T01:05:29.164 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-04-25T01:05:29.164 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-04-25T01:05:29.164 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-04-25T01:05:29.164 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-04-25T01:05:29.164 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-04-25T01:05:29.164 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-04-25T01:05:29.165 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-04-25T01:05:29.165 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-04-25T01:05:29.165 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-04-25T01:05:29.165 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-04-25T01:05:29.165 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-04-25T01:05:29.165 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-04-25T01:05:29.165 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-04-25T01:05:29.165 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-04-25T01:05:29.165 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-04-25T01:05:29.165 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-04-25T01:05:29.165 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-04-25T01:05:29.165 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-04-25T01:05:29.165 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-04-25T01:05:29.165 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-04-25T01:05:29.165 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-04-25T01:05:29.166 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-04-25T01:05:29.166 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-04-25T01:05:29.166 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-04-25T01:05:29.166 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-04-25T01:05:29.166 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-04-25T01:05:29.166 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-04-25T01:05:29.166 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-04-25T01:05:29.166 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-04-25T01:05:29.166 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-04-25T01:05:29.166 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-04-25T01:05:29.166 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-04-25T01:05:29.166 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-04-25T01:05:29.166 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-04-25T01:05:29.166 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-04-25T01:05:29.167 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-04-25T01:05:29.167 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-04-25T01:05:29.167 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-04-25T01:05:29.167 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-04-25T01:05:29.167 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-25T01:05:29.167 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-25T01:05:29.167 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-25T01:05:29.167 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-25T01:05:29.167 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-25T01:05:29.167 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-25T01:05:29.167 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-25T01:05:29.167 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-25T01:05:29.167 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-25T01:05:29.167 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-25T01:05:29.168 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-25T01:05:29.168 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-25T01:05:29.168 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-04-25T01:05:29.168 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-04-25T01:05:29.168 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-04-25T01:05:29.168 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-25T01:05:29.168 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-04-25T01:05:29.168 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-04-25T01:05:29.168 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-04-25T01:05:29.168 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-04-25T01:05:29.168 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-04-25T01:05:29.168 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-04-25T01:05:29.168 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-04-25T01:05:29.168 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-04-25T01:05:29.168 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-04-25T01:05:29.169 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-04-25T01:05:29.169 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-04-25T01:05:29.169 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-04-25T01:05:29.169 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-04-25T01:05:29.169 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-04-25T01:05:29.169 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-04-25T01:05:29.169 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-04-25T01:05:29.169 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-04-25T01:05:29.169 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-04-25T01:05:29.169 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-04-25T01:05:29.169 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-04-25T01:05:29.169 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-04-25T01:05:29.169 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-04-25T01:05:29.169 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-04-25T01:05:29.170 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-04-25T01:05:29.170 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-04-25T01:05:29.170 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-04-25T01:05:29.170 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-04-25T01:05:29.170 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-25T01:05:29.170 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-25T01:05:29.170 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-04-25T01:05:29.170 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-25T01:05:29.170 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-25T01:05:29.170 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-25T01:05:29.170 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-25T01:05:29.170 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-25T01:05:29.170 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-25T01:05:29.170 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-25T01:05:29.170 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-25T01:05:29.171 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-25T01:05:29.171 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-25T01:05:29.171 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-25T01:05:29.171 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-25T01:05:29.171 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-25T01:05:29.171 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-25T01:05:29.171 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-25T01:05:29.171 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-25T01:05:29.171 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-25T01:05:29.171 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-25T01:05:29.171 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-25T01:05:29.171 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-25T01:05:29.171 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-25T01:05:29.171 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-25T01:05:29.172 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-25T01:05:29.172 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-04-25T01:05:29.172 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-04-25T01:05:29.172 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-25T01:05:29.172 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-04-25T01:05:29.172 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-04-25T01:05:29.172 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-04-25T01:05:29.172 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-04-25T01:05:29.172 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-25T01:05:29.172 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-04-25T01:05:29.172 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-25T01:05:29.172 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-25T01:05:29.172 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-25T01:05:29.172 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-25T01:05:29.172 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-25T01:05:29.173 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-04-25T01:05:29.173 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-04-25T01:05:29.173 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-25T01:05:29.173 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-04-25T01:05:29.173 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-25T01:05:29.173 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-04-25T01:05:29.173 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-25T01:05:29.173 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-04-25T01:05:29.173 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-25T01:05:29.173 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-04-25T01:05:29.173 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-04-25T01:05:29.173 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-25T01:05:29.173 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-04-25T01:05:29.173 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-25T01:05:29.173 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-04-25T01:05:29.174 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-25T01:05:29.174 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-04-25T01:05:29.174 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-25T01:05:29.174 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-04-25T01:05:29.174 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-25T01:05:29.174 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-04-25T01:05:29.174 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-04-25T01:05:29.174 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-25T01:05:29.174 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-25T01:05:29.174 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-04-25T01:05:29.174 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-04-25T01:05:29.174 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-04-25T01:05:29.175 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-25T01:05:29.175 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-04-25T01:05:29.175 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-04-25T01:05:29.175 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-04-25T01:05:29.175 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-04-25T01:05:29.175 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-25T01:05:29.175 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-04-25T01:05:29.175 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-25T01:05:29.175 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-25T01:05:29.175 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-25T01:05:29.175 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-04-25T01:05:29.175 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-25T01:05:29.175 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-04-25T01:05:29.175 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-04-25T01:05:29.175 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-04-25T01:05:29.176 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-25T01:05:29.176 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-04-25T01:05:29.176 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-04-25T01:05:29.176 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-04-25T01:05:29.176 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-04-25T01:05:29.176 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-04-25T01:05:29.176 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-04-25T01:05:29.176 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-04-25T01:05:29.176 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-04-25T01:05:29.176 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-25T01:05:29.176 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-25T01:05:29.176 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-25T01:05:29.176 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-25T01:05:29.176 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-25T01:05:29.177 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-25T01:05:29.177 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-25T01:05:29.177 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-25T01:05:29.177 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-25T01:05:29.177 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-04-25T01:05:29.177 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-04-25T01:05:29.177 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-04-25T01:05:29.177 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-04-25T01:05:29.177 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-04-25T01:05:29.177 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-04-25T01:05:29.177 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-04-25T01:05:29.177 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-04-25T01:05:29.177 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-04-25T01:05:29.177 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-04-25T01:05:29.177 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-04-25T01:05:29.178 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-04-25T01:05:29.178 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-04-25T01:05:29.178 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-04-25T01:05:29.178 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-04-25T01:05:29.178 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-04-25T01:05:29.178 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-04-25T01:05:29.178 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-04-25T01:05:29.178 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-04-25T01:05:29.178 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-25T01:05:29.178 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-04-25T01:05:29.178 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-04-25T01:05:29.178 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-04-25T01:05:29.178 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-04-25T01:05:29.179 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-04-25T01:05:29.179 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-04-25T01:05:29.179 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-04-25T01:05:29.179 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-04-25T01:05:29.179 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-04-25T01:05:29.179 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-04-25T01:05:29.179 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-04-25T01:05:29.179 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-04-25T01:05:29.179 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-04-25T01:05:29.179 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-04-25T01:05:29.179 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-04-25T01:05:29.179 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-04-25T01:05:29.179 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-04-25T01:05:29.179 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-04-25T01:05:29.180 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-04-25T01:05:29.180 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-04-25T01:05:29.180 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-04-25T01:05:29.180 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-04-25T01:05:29.180 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-04-25T01:05:29.180 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-25T01:05:29.180 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-04-25T01:05:29.180 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-04-25T01:05:29.180 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-04-25T01:05:29.180 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-04-25T01:05:29.180 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-04-25T01:05:29.180 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-04-25T01:05:29.180 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-04-25T01:05:29.180 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-25T01:05:29.180 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-04-25T01:05:29.181 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-04-25T01:05:29.181 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-04-25T01:05:29.181 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-04-25T01:05:29.181 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-04-25T01:05:29.181 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-04-25T01:05:29.181 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-25T01:05:29.181 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-25T01:05:29.181 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-04-25T01:05:29.181 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-04-25T01:05:29.181 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-04-25T01:05:29.181 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-04-25T01:05:29.181 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-04-25T01:05:29.181 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-25T01:05:29.181 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-04-25T01:05:29.181 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-25T01:05:29.182 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-25T01:05:29.182 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-25T01:05:29.182 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-04-25T01:05:29.182 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-04-25T01:05:29.182 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-04-25T01:05:29.182 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-04-25T01:05:29.182 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-04-25T01:05:29.182 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-25T01:05:29.182 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-04-25T01:05:29.182 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-04-25T01:05:29.182 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-04-25T01:05:29.182 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-04-25T01:05:29.182 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-04-25T01:05:29.182 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-25T01:05:29.183 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-25T01:05:29.183 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-25T01:05:29.183 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-04-25T01:05:29.183 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-25T01:05:29.183 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-04-25T01:05:29.183 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-04-25T01:05:29.183 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-04-25T01:05:29.183 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-04-25T01:05:29.183 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-25T01:05:29.183 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-25T01:05:29.183 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-25T01:05:29.183 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-25T01:05:29.183 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-04-25T01:05:29.183 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-25T01:05:29.183 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-25T01:05:29.184 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-25T01:05:29.184 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-25T01:05:29.184 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-04-25T01:05:29.184 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-25T01:05:29.184 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-04-25T01:05:29.184 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-25T01:05:29.184 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-04-25T01:05:29.184 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-25T01:05:29.184 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-04-25T01:05:29.184 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-25T01:05:29.184 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-04-25T01:05:29.184 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-25T01:05:29.184 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-25T01:05:29.184 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-25T01:05:29.184 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-04-25T01:05:29.185 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-04-25T01:05:29.185 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-25T01:05:29.185 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-25T01:05:29.185 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-25T01:05:29.185 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-25T01:05:29.185 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-25T01:05:29.185 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-04-25T01:05:29.185 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-25T01:05:29.185 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-25T01:05:29.185 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-04-25T01:05:29.185 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-25T01:05:29.185 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-25T01:05:29.185 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-25T01:05:29.185 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-25T01:05:29.185 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-25T01:05:29.186 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-04-25T01:05:29.186 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-25T01:05:29.186 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-25T01:05:29.186 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-04-25T01:05:29.186 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-04-25T01:05:29.186 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-25T01:05:29.186 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-04-25T01:05:29.186 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-25T01:05:29.186 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-25T01:05:29.186 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-25T01:05:29.186 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-25T01:05:29.186 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-25T01:05:29.186 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-25T01:05:29.187 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-25T01:05:29.187 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-25T01:05:29.187 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-25T01:05:29.187 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-25T01:05:29.187 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-04-25T01:05:29.187 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-04-25T01:05:29.187 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-25T01:05:29.187 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-04-25T01:05:29.187 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-04-25T01:05:29.187 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-25T01:05:29.187 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-25T01:05:29.187 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-25T01:05:29.187 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-25T01:05:29.187 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-25T01:05:29.188 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-25T01:05:29.188 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-04-25T01:05:29.188 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-25T01:05:29.188 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-25T01:05:29.188 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-04-25T01:05:29.188 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-04-25T01:05:29.188 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-04-25T01:05:29.188 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-25T01:05:29.188 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-04-25T01:05:29.188 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-04-25T01:05:29.188 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-04-25T01:05:29.188 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-25T01:05:29.188 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-04-25T01:05:29.188 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-25T01:05:29.188 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-04-25T01:05:29.189 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-25T01:05:29.189 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-04-25T01:05:29.189 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-25T01:05:29.189 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-04-25T01:05:29.189 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-25T01:05:29.189 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-04-25T01:05:29.189 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-25T01:05:29.189 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-04-25T01:05:29.189 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-25T01:05:29.189 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-04-25T01:05:29.189 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-25T01:05:29.189 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-25T01:05:29.189 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-25T01:05:29.189 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-25T01:05:29.190 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-25T01:05:29.190 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-04-25T01:05:29.190 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-04-25T01:05:29.190 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-25T01:05:29.190 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-25T01:05:29.190 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-25T01:05:29.190 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-04-25T01:05:29.190 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-25T01:05:29.190 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-25T01:05:29.190 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-25T01:05:29.190 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-25T01:05:29.190 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-25T01:05:29.190 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-25T01:05:29.190 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-04-25T01:05:29.191 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-25T01:05:29.191 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-04-25T01:05:29.191 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-25T01:05:29.191 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-25T01:05:29.191 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-04-25T01:05:29.191 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-25T01:05:29.191 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-25T01:05:29.191 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-04-25T01:05:29.191 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-25T01:05:29.191 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-04-25T01:05:29.191 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-25T01:05:29.191 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-04-25T01:05:29.191 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-25T01:05:29.191 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-25T01:05:29.191 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-04-25T01:05:29.192 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-25T01:05:29.192 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-25T01:05:29.192 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-25T01:05:29.192 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-25T01:05:29.192 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-25T01:05:29.192 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-04-25T01:05:29.192 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-25T01:05:29.192 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-04-25T01:05:29.192 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-25T01:05:29.192 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-25T01:05:29.192 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-25T01:05:29.192 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-25T01:05:29.192 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-04-25T01:05:29.192 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-25T01:05:29.192 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-04-25T01:05:29.193 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-25T01:05:29.193 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-04-25T01:05:29.193 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-25T01:05:29.193 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-04-25T01:05:29.193 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-25T01:05:29.193 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-25T01:05:29.193 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-04-25T01:05:29.193 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-25T01:05:29.193 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-25T01:05:29.193 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-25T01:05:29.193 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-25T01:05:29.193 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-04-25T01:05:29.193 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-25T01:05:29.193 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-25T01:05:29.194 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-25T01:05:29.194 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-04-25T01:05:29.194 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-25T01:05:29.194 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-25T01:05:29.194 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-25T01:05:29.194 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-25T01:05:29.194 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-04-25T01:05:29.194 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-25T01:05:29.194 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-04-25T01:05:29.194 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-25T01:05:29.194 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-04-25T01:05:29.194 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-25T01:05:29.194 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-25T01:05:29.194 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-04-25T01:05:29.195 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-25T01:05:29.195 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-25T01:05:29.195 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-04-25T01:05:29.195 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-25T01:05:29.195 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-04-25T01:05:29.195 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-04-25T01:05:29.195 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-25T01:05:29.195 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-04-25T01:05:29.195 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-25T01:05:29.195 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-04-25T01:05:29.195 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-04-25T01:05:29.195 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-25T01:05:29.195 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-04-25T01:05:29.195 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-04-25T01:05:29.196 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-04-25T01:05:29.196 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-04-25T01:05:29.196 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-25T01:05:29.196 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-25T01:05:29.196 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-25T01:05:29.196 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-04-25T01:05:29.196 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-04-25T01:05:29.196 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-04-25T01:05:29.196 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-04-25T01:05:29.196 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-04-25T01:05:29.196 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-25T01:05:29.196 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-25T01:05:29.196 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-04-25T01:05:29.196 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-04-25T01:05:29.196 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-25T01:05:29.196 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-25T01:05:29.197 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-25T01:05:29.197 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-04-25T01:05:29.197 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-04-25T01:05:29.197 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-04-25T01:05:29.197 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-04-25T01:05:29.197 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-04-25T01:05:29.197 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-04-25T01:05:29.197 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-04-25T01:05:29.197 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-04-25T01:05:29.197 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-04-25T01:05:29.197 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-04-25T01:05:29.197 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-04-25T01:05:29.197 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-04-25T01:05:29.197 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-04-25T01:05:29.197 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-04-25T01:05:29.198 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-04-25T01:05:29.198 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-04-25T01:05:29.198 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-04-25T01:05:29.198 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-04-25T01:05:29.198 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-25T01:05:29.198 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-04-25T01:05:29.198 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-25T01:05:29.198 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-25T01:05:29.198 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-25T01:05:29.198 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-25T01:05:29.198 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-04-25T01:05:29.198 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-04-25T01:05:29.198 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-04-25T01:05:29.198 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-04-25T01:05:29.199 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-04-25T01:05:29.199 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-04-25T01:05:29.199 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-04-25T01:05:29.199 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-04-25T01:05:29.199 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-04-25T01:05:29.199 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-04-25T01:05:29.199 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-25T01:05:29.199 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-25T01:05:29.199 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-25T01:05:29.199 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-04-25T01:05:29.199 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-25T01:05:29.199 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-04-25T01:05:29.199 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-04-25T01:05:29.199 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-04-25T01:05:29.199 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-04-25T01:05:29.200 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-25T01:05:29.200 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-04-25T01:05:29.200 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-04-25T01:05:29.200 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-04-25T01:05:29.200 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-25T01:05:29.200 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-04-25T01:05:29.200 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-04-25T01:05:29.200 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-04-25T01:05:29.200 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-04-25T01:05:29.200 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-04-25T01:05:29.200 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-04-25T01:05:29.200 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-04-25T01:05:29.200 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-04-25T01:05:29.200 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-04-25T01:05:29.201 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-04-25T01:05:29.201 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-04-25T01:05:29.201 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-04-25T01:05:29.201 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-04-25T01:05:29.201 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-04-25T01:05:29.201 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-25T01:05:29.201 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-04-25T01:05:29.201 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-25T01:05:29.201 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-04-25T01:05:29.201 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-04-25T01:05:29.201 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-04-25T01:05:29.201 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-04-25T01:05:29.201 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-25T01:05:29.201 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-04-25T01:05:29.202 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-04-25T01:05:29.202 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-25T01:05:29.202 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-04-25T01:05:29.202 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-04-25T01:05:29.202 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-25T01:05:29.202 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-04-25T01:05:29.202 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-25T01:05:29.202 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-25T01:05:29.202 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-25T01:05:29.202 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-25T01:05:29.202 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-25T01:05:29.202 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-25T01:05:29.202 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-25T01:05:29.202 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-25T01:05:29.203 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-25T01:05:29.203 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-04-25T01:05:29.203 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-04-25T01:05:29.203 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-25T01:05:29.203 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-04-25T01:05:29.203 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-25T01:05:29.203 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-25T01:05:29.203 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-25T01:05:29.203 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-25T01:05:29.203 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-25T01:05:29.203 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-25T01:05:29.203 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-25T01:05:29.203 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-25T01:05:29.203 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-25T01:05:29.203 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-04-25T01:05:29.204 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-04-25T01:05:29.204 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-25T01:05:29.204 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-25T01:05:29.204 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-04-25T01:05:29.204 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-04-25T01:05:29.204 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-25T01:05:29.204 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-25T01:05:29.204 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-04-25T01:05:29.204 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-04-25T01:05:29.204 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-04-25T01:05:29.204 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-04-25T01:05:29.204 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-04-25T01:05:29.204 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-04-25T01:05:29.204 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-04-25T01:05:29.204 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-04-25T01:05:29.205 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-25T01:05:29.205 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-04-25T01:05:29.205 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-04-25T01:05:29.205 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-25T01:05:29.205 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-25T01:05:29.205 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-25T01:05:29.205 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-25T01:05:29.205 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-25T01:05:29.205 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-25T01:05:29.205 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-25T01:05:29.205 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-25T01:05:29.205 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-25T01:05:29.205 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-25T01:05:29.205 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-04-25T01:05:29.206 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-25T01:05:29.206 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-25T01:05:29.206 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-25T01:05:29.206 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-25T01:05:29.206 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-25T01:05:29.206 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-25T01:05:29.206 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-04-25T01:05:29.206 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-25T01:05:29.206 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-25T01:05:29.206 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-25T01:05:29.206 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-25T01:05:29.206 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-25T01:05:29.206 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-25T01:05:29.206 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-25T01:05:29.207 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-25T01:05:29.207 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-25T01:05:29.207 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-25T01:05:29.207 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-25T01:05:29.207 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-25T01:05:29.207 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-04-25T01:05:29.207 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-04-25T01:05:29.207 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-04-25T01:05:29.207 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-04-25T01:05:29.207 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-04-25T01:05:29.207 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-04-25T01:05:29.207 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-04-25T01:05:29.207 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-04-25T01:05:29.207 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-25T01:05:29.207 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-04-25T01:05:29.207 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-04-25T01:05:29.208 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-25T01:05:29.208 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-25T01:05:29.208 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-04-25T01:05:29.208 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-04-25T01:05:29.208 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-25T01:05:29.208 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-04-25T01:05:29.208 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-04-25T01:05:29.208 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-25T01:05:29.208 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-04-25T01:05:29.208 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-04-25T01:05:29.208 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-04-25T01:05:29.208 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-25T01:05:29.208 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-04-25T01:05:29.208 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-25T01:05:29.208 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-04-25T01:05:29.209 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-04-25T01:05:29.209 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-25T01:05:29.209 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-04-25T01:05:29.209 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-25T01:05:29.209 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-25T01:05:29.209 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-25T01:05:29.209 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-25T01:05:29.209 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-25T01:05:29.209 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-04-25T01:05:29.209 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-04-25T01:05:29.209 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-25T01:05:29.209 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-25T01:05:29.209 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-25T01:05:29.209 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-25T01:05:29.210 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-04-25T01:05:29.210 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-04-25T01:05:29.210 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-25T01:05:29.210 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-04-25T01:05:29.210 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-04-25T01:05:29.210 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-25T01:05:29.210 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-25T01:05:29.210 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-04-25T01:05:29.210 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-04-25T01:05:29.210 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-04-25T01:05:29.210 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-04-25T01:05:29.210 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-04-25T01:05:29.210 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-04-25T01:05:29.210 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-04-25T01:05:29.211 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-04-25T01:05:29.211 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-04-25T01:05:29.211 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-04-25T01:05:29.211 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-25T01:05:29.211 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-25T01:05:29.211 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-04-25T01:05:29.211 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-04-25T01:05:29.211 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-04-25T01:05:29.211 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-25T01:05:29.211 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-04-25T01:05:29.211 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-04-25T01:05:29.211 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-04-25T01:05:29.211 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-25T01:05:29.211 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-04-25T01:05:29.212 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-04-25T01:05:29.212 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-25T01:05:29.212 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-25T01:05:29.212 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-25T01:05:29.212 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-04-25T01:05:29.212 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-25T01:05:29.212 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-04-25T01:05:29.212 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-25T01:05:29.212 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-04-25T01:05:29.212 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-04-25T01:05:29.212 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-25T01:05:29.212 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-04-25T01:05:29.212 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-04-25T01:05:29.212 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-04-25T01:05:29.212 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-25T01:05:29.213 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-25T01:05:29.213 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-04-25T01:05:29.213 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-25T01:05:29.213 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-04-25T01:05:29.213 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-04-25T01:05:29.213 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-25T01:05:29.213 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-25T01:05:29.213 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-25T01:05:29.213 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-25T01:05:29.213 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-25T01:05:29.213 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-04-25T01:05:29.213 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-04-25T01:05:29.213 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-04-25T01:05:29.213 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-04-25T01:05:29.213 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-25T01:05:29.214 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-04-25T01:05:29.214 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-25T01:05:29.214 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-04-25T01:05:29.214 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-04-25T01:05:29.214 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-04-25T01:05:29.214 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-04-25T01:05:29.214 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-04-25T01:05:29.214 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-25T01:05:29.214 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-04-25T01:05:29.214 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-25T01:05:29.214 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-04-25T01:05:29.214 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-04-25T01:05:29.214 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-04-25T01:05:29.214 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-04-25T01:05:29.215 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-04-25T01:05:29.215 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-04-25T01:05:29.215 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-25T01:05:29.215 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-04-25T01:05:29.215 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-04-25T01:05:29.215 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-04-25T01:05:29.215 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-25T01:05:29.215 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-25T01:05:29.215 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-25T01:05:29.215 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-25T01:05:29.215 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-25T01:05:29.215 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-25T01:05:29.215 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-25T01:05:29.215 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-25T01:05:29.216 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-25T01:05:29.216 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-25T01:05:29.216 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-25T01:05:29.216 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-25T01:05:29.216 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-25T01:05:29.216 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-25T01:05:29.216 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-25T01:05:29.216 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-25T01:05:29.216 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-25T01:05:29.216 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-25T01:05:29.216 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-04-25T01:05:29.216 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-04-25T01:05:29.216 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-25T01:05:29.216 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-04-25T01:05:29.217 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-04-25T01:05:29.217 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-04-25T01:05:29.217 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-04-25T01:05:29.217 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-04-25T01:05:29.217 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-25T01:05:29.217 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-04-25T01:05:29.217 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-04-25T01:05:29.217 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-25T01:05:29.217 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-04-25T01:05:29.217 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-04-25T01:05:29.217 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-25T01:05:29.217 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-04-25T01:05:29.217 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-25T01:05:29.218 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-04-25T01:05:29.218 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-04-25T01:05:29.218 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-04-25T01:05:29.218 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-04-25T01:05:29.218 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-04-25T01:05:29.218 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-04-25T01:05:29.218 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-04-25T01:05:29.218 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-04-25T01:05:29.218 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-04-25T01:05:29.218 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-25T01:05:29.218 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-04-25T01:05:29.218 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-04-25T01:05:29.218 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-04-25T01:05:29.218 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-25T01:05:29.219 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-04-25T01:05:29.219 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-04-25T01:05:29.219 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-04-25T01:05:29.219 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-04-25T01:05:29.219 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-04-25T01:05:29.219 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-04-25T01:05:29.219 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-04-25T01:05:29.219 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-04-25T01:05:29.219 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-04-25T01:05:29.219 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-04-25T01:05:29.219 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-04-25T01:05:29.219 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-25T01:05:29.219 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-04-25T01:05:29.219 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-04-25T01:05:29.219 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-04-25T01:05:29.220 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-25T01:05:29.220 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-25T01:05:29.220 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-04-25T01:05:29.220 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-04-25T01:05:29.220 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-25T01:05:29.220 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-04-25T01:05:29.220 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-04-25T01:05:29.220 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-04-25T01:05:29.220 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-04-25T01:05:29.220 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-04-25T01:05:29.220 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-04-25T01:05:29.220 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-04-25T01:05:29.220 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-25T01:05:29.220 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-04-25T01:05:29.220 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-04-25T01:05:29.221 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-04-25T01:05:29.221 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-25T01:05:29.221 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-25T01:05:29.221 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-04-25T01:05:29.221 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-25T01:05:29.221 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-25T01:05:29.221 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-25T01:05:29.221 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-04-25T01:05:29.221 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-04-25T01:05:29.221 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-04-25T01:05:29.221 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-04-25T01:05:29.221 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-25T01:05:29.221 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-04-25T01:05:29.221 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-25T01:05:29.221 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-25T01:05:29.222 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-25T01:05:29.222 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-04-25T01:05:29.222 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-04-25T01:05:29.222 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-04-25T01:05:29.222 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-04-25T01:05:29.222 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-04-25T01:05:29.222 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-04-25T01:05:29.222 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-04-25T01:05:29.222 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-04-25T01:05:29.222 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-04-25T01:05:29.222 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-04-25T01:05:29.222 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-04-25T01:05:29.222 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-04-25T01:05:29.222 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-04-25T01:05:29.223 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-04-25T01:05:29.223 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-25T01:05:29.223 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-04-25T01:05:29.223 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-04-25T01:05:29.223 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-25T01:05:29.223 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-25T01:05:29.223 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-04-25T01:05:29.223 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-25T01:05:29.223 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-04-25T01:05:29.223 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-04-25T01:05:29.223 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-25T01:05:29.223 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-04-25T01:05:29.223 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-04-25T01:05:29.223 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-25T01:05:29.223 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-25T01:05:29.224 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-25T01:05:29.224 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-25T01:05:29.224 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-25T01:05:29.224 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-25T01:05:29.224 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-25T01:05:29.224 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-04-25T01:05:29.224 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-25T01:05:29.224 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-25T01:05:29.224 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-25T01:05:29.224 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-25T01:05:29.224 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-25T01:05:29.224 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-04-25T01:05:29.224 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-04-25T01:05:29.224 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-25T01:05:29.224 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-25T01:05:29.225 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-25T01:05:29.225 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-04-25T01:05:29.225 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-04-25T01:05:29.225 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-25T01:05:29.225 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-25T01:05:29.225 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-04-25T01:05:29.225 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-25T01:05:29.225 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-04-25T01:05:29.225 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-04-25T01:05:29.225 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-25T01:05:29.225 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-04-25T01:05:29.225 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-04-25T01:05:29.225 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-04-25T01:05:29.225 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-04-25T01:05:29.225 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-04-25T01:05:29.226 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-04-25T01:05:29.226 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-25T01:05:29.226 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-04-25T01:05:29.226 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-04-25T01:05:29.226 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-04-25T01:05:29.226 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-04-25T01:05:29.226 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-04-25T01:05:29.226 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-25T01:05:29.226 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-25T01:05:29.226 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-04-25T01:05:29.226 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-04-25T01:05:29.226 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-04-25T01:05:29.226 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-04-25T01:05:29.226 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-04-25T01:05:29.227 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-04-25T01:05:29.227 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-25T01:05:29.227 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-25T01:05:29.227 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-25T01:05:29.227 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-25T01:05:29.227 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-25T01:05:29.227 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-25T01:05:29.227 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-25T01:05:29.227 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-25T01:05:29.227 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-25T01:05:29.227 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-25T01:05:29.227 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-25T01:05:29.227 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-04-25T01:05:29.227 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-04-25T01:05:29.227 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-25T01:05:29.228 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-25T01:05:29.228 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-25T01:05:29.228 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-04-25T01:05:29.228 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-04-25T01:05:29.228 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-25T01:05:29.228 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-25T01:05:29.228 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-04-25T01:05:29.228 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-04-25T01:05:29.228 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-25T01:05:29.228 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-25T01:05:29.228 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-04-25T01:05:29.228 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-04-25T01:05:29.228 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-04-25T01:05:29.228 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-25T01:05:29.229 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-25T01:05:29.229 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-25T01:05:29.229 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-25T01:05:29.229 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-04-25T01:05:29.229 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-25T01:05:29.229 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-04-25T01:05:29.229 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-25T01:05:29.229 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-25T01:05:29.229 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-25T01:05:29.229 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-04-25T01:05:29.229 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-04-25T01:05:29.229 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-04-25T01:05:29.229 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-25T01:05:29.229 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-04-25T01:05:29.230 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-04-25T01:05:29.230 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-04-25T01:05:29.230 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-04-25T01:05:29.230 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-25T01:05:29.230 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-04-25T01:05:29.230 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-04-25T01:05:29.230 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-25T01:05:29.230 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-04-25T01:05:29.230 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-04-25T01:05:29.230 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-04-25T01:05:29.230 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-04-25T01:05:29.230 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-04-25T01:05:29.230 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-04-25T01:05:29.230 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-04-25T01:05:29.231 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-04-25T01:05:29.231 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-04-25T01:05:29.231 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-04-25T01:05:29.231 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-04-25T01:05:29.231 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-04-25T01:05:29.231 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-04-25T01:05:29.231 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-04-25T01:05:29.231 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-04-25T01:05:29.231 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-04-25T01:05:29.231 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-04-25T01:05:29.231 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-04-25T01:05:29.231 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-04-25T01:05:29.231 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-04-25T01:05:29.231 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-04-25T01:05:29.231 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-25T01:05:29.232 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-04-25T01:05:29.232 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-04-25T01:05:29.232 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-04-25T01:05:29.232 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-04-25T01:05:29.232 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-04-25T01:05:29.232 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-04-25T01:05:29.232 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-04-25T01:05:29.232 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-04-25T01:05:29.232 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-04-25T01:05:29.232 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-04-25T01:05:29.232 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-04-25T01:05:29.232 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-04-25T01:05:29.232 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-04-25T01:05:29.232 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-04-25T01:05:29.232 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-04-25T01:05:29.233 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-04-25T01:05:29.233 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-04-25T01:05:29.233 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-04-25T01:05:29.233 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-04-25T01:05:29.233 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-04-25T01:05:29.233 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-04-25T01:05:29.233 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-04-25T01:05:29.233 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-04-25T01:05:29.233 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-04-25T01:05:29.233 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-04-25T01:05:29.233 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-04-25T01:05:29.233 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-04-25T01:05:29.233 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-25T01:05:29.233 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-04-25T01:05:29.234 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-25T01:05:29.234 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-04-25T01:05:29.234 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-25T01:05:29.234 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-04-25T01:05:29.234 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-04-25T01:05:29.234 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-04-25T01:05:29.234 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-04-25T01:05:29.234 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-04-25T01:05:29.234 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-04-25T01:05:29.234 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-04-25T01:05:29.234 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-04-25T01:05:29.234 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-04-25T01:05:29.234 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-04-25T01:05:29.234 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-04-25T01:05:29.235 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-04-25T01:05:29.235 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-04-25T01:05:29.235 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-04-25T01:05:29.235 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-04-25T01:05:29.235 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-04-25T01:05:29.235 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-04-25T01:05:29.235 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-04-25T01:05:29.235 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-04-25T01:05:29.235 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-04-25T01:05:29.235 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-04-25T01:05:29.235 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-04-25T01:05:29.235 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-04-25T01:05:29.235 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-04-25T01:05:29.235 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-04-25T01:05:29.235 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-04-25T01:05:29.236 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-04-25T01:05:29.236 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-04-25T01:05:29.236 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-04-25T01:05:29.236 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-04-25T01:05:29.236 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-04-25T01:05:29.236 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-04-25T01:05:29.236 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-04-25T01:05:29.236 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-04-25T01:05:29.236 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-04-25T01:05:29.236 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-04-25T01:05:29.236 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-04-25T01:05:29.236 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-04-25T01:05:29.236 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-04-25T01:05:29.236 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-04-25T01:05:29.237 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-04-25T01:05:29.237 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-04-25T01:05:29.237 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-04-25T01:05:29.237 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-04-25T01:05:29.237 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-04-25T01:05:29.237 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-04-25T01:05:29.237 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-04-25T01:05:29.237 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-04-25T01:05:29.237 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-04-25T01:05:29.237 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-04-25T01:05:29.237 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-04-25T01:05:29.237 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-04-25T01:05:29.237 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-04-25T01:05:29.237 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-04-25T01:05:29.238 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-04-25T01:05:29.238 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-04-25T01:05:29.238 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-04-25T01:05:29.238 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-04-25T01:05:29.238 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-04-25T01:05:29.238 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-25T01:05:29.238 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-04-25T01:05:29.238 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-04-25T01:05:29.238 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-04-25T01:05:29.238 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-25T01:05:29.238 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-04-25T01:05:29.238 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-25T01:05:29.238 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-04-25T01:05:29.239 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-25T01:05:29.239 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-04-25T01:05:29.239 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-25T01:05:29.239 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-25T01:05:29.239 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-25T01:05:29.239 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-04-25T01:05:29.239 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-04-25T01:05:29.239 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-04-25T01:05:29.239 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-04-25T01:05:29.239 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-04-25T01:05:29.239 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-04-25T01:05:29.239 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-04-25T01:05:29.239 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-04-25T01:05:29.239 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-04-25T01:05:29.239 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-04-25T01:05:29.240 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-04-25T01:05:29.240 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-04-25T01:05:29.240 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-04-25T01:05:29.240 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-04-25T01:05:29.240 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-04-25T01:05:29.240 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-04-25T01:05:29.240 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-04-25T01:05:29.240 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-04-25T01:05:29.240 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-04-25T01:05:29.240 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-04-25T01:05:29.240 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-04-25T01:05:29.240 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-04-25T01:05:29.240 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-04-25T01:05:29.240 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-04-25T01:05:29.240 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-04-25T01:05:29.241 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-04-25T01:05:29.241 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-04-25T01:05:29.241 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-04-25T01:05:29.241 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-04-25T01:05:29.241 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-25T01:05:29.241 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-04-25T01:05:29.241 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-04-25T01:05:29.241 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-04-25T01:05:29.241 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-04-25T01:05:29.241 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-04-25T01:05:29.241 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-04-25T01:05:29.241 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-04-25T01:05:29.241 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-04-25T01:05:29.241 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-04-25T01:05:29.241 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-04-25T01:05:29.242 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-04-25T01:05:29.242 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-04-25T01:05:29.242 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-04-25T01:05:29.242 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-04-25T01:05:29.242 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-04-25T01:05:29.242 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-04-25T01:05:29.242 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-04-25T01:05:29.242 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-25T01:05:29.242 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-04-25T01:05:29.242 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-04-25T01:05:29.242 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-04-25T01:05:29.242 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-04-25T01:05:29.242 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-04-25T01:05:29.243 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-04-25T01:05:29.243 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-04-25T01:05:29.243 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-04-25T01:05:29.243 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-04-25T01:05:29.243 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-04-25T01:05:29.243 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-04-25T01:05:29.243 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-04-25T01:05:29.243 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-04-25T01:05:29.243 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-25T01:05:29.243 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-04-25T01:05:29.243 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-04-25T01:05:29.243 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-04-25T01:05:29.243 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-04-25T01:05:29.243 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-04-25T01:05:29.244 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-04-25T01:05:29.244 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-04-25T01:05:29.244 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-04-25T01:05:29.244 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-04-25T01:05:29.244 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-04-25T01:05:29.244 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-04-25T01:05:29.244 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-04-25T01:05:29.244 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-04-25T01:05:29.244 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-04-25T01:05:29.244 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-04-25T01:05:29.244 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-04-25T01:05:29.244 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-04-25T01:05:29.244 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-04-25T01:05:29.244 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-04-25T01:05:29.244 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-04-25T01:05:29.245 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-04-25T01:05:29.245 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-04-25T01:05:29.245 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-25T01:05:29.245 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-25T01:05:29.245 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-25T01:05:29.245 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-25T01:05:29.245 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-25T01:05:29.245 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-25T01:05:29.245 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-25T01:05:29.245 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-25T01:05:29.245 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-25T01:05:29.245 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-25T01:05:29.245 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-25T01:05:29.245 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-25T01:05:29.246 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-25T01:05:29.246 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-25T01:05:29.246 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-25T01:05:29.246 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-25T01:05:29.246 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-25T01:05:29.246 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-25T01:05:29.246 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-25T01:05:29.246 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-25T01:05:29.246 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-04-25T01:05:29.246 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-25T01:05:29.246 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-25T01:05:29.246 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-25T01:05:29.246 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-04-25T01:05:29.246 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-04-25T01:05:29.247 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-04-25T01:05:29.247 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-04-25T01:05:29.247 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-04-25T01:05:29.247 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-04-25T01:05:29.247 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-04-25T01:05:29.247 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-04-25T01:05:29.247 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-04-25T01:05:29.247 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-04-25T01:05:29.247 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-04-25T01:05:29.247 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-04-25T01:05:29.247 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-04-25T01:05:29.247 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-04-25T01:05:29.247 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-04-25T01:05:29.247 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-04-25T01:05:29.247 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-04-25T01:05:29.248 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-04-25T01:05:29.248 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-04-25T01:05:29.248 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-04-25T01:05:29.248 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-25T01:05:29.248 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-04-25T01:05:29.248 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-25T01:05:29.248 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-04-25T01:05:29.248 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-04-25T01:05:29.248 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-04-25T01:05:29.248 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-04-25T01:05:29.248 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-04-25T01:05:29.248 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-04-25T01:05:29.248 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-04-25T01:05:29.248 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-04-25T01:05:29.248 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-04-25T01:05:29.249 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-04-25T01:05:29.249 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-04-25T01:05:29.249 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-04-25T01:05:29.249 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-25T01:05:29.249 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-25T01:05:29.249 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-25T01:05:29.249 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-25T01:05:29.249 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-25T01:05:29.249 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-25T01:05:29.249 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-25T01:05:29.249 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-25T01:05:29.249 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-25T01:05:29.249 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-25T01:05:29.249 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-25T01:05:29.249 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-25T01:05:29.250 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-25T01:05:29.250 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-25T01:05:29.250 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-25T01:05:29.250 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-25T01:05:29.250 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-25T01:05:29.250 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-25T01:05:29.250 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-25T01:05:29.250 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-25T01:05:29.250 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-25T01:05:29.250 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-04-25T01:05:29.250 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-04-25T01:05:29.250 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-04-25T01:05:29.250 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-04-25T01:05:29.250 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-04-25T01:05:29.251 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-04-25T01:05:29.251 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-04-25T01:05:29.251 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-04-25T01:05:29.251 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-04-25T01:05:29.251 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-04-25T01:05:29.251 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-04-25T01:05:29.251 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-04-25T01:05:29.251 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-04-25T01:05:29.251 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-04-25T01:05:29.251 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-04-25T01:05:29.251 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-04-25T01:05:29.254 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-25T01:05:29.254 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-04-25T01:05:29.254 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-04-25T01:05:29.254 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-04-25T01:05:29.254 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-04-25T01:05:29.254 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-04-25T01:05:29.254 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-25T01:05:29.254 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-04-25T01:05:29.254 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-04-25T01:05:29.254 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-04-25T01:05:29.254 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-25T01:05:29.254 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-25T01:05:29.255 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-25T01:05:29.255 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-25T01:05:29.255 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-25T01:05:29.255 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-25T01:05:29.255 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-25T01:05:29.255 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-25T01:05:29.255 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-25T01:05:29.255 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-25T01:05:29.255 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-25T01:05:29.255 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-25T01:05:29.255 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-25T01:05:29.255 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-25T01:05:29.255 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-25T01:05:29.256 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-25T01:05:29.256 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-25T01:05:29.256 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-25T01:05:29.256 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-25T01:05:29.256 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-25T01:05:29.256 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-25T01:05:29.256 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-25T01:05:29.256 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-25T01:05:29.256 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-25T01:05:29.256 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-04-25T01:05:29.256 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-04-25T01:05:29.256 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-25T01:05:29.256 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-04-25T01:05:29.256 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-04-25T01:05:29.256 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-04-25T01:05:29.257 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-04-25T01:05:29.257 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-25T01:05:29.257 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-04-25T01:05:29.257 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-04-25T01:05:29.257 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-04-25T01:05:29.257 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-04-25T01:05:29.257 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-04-25T01:05:29.257 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-04-25T01:05:29.257 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-04-25T01:05:29.257 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-04-25T01:05:29.257 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-04-25T01:05:29.257 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-04-25T01:05:29.257 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-04-25T01:05:29.257 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-04-25T01:05:29.257 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-04-25T01:05:29.258 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-04-25T01:05:29.258 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-25T01:05:29.258 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-25T01:05:29.258 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-25T01:05:29.258 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-25T01:05:29.258 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-25T01:05:29.258 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-25T01:05:29.258 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-25T01:05:29.258 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-25T01:05:29.258 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-25T01:05:29.258 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-25T01:05:29.258 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-25T01:05:29.258 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-25T01:05:29.258 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-25T01:05:29.259 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-25T01:05:29.259 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-25T01:05:29.259 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-04-25T01:05:29.259 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-25T01:05:29.259 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-25T01:05:29.259 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-25T01:05:29.259 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-25T01:05:29.259 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-25T01:05:29.259 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-25T01:05:29.259 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-25T01:05:29.259 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-25T01:05:29.259 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-25T01:05:29.259 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-25T01:05:29.259 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-25T01:05:29.259 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-25T01:05:29.260 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-25T01:05:29.260 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-25T01:05:29.260 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-04-25T01:05:29.260 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-04-25T01:05:29.260 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-25T01:05:29.260 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-25T01:05:29.260 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-25T01:05:29.260 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-25T01:05:29.260 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-25T01:05:29.260 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-25T01:05:29.260 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-25T01:05:29.260 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-25T01:05:29.260 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-25T01:05:29.260 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-25T01:05:29.261 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-25T01:05:29.261 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-25T01:05:29.261 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-25T01:05:29.261 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-04-25T01:05:29.261 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-25T01:05:29.261 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-04-25T01:05:29.261 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-04-25T01:05:29.261 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-25T01:05:29.261 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-25T01:05:29.261 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-04-25T01:05:29.261 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-04-25T01:05:29.261 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-25T01:05:29.261 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-25T01:05:29.261 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-04-25T01:05:29.262 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-04-25T01:05:29.262 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-25T01:05:29.262 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-04-25T01:05:29.262 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-25T01:05:29.262 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-25T01:05:29.262 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-04-25T01:05:29.262 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-04-25T01:05:29.262 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-04-25T01:05:29.262 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-25T01:05:29.262 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-04-25T01:05:29.262 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-25T01:05:29.262 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-04-25T01:05:29.262 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-04-25T01:05:29.262 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-04-25T01:05:29.263 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-04-25T01:05:29.263 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-04-25T01:05:29.263 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-04-25T01:05:29.263 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-04-25T01:05:29.263 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-04-25T01:05:29.263 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-04-25T01:05:29.263 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-04-25T01:05:29.263 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-04-25T01:05:29.263 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-04-25T01:05:29.263 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-04-25T01:05:29.263 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-04-25T01:05:29.263 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-25T01:05:29.263 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-25T01:05:29.263 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-25T01:05:29.263 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-04-25T01:05:29.264 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-04-25T01:05:29.264 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-04-25T01:05:29.264 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-04-25T01:05:29.264 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-25T01:05:29.264 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-04-25T01:05:29.264 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-04-25T01:05:29.264 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-25T01:05:29.264 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-04-25T01:05:29.264 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-04-25T01:05:29.264 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-25T01:05:29.264 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-04-25T01:05:29.264 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-25T01:05:29.264 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-04-25T01:05:29.264 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-25T01:05:29.264 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-25T01:05:29.265 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-04-25T01:05:29.265 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-04-25T01:05:29.265 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-04-25T01:05:29.265 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-04-25T01:05:29.265 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-04-25T01:05:29.265 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-04-25T01:05:29.265 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-04-25T01:05:29.265 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-04-25T01:05:29.265 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-04-25T01:05:29.265 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-04-25T01:05:29.265 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-04-25T01:05:29.265 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-04-25T01:05:29.265 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-04-25T01:05:29.265 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-04-25T01:05:29.265 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-04-25T01:05:29.266 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-04-25T01:05:29.266 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-04-25T01:05:29.266 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-04-25T01:05:29.266 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-04-25T01:05:29.266 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-04-25T01:05:29.266 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-04-25T01:05:29.266 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-04-25T01:05:29.266 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-04-25T01:05:29.266 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-04-25T01:05:29.266 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-04-25T01:05:29.266 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-04-25T01:05:29.266 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-04-25T01:05:29.266 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-04-25T01:05:29.266 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-04-25T01:05:29.267 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-04-25T01:05:29.267 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-04-25T01:05:29.267 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-04-25T01:05:29.267 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-04-25T01:05:29.267 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-04-25T01:05:29.267 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-04-25T01:05:29.267 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-04-25T01:05:29.267 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-04-25T01:05:29.267 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-04-25T01:05:29.267 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-04-25T01:05:29.267 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-04-25T01:05:29.267 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-04-25T01:05:29.267 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-04-25T01:05:29.267 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-04-25T01:05:29.267 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-04-25T01:05:29.268 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-04-25T01:05:29.268 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-04-25T01:05:29.268 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-04-25T01:05:29.268 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-04-25T01:05:29.268 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-04-25T01:05:29.268 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-04-25T01:05:29.268 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-04-25T01:05:29.268 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-04-25T01:05:29.268 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-04-25T01:05:29.268 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-04-25T01:05:29.268 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-04-25T01:05:29.268 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-04-25T01:05:29.268 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-04-25T01:05:29.268 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-04-25T01:05:29.269 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-04-25T01:05:29.269 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-04-25T01:05:29.269 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-04-25T01:05:29.269 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-04-25T01:05:29.269 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-04-25T01:05:29.269 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-04-25T01:05:29.269 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-04-25T01:05:29.269 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-04-25T01:05:29.269 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-04-25T01:05:29.269 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-04-25T01:05:29.269 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-04-25T01:05:29.269 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-04-25T01:05:29.269 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-04-25T01:05:29.269 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-04-25T01:05:29.270 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-04-25T01:05:29.270 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-04-25T01:05:29.270 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-04-25T01:05:29.270 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-04-25T01:05:29.270 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-04-25T01:05:29.270 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-04-25T01:05:29.270 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-04-25T01:05:29.270 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-04-25T01:05:29.270 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-04-25T01:05:29.270 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-04-25T01:05:29.270 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-04-25T01:05:29.270 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-04-25T01:05:29.270 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-04-25T01:05:29.270 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-04-25T01:05:29.270 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-04-25T01:05:29.271 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-04-25T01:05:29.271 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-04-25T01:05:29.271 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-04-25T01:05:29.271 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-04-25T01:05:29.271 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-04-25T01:05:29.271 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-04-25T01:05:29.271 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-04-25T01:05:29.271 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-04-25T01:05:29.271 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-04-25T01:05:29.271 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-04-25T01:05:29.271 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-04-25T01:05:29.271 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-04-25T01:05:29.271 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-04-25T01:05:29.271 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-04-25T01:05:29.271 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-04-25T01:05:29.272 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-04-25T01:05:29.272 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-04-25T01:05:29.272 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-04-25T01:05:29.272 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-04-25T01:05:29.272 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-04-25T01:05:29.272 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-04-25T01:05:29.272 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-04-25T01:05:29.272 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-04-25T01:05:29.272 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-04-25T01:05:29.272 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-04-25T01:05:29.272 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-04-25T01:05:29.272 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-04-25T01:05:29.272 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-04-25T01:05:29.272 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-25T01:05:29.272 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-25T01:05:29.273 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-04-25T01:05:29.273 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-04-25T01:05:29.273 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-25T01:05:29.273 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-04-25T01:05:29.273 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-04-25T01:05:29.273 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-04-25T01:05:29.273 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-04-25T01:05:29.273 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-04-25T01:05:29.273 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-04-25T01:05:29.273 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-25T01:05:29.273 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-25T01:05:29.273 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-04-25T01:05:29.273 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-25T01:05:29.273 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-25T01:05:29.274 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-04-25T01:05:29.274 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-25T01:05:29.274 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-04-25T01:05:29.274 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-04-25T01:05:29.274 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-04-25T01:05:29.274 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-25T01:05:29.274 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-04-25T01:05:29.274 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-04-25T01:05:29.274 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-04-25T01:05:29.274 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-04-25T01:05:29.274 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-04-25T01:05:29.274 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-04-25T01:05:29.274 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-04-25T01:05:29.274 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-04-25T01:05:29.275 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-04-25T01:05:29.275 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-04-25T01:05:29.275 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-04-25T01:05:29.275 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-04-25T01:05:29.275 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-04-25T01:05:29.275 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-04-25T01:05:29.275 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-04-25T01:05:29.275 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-25T01:05:29.278 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-04-25T01:05:29.279 DEBUG:teuthology.orchestra.run.smithi103:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-25T01:05:29.300 INFO:teuthology.orchestra.run.smithi103.stdout:postinstall scriptlet (using /bin/sh): 2024-04-25T01:05:29.301 INFO:teuthology.orchestra.run.smithi103.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-25T01:05:29.301 INFO:teuthology.orchestra.run.smithi103.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-25T01:05:29.301 INFO:teuthology.orchestra.run.smithi103.stdout:fi 2024-04-25T01:05:29.301 INFO:teuthology.orchestra.run.smithi103.stdout:for file in vmlinuz System.map config; do 2024-04-25T01:05:29.301 INFO:teuthology.orchestra.run.smithi103.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-04-25T01:05:29.301 INFO:teuthology.orchestra.run.smithi103.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-04-25T01:05:29.301 INFO:teuthology.orchestra.run.smithi103.stdout: fi 2024-04-25T01:05:29.301 INFO:teuthology.orchestra.run.smithi103.stdout:done 2024-04-25T01:05:29.301 INFO:teuthology.orchestra.run.smithi103.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-04-25T01:05:29.301 INFO:teuthology.orchestra.run.smithi103.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-04-25T01:05:29.301 INFO:teuthology.orchestra.run.smithi103.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-25T01:05:29.301 INFO:teuthology.orchestra.run.smithi103.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-25T01:05:29.301 INFO:teuthology.orchestra.run.smithi103.stdout:fi 2024-04-25T01:05:29.301 INFO:teuthology.orchestra.run.smithi103.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-04-25T01:05:29.302 INFO:teuthology.orchestra.run.smithi103.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-04-25T01:05:29.302 INFO:teuthology.orchestra.run.smithi103.stdout:fi 2024-04-25T01:05:29.302 INFO:teuthology.orchestra.run.smithi103.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-25T01:05:29.302 INFO:teuthology.orchestra.run.smithi103.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-25T01:05:29.302 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-04-25T01:05:29.302 INFO:teuthology.orchestra.run.smithi103.stdout:fi 2024-04-25T01:05:29.303 DEBUG:teuthology.orchestra.run.smithi103:> sudo rpm -qi grub2-tools 2024-04-25T01:05:29.374 INFO:teuthology.orchestra.run.smithi103.stdout:Name : grub2-tools 2024-04-25T01:05:29.374 INFO:teuthology.orchestra.run.smithi103.stdout:Epoch : 1 2024-04-25T01:05:29.374 INFO:teuthology.orchestra.run.smithi103.stdout:Version : 2.06 2024-04-25T01:05:29.374 INFO:teuthology.orchestra.run.smithi103.stdout:Release : 68.el9 2024-04-25T01:05:29.374 INFO:teuthology.orchestra.run.smithi103.stdout:Architecture: x86_64 2024-04-25T01:05:29.374 INFO:teuthology.orchestra.run.smithi103.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-04-25T01:05:29.374 INFO:teuthology.orchestra.run.smithi103.stdout:Group : Unspecified 2024-04-25T01:05:29.375 INFO:teuthology.orchestra.run.smithi103.stdout:Size : 8264254 2024-04-25T01:05:29.375 INFO:teuthology.orchestra.run.smithi103.stdout:License : GPLv3+ 2024-04-25T01:05:29.375 INFO:teuthology.orchestra.run.smithi103.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-25T01:05:29.375 INFO:teuthology.orchestra.run.smithi103.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-25T01:05:29.375 INFO:teuthology.orchestra.run.smithi103.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-25T01:05:29.375 INFO:teuthology.orchestra.run.smithi103.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-25T01:05:29.375 INFO:teuthology.orchestra.run.smithi103.stdout:Packager : builder@centos.org 2024-04-25T01:05:29.375 INFO:teuthology.orchestra.run.smithi103.stdout:Vendor : CentOS 2024-04-25T01:05:29.375 INFO:teuthology.orchestra.run.smithi103.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-25T01:05:29.375 INFO:teuthology.orchestra.run.smithi103.stdout:Summary : Support tools for GRUB. 2024-04-25T01:05:29.375 INFO:teuthology.orchestra.run.smithi103.stdout:Description : 2024-04-25T01:05:29.375 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:05:29.375 INFO:teuthology.orchestra.run.smithi103.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-25T01:05:29.375 INFO:teuthology.orchestra.run.smithi103.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-25T01:05:29.375 INFO:teuthology.orchestra.run.smithi103.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-25T01:05:29.375 INFO:teuthology.orchestra.run.smithi103.stdout:hardware devices. 2024-04-25T01:05:29.376 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:05:29.376 INFO:teuthology.orchestra.run.smithi103.stdout:This subpackage provides tools for support of all platforms. 2024-04-25T01:05:29.376 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-25T01:05:29.377 INFO:teuthology.task.kernel:Updating grub on smithi103 to boot 6.9.0-rc3-g806ec36b2112 2024-04-25T01:05:29.377 DEBUG:teuthology.orchestra.run.smithi103:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-25T01:05:29.872 INFO:teuthology.orchestra.run.smithi103.stderr:Generating grub configuration file ... 2024-04-25T01:05:31.293 INFO:teuthology.orchestra.run.smithi103.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-25T01:05:31.328 INFO:teuthology.orchestra.run.smithi103.stderr:done 2024-04-25T01:05:31.330 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2024-04-25T01:05:31.370 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.tbjLhMNGqT 2024-04-25T01:05:31.370 DEBUG:teuthology.orchestra.run.smithi103:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.tbjLhMNGqT 2024-04-25T01:05:31.435 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.tbjLhMNGqT 2024-04-25T01:05:31.584 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.tbjLhMNGqT is 6KB 2024-04-25T01:05:31.632 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.tbjLhMNGqT 2024-04-25T01:05:31.647 DEBUG:teuthology.orchestra.run.smithi103:> sudo /bin/ls /boot/loader/entries || true 2024-04-25T01:05:31.707 INFO:teuthology.orchestra.run.smithi103.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-04-25T01:05:31.708 INFO:teuthology.orchestra.run.smithi103.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-04-25T01:05:31.708 INFO:teuthology.orchestra.run.smithi103.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-04-25T01:05:31.708 INFO:teuthology.orchestra.run.smithi103.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112.conf 2024-04-25T01:05:31.709 DEBUG:teuthology.orchestra.run.smithi103:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112 2024-04-25T01:05:31.880 DEBUG:teuthology.orchestra.run.smithi103:> sudo shutdown -r now 2024-04-25T01:05:44.303 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-25T01:05:44.304 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-25T01:05:44.305 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:06:01.899 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-04-25T01:06:01.899 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-25T01:06:01.899 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:06:02.654 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-04-25T01:06:11.663 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-25T01:06:11.663 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:06:14.718 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-04-25T01:06:20.318 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-04-25T01:06:26.725 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-25T01:06:26.726 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:06:29.322 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-25T01:06:29.323 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:06:29.790 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-04-25T01:06:32.382 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-04-25T01:06:44.394 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-25T01:06:44.395 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:06:44.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-25T01:06:44.792 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:07:44.458 DEBUG:teuthology.orchestra.remote:timed out 2024-04-25T01:07:44.795 DEBUG:teuthology.orchestra.remote:timed out 2024-04-25T01:07:59.474 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-25T01:07:59.475 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:07:59.886 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-04-25T01:08:00.460 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-25T01:08:00.461 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "6.9.0-rc3-g806ec36b2112"... 2024-04-25T01:08:00.461 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-04-25T01:08:00.517 INFO:teuthology.orchestra.run.smithi103.stdout:6.9.0-rc3-g806ec36b2112 2024-04-25T01:08:00.518 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-04-25T01:08:00.518 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-25T01:08:00.518 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-25T01:08:01.519 DEBUG:teuthology.parallel:result is None 2024-04-25T01:08:02.796 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-25T01:08:02.796 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:08:03.182 DEBUG:teuthology.orchestra.run.smithi090:> true 2024-04-25T01:08:03.876 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-25T01:08:03.876 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-g806ec36b2112"... 2024-04-25T01:08:03.876 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2024-04-25T01:08:03.928 INFO:teuthology.orchestra.run.smithi090.stdout:6.9.0-rc3-g806ec36b2112 2024-04-25T01:08:03.928 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-04-25T01:08:03.928 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-25T01:08:03.928 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-25T01:08:04.929 DEBUG:teuthology.parallel:result is None 2024-04-25T01:08:04.930 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-25T01:08:04.940 INFO:teuthology.task.internal:Creating test directory... 2024-04-25T01:08:04.941 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T01:08:04.948 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T01:08:05.005 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-25T01:08:05.012 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-25T01:08:05.019 INFO:teuthology.task.internal:Creating archive directory... 2024-04-25T01:08:05.019 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T01:08:05.024 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T01:08:05.109 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-25T01:08:05.115 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-25T01:08:05.116 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T01:08:05.122 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T01:08:05.259 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:08:05.271 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:08:05.319 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:08:05.331 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:08:05.336 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-25T01:08:05.343 INFO:teuthology.task.internal:Configuring sudo... 2024-04-25T01:08:05.343 DEBUG:teuthology.orchestra.run.smithi090:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T01:08:05.348 DEBUG:teuthology.orchestra.run.smithi103:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T01:08:05.436 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-25T01:08:05.445 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-25T01:08:05.446 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T01:08:05.453 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T01:08:05.508 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:08:05.586 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:08:05.692 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:08:05.769 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:08:05.853 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:08:05.853 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T01:08:05.935 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:08:06.021 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:08:06.123 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:08:06.206 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:08:06.295 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:08:06.295 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T01:08:06.374 DEBUG:teuthology.orchestra.run.smithi090:> sudo service rsyslog restart 2024-04-25T01:08:06.382 DEBUG:teuthology.orchestra.run.smithi103:> sudo service rsyslog restart 2024-04-25T01:08:06.492 INFO:teuthology.orchestra.run.smithi090.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:08:06.537 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:08:06.968 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-25T01:08:06.976 INFO:teuthology.task.internal:Starting timer... 2024-04-25T01:08:06.976 INFO:teuthology.run_tasks:Running task pcp... 2024-04-25T01:08:06.985 INFO:teuthology.run_tasks:Running task selinux... 2024-04-25T01:08:06.993 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-25T01:08:06.993 DEBUG:teuthology.orchestra.run.smithi090:> sudo service auditd rotate 2024-04-25T01:08:07.110 INFO:teuthology.orchestra.run.smithi090.stdout:Rotating logs: 2024-04-25T01:08:07.115 DEBUG:teuthology.orchestra.run.smithi103:> sudo service auditd rotate 2024-04-25T01:08:07.231 INFO:teuthology.orchestra.run.smithi103.stdout:Rotating logs: 2024-04-25T01:08:07.235 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-25T01:08:07.235 DEBUG:teuthology.orchestra.run.smithi090:> /usr/sbin/getenforce 2024-04-25T01:08:07.296 INFO:teuthology.orchestra.run.smithi090.stdout:Permissive 2024-04-25T01:08:07.296 DEBUG:teuthology.orchestra.run.smithi103:> /usr/sbin/getenforce 2024-04-25T01:08:07.347 INFO:teuthology.orchestra.run.smithi103.stdout:Permissive 2024-04-25T01:08:07.347 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi090.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi103.front.sepia.ceph.com': 'permissive'} 2024-04-25T01:08:07.348 DEBUG:teuthology.orchestra.run.smithi090:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:08:07.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:08:07.425 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:08:07.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:08:07.504 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-25T01:08:07.504 DEBUG:teuthology.orchestra.run.smithi090:> sudo /usr/sbin/setenforce permissive 2024-04-25T01:08:07.581 DEBUG:teuthology.orchestra.run.smithi103:> sudo /usr/sbin/setenforce permissive 2024-04-25T01:08:07.655 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-25T01:08:07.665 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-25T01:08:07.682 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-25T01:08:07.692 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi090.front.sepia.ceph.com,smithi103.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T01:13:13.250 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi090.front.sepia.ceph.com'), Remote(name='ubuntu@smithi103.front.sepia.ceph.com')] 2024-04-25T01:13:13.251 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-25T01:13:13.252 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:13:13.361 DEBUG:teuthology.orchestra.run.smithi090:> true 2024-04-25T01:13:13.519 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-25T01:13:13.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-25T01:13:13.519 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:13:13.627 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-04-25T01:13:13.769 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-25T01:13:13.770 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T01:13:13.781 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T01:13:13.781 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T01:13:13.782 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:13:13.789 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T01:13:13.789 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:13:13.874 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T01:13:13.876 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T01:13:13.916 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T01:13:13.917 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T01:13:13.985 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: ntpd: command not found 2024-04-25T01:13:13.997 INFO:teuthology.orchestra.run.smithi103.stderr:sudo: ntpd: command not found 2024-04-25T01:13:14.017 INFO:teuthology.orchestra.run.smithi090.stdout:506 Cannot talk to daemon 2024-04-25T01:13:14.027 INFO:teuthology.orchestra.run.smithi103.stdout:506 Cannot talk to daemon 2024-04-25T01:13:14.061 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T01:13:14.064 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T01:13:14.103 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T01:13:14.104 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T01:13:14.237 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:13:14.243 INFO:teuthology.orchestra.run.smithi090.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:13:14.246 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:13:14.246 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-04-25T01:13:14.246 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:13:14.246 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:13:14.246 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:13:14.246 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:13:14.252 INFO:teuthology.orchestra.run.smithi090.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:13:14.252 INFO:teuthology.orchestra.run.smithi090.stdout:=============================================================================== 2024-04-25T01:13:14.252 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:13:14.252 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:13:14.252 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:13:14.252 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:13:14.253 INFO:teuthology.run_tasks:Running task install... 2024-04-25T01:13:14.262 DEBUG:teuthology.task.install:project ceph 2024-04-25T01:13:14.262 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}} 2024-04-25T01:13:14.262 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'} 2024-04-25T01:13:14.262 INFO:teuthology.task.install:Using flavor: default 2024-04-25T01:13:14.268 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-25T01:13:14.268 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-25T01:13:14.268 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'tag': None, 'wait_for_package': False} 2024-04-25T01:13:14.269 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:13:14.271 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'tag': None, 'wait_for_package': False} 2024-04-25T01:13:14.271 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:13:14.420 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/ 2024-04-25T01:13:14.421 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2445.gd2c45d97 2024-04-25T01:13:14.492 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/ 2024-04-25T01:13:14.493 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2445.gd2c45d97 2024-04-25T01:13:14.529 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T01:13:14.530 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:13:14.530 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T01:13:14.598 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T01:13:14.599 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:13:14.599 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T01:13:14.623 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T01:13:14.623 DEBUG:teuthology.orchestra.run.smithi103:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2c45d977142e024a59b68a8f1dce8f0b9840551/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T01:13:14.691 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T01:13:14.692 DEBUG:teuthology.orchestra.run.smithi090:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2c45d977142e024a59b68a8f1dce8f0b9840551/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T01:13:14.749 DEBUG:teuthology.orchestra.run.smithi103:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T01:13:14.822 DEBUG:teuthology.orchestra.run.smithi090:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T01:13:14.855 DEBUG:teuthology.orchestra.run.smithi103:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T01:13:14.930 INFO:teuthology.orchestra.run.smithi103.stdout:check_obsoletes = 1 2024-04-25T01:13:14.933 DEBUG:teuthology.orchestra.run.smithi090:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T01:13:14.934 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum clean all 2024-04-25T01:13:15.010 INFO:teuthology.orchestra.run.smithi090.stdout:check_obsoletes = 1 2024-04-25T01:13:15.014 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum clean all 2024-04-25T01:13:15.256 INFO:teuthology.orchestra.run.smithi103.stdout:48 files removed 2024-04-25T01:13:15.295 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T01:13:15.327 INFO:teuthology.orchestra.run.smithi090.stdout:48 files removed 2024-04-25T01:13:15.365 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T01:13:16.149 INFO:teuthology.orchestra.run.smithi103.stdout:ceph packages for x86_64 198 kB/s | 83 kB 00:00 2024-04-25T01:13:16.293 INFO:teuthology.orchestra.run.smithi090.stdout:ceph packages for x86_64 169 kB/s | 83 kB 00:00 2024-04-25T01:13:16.742 INFO:teuthology.orchestra.run.smithi103.stdout:ceph noarch packages 23 kB/s | 11 kB 00:00 2024-04-25T01:13:16.910 INFO:teuthology.orchestra.run.smithi090.stdout:ceph noarch packages 22 kB/s | 11 kB 00:00 2024-04-25T01:13:17.276 INFO:teuthology.orchestra.run.smithi103.stdout:ceph source packages 4.2 kB/s | 1.9 kB 00:00 2024-04-25T01:13:17.452 INFO:teuthology.orchestra.run.smithi090.stdout:ceph source packages 4.1 kB/s | 1.9 kB 00:00 2024-04-25T01:13:18.358 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - BaseOS 9.8 MB/s | 8.0 MB 00:00 2024-04-25T01:13:19.174 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.0 MB 00:01 2024-04-25T01:13:20.360 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-04-25T01:13:21.608 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-04-25T01:13:24.804 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-25T01:13:26.381 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - Extras packages 72 kB/s | 16 kB 00:00 2024-04-25T01:13:27.858 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-25T01:13:28.217 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 2.3 MB/s | 6.2 MB 00:02 2024-04-25T01:13:29.893 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 50 kB/s | 16 kB 00:00 2024-04-25T01:13:32.436 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 8.9 MB/s | 21 MB 00:02 2024-04-25T01:13:33.766 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-25T01:13:36.143 INFO:teuthology.orchestra.run.smithi090.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:13:36.144 INFO:teuthology.orchestra.run.smithi090.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:13:36.164 INFO:teuthology.orchestra.run.smithi090.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T01:13:36.166 INFO:teuthology.orchestra.run.smithi090.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T01:13:36.167 INFO:teuthology.orchestra.run.smithi090.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T01:13:36.168 INFO:teuthology.orchestra.run.smithi090.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T01:13:36.171 INFO:teuthology.orchestra.run.smithi090.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T01:13:36.174 INFO:teuthology.orchestra.run.smithi090.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T01:13:36.175 INFO:teuthology.orchestra.run.smithi090.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T01:13:36.245 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-25T01:13:36.270 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================= 2024-04-25T01:13:36.270 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-04-25T01:13:36.270 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================= 2024-04-25T01:13:36.270 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-04-25T01:13:36.270 INFO:teuthology.orchestra.run.smithi090.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T01:13:36.271 INFO:teuthology.orchestra.run.smithi090.stdout: ceph x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 6.6 k 2024-04-25T01:13:36.271 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.4 M 2024-04-25T01:13:36.271 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 890 k 2024-04-25T01:13:36.271 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 145 k 2024-04-25T01:13:36.271 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 1.5 M 2024-04-25T01:13:36.271 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 141 k 2024-04-25T01:13:36.271 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 3.5 M 2024-04-25T01:13:36.271 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 7.4 M 2024-04-25T01:13:36.271 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 49 k 2024-04-25T01:13:36.271 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 10 M 2024-04-25T01:13:36.271 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 48 M 2024-04-25T01:13:36.271 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-volume noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 264 k 2024-04-25T01:13:36.271 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 762 k 2024-04-25T01:13:36.272 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 222 k 2024-04-25T01:13:36.272 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-shell x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 27 k 2024-04-25T01:13:36.272 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-top noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 18 k 2024-04-25T01:13:36.272 INFO:teuthology.orchestra.run.smithi090.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T01:13:36.272 INFO:teuthology.orchestra.run.smithi090.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T01:13:36.272 INFO:teuthology.orchestra.run.smithi090.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T01:13:36.272 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 32 k 2024-04-25T01:13:36.272 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 745 k 2024-04-25T01:13:36.272 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 127 k 2024-04-25T01:13:36.272 INFO:teuthology.orchestra.run.smithi090.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T01:13:36.272 INFO:teuthology.orchestra.run.smithi090.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T01:13:36.272 INFO:teuthology.orchestra.run.smithi090.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T01:13:36.272 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T01:13:36.273 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T01:13:36.273 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T01:13:36.273 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 162 k 2024-04-25T01:13:36.273 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 321 k 2024-04-25T01:13:36.273 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 299 k 2024-04-25T01:13:36.273 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 99 k 2024-04-25T01:13:36.273 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 85 k 2024-04-25T01:13:36.273 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:13:36.273 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 171 k 2024-04-25T01:13:36.273 INFO:teuthology.orchestra.run.smithi090.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T01:13:36.273 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrading: 2024-04-25T01:13:36.273 INFO:teuthology.orchestra.run.smithi090.stdout: librados2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.4 M 2024-04-25T01:13:36.274 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:13:36.274 INFO:teuthology.orchestra.run.smithi090.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T01:13:36.274 INFO:teuthology.orchestra.run.smithi090.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T01:13:36.274 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-04-25T01:13:36.274 INFO:teuthology.orchestra.run.smithi090.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T01:13:36.274 INFO:teuthology.orchestra.run.smithi090.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T01:13:36.274 INFO:teuthology.orchestra.run.smithi090.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T01:13:36.274 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 21 M 2024-04-25T01:13:36.274 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 32 k 2024-04-25T01:13:36.274 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 2.4 M 2024-04-25T01:13:36.274 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 249 k 2024-04-25T01:13:36.274 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 4.7 M 2024-04-25T01:13:36.274 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 17 M 2024-04-25T01:13:36.275 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 15 k 2024-04-25T01:13:36.275 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 25 k 2024-04-25T01:13:36.275 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T01:13:36.275 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T01:13:36.275 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T01:13:36.275 INFO:teuthology.orchestra.run.smithi090.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T01:13:36.275 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T01:13:36.275 INFO:teuthology.orchestra.run.smithi090.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T01:13:36.275 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T01:13:36.275 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T01:13:36.275 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 163 k 2024-04-25T01:13:36.275 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T01:13:36.275 INFO:teuthology.orchestra.run.smithi090.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T01:13:36.276 INFO:teuthology.orchestra.run.smithi090.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T01:13:36.276 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T01:13:36.276 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T01:13:36.276 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 501 k 2024-04-25T01:13:36.276 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T01:13:36.276 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.3 M 2024-04-25T01:13:36.276 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T01:13:36.276 INFO:teuthology.orchestra.run.smithi090.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T01:13:36.276 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T01:13:36.276 INFO:teuthology.orchestra.run.smithi090.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T01:13:36.276 INFO:teuthology.orchestra.run.smithi090.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T01:13:36.276 INFO:teuthology.orchestra.run.smithi090.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T01:13:36.276 INFO:teuthology.orchestra.run.smithi090.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T01:13:36.277 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T01:13:36.277 INFO:teuthology.orchestra.run.smithi090.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T01:13:36.277 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T01:13:36.277 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T01:13:36.277 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T01:13:36.277 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T01:13:36.277 INFO:teuthology.orchestra.run.smithi090.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T01:13:36.277 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T01:13:36.277 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T01:13:36.277 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T01:13:36.277 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 45 k 2024-04-25T01:13:36.277 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 133 k 2024-04-25T01:13:36.277 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T01:13:36.278 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T01:13:36.278 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T01:13:36.278 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T01:13:36.278 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T01:13:36.278 INFO:teuthology.orchestra.run.smithi090.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T01:13:36.278 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T01:13:36.278 INFO:teuthology.orchestra.run.smithi090.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T01:13:36.278 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T01:13:36.278 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T01:13:36.278 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T01:13:36.278 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T01:13:36.278 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T01:13:36.279 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T01:13:36.279 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T01:13:36.279 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T01:13:36.279 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T01:13:36.279 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T01:13:36.279 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T01:13:36.279 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T01:13:36.279 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T01:13:36.279 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T01:13:36.279 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T01:13:36.279 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T01:13:36.279 INFO:teuthology.orchestra.run.smithi090.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T01:13:36.279 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T01:13:36.279 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T01:13:36.280 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T01:13:36.280 INFO:teuthology.orchestra.run.smithi090.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T01:13:36.280 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T01:13:36.280 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T01:13:36.280 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T01:13:36.280 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T01:13:36.280 INFO:teuthology.orchestra.run.smithi090.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T01:13:36.280 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T01:13:36.280 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T01:13:36.280 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T01:13:36.280 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T01:13:36.280 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T01:13:36.280 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T01:13:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T01:13:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T01:13:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T01:13:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T01:13:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T01:13:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T01:13:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T01:13:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T01:13:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T01:13:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T01:13:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T01:13:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T01:13:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T01:13:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T01:13:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T01:13:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T01:13:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T01:13:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T01:13:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T01:13:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T01:13:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T01:13:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T01:13:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T01:13:36.282 INFO:teuthology.orchestra.run.smithi090.stdout:Installing weak dependencies: 2024-04-25T01:13:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T01:13:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:13:36.282 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-04-25T01:13:36.282 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================= 2024-04-25T01:13:36.283 INFO:teuthology.orchestra.run.smithi090.stdout:Install 148 Packages 2024-04-25T01:13:36.283 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrade 4 Packages 2024-04-25T01:13:36.283 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:13:36.283 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 220 M 2024-04-25T01:13:36.283 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-04-25T01:13:37.627 INFO:teuthology.orchestra.run.smithi090.stdout:(1/152): ceph-19.0.0-2445.gd2c45d97.el9.x86_64. 21 kB/s | 6.6 kB 00:00 2024-04-25T01:13:38.028 INFO:teuthology.orchestra.run.smithi090.stdout:(2/152): ceph-base-19.0.0-2445.gd2c45d97.el9.x8 7.5 MB/s | 5.4 MB 00:00 2024-04-25T01:13:38.078 INFO:teuthology.orchestra.run.smithi090.stdout:(3/152): ceph-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.9 MB/s | 890 kB 00:00 2024-04-25T01:13:38.103 INFO:teuthology.orchestra.run.smithi090.stdout:(4/152): ceph-immutable-object-cache-19.0.0-244 1.9 MB/s | 145 kB 00:00 2024-04-25T01:13:38.263 INFO:teuthology.orchestra.run.smithi090.stdout:(5/152): ceph-mgr-19.0.0-2445.gd2c45d97.el9.x86 9.5 MB/s | 1.5 MB 00:00 2024-04-25T01:13:38.454 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 21 kB/s | 1.7 kB 00:00 2024-04-25T01:13:38.671 INFO:teuthology.orchestra.run.smithi090.stdout:(6/152): ceph-mon-19.0.0-2445.gd2c45d97.el9.x86 12 MB/s | 4.7 MB 00:00 2024-04-25T01:13:39.346 INFO:teuthology.orchestra.run.smithi090.stdout:(7/152): ceph-mds-19.0.0-2445.gd2c45d97.el9.x86 1.9 MB/s | 2.4 MB 00:01 2024-04-25T01:13:39.776 INFO:teuthology.orchestra.run.smithi090.stdout:(8/152): ceph-common-19.0.0-2445.gd2c45d97.el9. 8.5 MB/s | 21 MB 00:02 2024-04-25T01:13:39.839 INFO:teuthology.orchestra.run.smithi090.stdout:(9/152): ceph-selinux-19.0.0-2445.gd2c45d97.el9 405 kB/s | 25 kB 00:00 2024-04-25T01:13:40.106 INFO:teuthology.orchestra.run.smithi090.stdout:(10/152): ceph-osd-19.0.0-2445.gd2c45d97.el9.x8 12 MB/s | 17 MB 00:01 2024-04-25T01:13:40.182 INFO:teuthology.orchestra.run.smithi090.stdout:(11/152): cephfs-mirror-19.0.0-2445.gd2c45d97.e 2.9 MB/s | 222 kB 00:00 2024-04-25T01:13:40.240 INFO:teuthology.orchestra.run.smithi090.stdout:(12/152): cephfs-shell-19.0.0-2445.gd2c45d97.el 471 kB/s | 27 kB 00:00 2024-04-25T01:13:40.307 INFO:teuthology.orchestra.run.smithi090.stdout:(13/152): libcephfs-devel-19.0.0-2445.gd2c45d97 482 kB/s | 32 kB 00:00 2024-04-25T01:13:40.433 INFO:teuthology.orchestra.run.smithi090.stdout:(14/152): libcephfs2-19.0.0-2445.gd2c45d97.el9. 5.9 MB/s | 745 kB 00:00 2024-04-25T01:13:40.500 INFO:teuthology.orchestra.run.smithi090.stdout:(15/152): libcephsqlite-19.0.0-2445.gd2c45d97.e 2.4 MB/s | 163 kB 00:00 2024-04-25T01:13:40.575 INFO:teuthology.orchestra.run.smithi090.stdout:(16/152): librados-devel-19.0.0-2445.gd2c45d97. 1.7 MB/s | 127 kB 00:00 2024-04-25T01:13:40.667 INFO:teuthology.orchestra.run.smithi090.stdout:(17/152): libradosstriper1-19.0.0-2445.gd2c45d9 5.4 MB/s | 501 kB 00:00 2024-04-25T01:13:40.812 INFO:teuthology.orchestra.run.smithi103.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:13:40.813 INFO:teuthology.orchestra.run.smithi103.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:13:40.833 INFO:teuthology.orchestra.run.smithi103.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T01:13:40.834 INFO:teuthology.orchestra.run.smithi103.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T01:13:40.836 INFO:teuthology.orchestra.run.smithi103.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T01:13:40.837 INFO:teuthology.orchestra.run.smithi103.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T01:13:40.839 INFO:teuthology.orchestra.run.smithi103.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T01:13:40.843 INFO:teuthology.orchestra.run.smithi103.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T01:13:40.844 INFO:teuthology.orchestra.run.smithi103.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T01:13:40.914 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-25T01:13:40.931 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-04-25T01:13:40.931 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-25T01:13:40.931 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-04-25T01:13:40.931 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-25T01:13:40.931 INFO:teuthology.orchestra.run.smithi103.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T01:13:40.931 INFO:teuthology.orchestra.run.smithi103.stdout: ceph x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 6.6 k 2024-04-25T01:13:40.931 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.4 M 2024-04-25T01:13:40.932 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 890 k 2024-04-25T01:13:40.932 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 145 k 2024-04-25T01:13:40.932 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 1.5 M 2024-04-25T01:13:40.932 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 141 k 2024-04-25T01:13:40.932 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 3.5 M 2024-04-25T01:13:40.932 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 7.4 M 2024-04-25T01:13:40.932 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 49 k 2024-04-25T01:13:40.932 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 10 M 2024-04-25T01:13:40.932 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 48 M 2024-04-25T01:13:40.932 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-volume noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 264 k 2024-04-25T01:13:40.932 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 762 k 2024-04-25T01:13:40.932 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 222 k 2024-04-25T01:13:40.932 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-shell x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 27 k 2024-04-25T01:13:40.932 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-top noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 18 k 2024-04-25T01:13:40.933 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T01:13:40.933 INFO:teuthology.orchestra.run.smithi103.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T01:13:40.933 INFO:teuthology.orchestra.run.smithi103.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T01:13:40.933 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 32 k 2024-04-25T01:13:40.933 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 745 k 2024-04-25T01:13:40.933 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 127 k 2024-04-25T01:13:40.933 INFO:teuthology.orchestra.run.smithi103.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T01:13:40.933 INFO:teuthology.orchestra.run.smithi103.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T01:13:40.933 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T01:13:40.933 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T01:13:40.933 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T01:13:40.933 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T01:13:40.933 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 162 k 2024-04-25T01:13:40.933 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 321 k 2024-04-25T01:13:40.934 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 299 k 2024-04-25T01:13:40.934 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 99 k 2024-04-25T01:13:40.934 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 85 k 2024-04-25T01:13:40.934 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:13:40.934 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 171 k 2024-04-25T01:13:40.934 INFO:teuthology.orchestra.run.smithi103.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T01:13:40.934 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrading: 2024-04-25T01:13:40.934 INFO:teuthology.orchestra.run.smithi103.stdout: librados2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.4 M 2024-04-25T01:13:40.934 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:13:40.934 INFO:teuthology.orchestra.run.smithi103.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T01:13:40.934 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T01:13:40.934 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-25T01:13:40.935 INFO:teuthology.orchestra.run.smithi103.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T01:13:40.935 INFO:teuthology.orchestra.run.smithi103.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T01:13:40.935 INFO:teuthology.orchestra.run.smithi103.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T01:13:40.935 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 21 M 2024-04-25T01:13:40.935 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 32 k 2024-04-25T01:13:40.935 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 2.4 M 2024-04-25T01:13:40.935 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 249 k 2024-04-25T01:13:40.935 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 4.7 M 2024-04-25T01:13:40.935 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 17 M 2024-04-25T01:13:40.935 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 15 k 2024-04-25T01:13:40.935 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 25 k 2024-04-25T01:13:40.935 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T01:13:40.935 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T01:13:40.935 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T01:13:40.936 INFO:teuthology.orchestra.run.smithi103.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T01:13:40.936 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T01:13:40.936 INFO:teuthology.orchestra.run.smithi103.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T01:13:40.936 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T01:13:40.936 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T01:13:40.936 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 163 k 2024-04-25T01:13:40.936 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T01:13:40.936 INFO:teuthology.orchestra.run.smithi103.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T01:13:40.936 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T01:13:40.936 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T01:13:40.936 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T01:13:40.936 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 501 k 2024-04-25T01:13:40.936 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T01:13:40.936 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.3 M 2024-04-25T01:13:40.937 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T01:13:40.937 INFO:teuthology.orchestra.run.smithi103.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T01:13:40.937 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T01:13:40.937 INFO:teuthology.orchestra.run.smithi103.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T01:13:40.937 INFO:teuthology.orchestra.run.smithi103.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T01:13:40.937 INFO:teuthology.orchestra.run.smithi103.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T01:13:40.937 INFO:teuthology.orchestra.run.smithi103.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T01:13:40.937 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T01:13:40.937 INFO:teuthology.orchestra.run.smithi103.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T01:13:40.937 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T01:13:40.937 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T01:13:40.937 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T01:13:40.937 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T01:13:40.937 INFO:teuthology.orchestra.run.smithi103.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T01:13:40.938 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T01:13:40.938 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T01:13:40.938 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T01:13:40.938 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 45 k 2024-04-25T01:13:40.938 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 133 k 2024-04-25T01:13:40.938 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T01:13:40.938 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T01:13:40.938 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T01:13:40.938 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T01:13:40.938 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T01:13:40.938 INFO:teuthology.orchestra.run.smithi103.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T01:13:40.938 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T01:13:40.938 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T01:13:40.939 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T01:13:40.939 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T01:13:40.939 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T01:13:40.939 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T01:13:40.939 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T01:13:40.939 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T01:13:40.939 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T01:13:40.939 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T01:13:40.939 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T01:13:40.939 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T01:13:40.939 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T01:13:40.939 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T01:13:40.939 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T01:13:40.939 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T01:13:40.940 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T01:13:40.940 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T01:13:40.940 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T01:13:40.940 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T01:13:40.940 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T01:13:40.940 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T01:13:40.940 INFO:teuthology.orchestra.run.smithi103.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T01:13:40.940 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T01:13:40.940 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T01:13:40.940 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T01:13:40.940 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T01:13:40.940 INFO:teuthology.orchestra.run.smithi103.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T01:13:40.940 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T01:13:40.940 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T01:13:40.941 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T01:13:40.941 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T01:13:40.941 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T01:13:40.941 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T01:13:40.941 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T01:13:40.941 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T01:13:40.941 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T01:13:40.941 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T01:13:40.941 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T01:13:40.941 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T01:13:40.941 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T01:13:40.941 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T01:13:40.941 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T01:13:40.941 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T01:13:40.942 INFO:teuthology.orchestra.run.smithi103.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T01:13:40.942 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T01:13:40.942 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T01:13:40.942 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T01:13:40.942 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T01:13:40.942 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T01:13:40.942 INFO:teuthology.orchestra.run.smithi103.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T01:13:40.942 INFO:teuthology.orchestra.run.smithi103.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T01:13:40.942 INFO:teuthology.orchestra.run.smithi103.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T01:13:40.942 INFO:teuthology.orchestra.run.smithi103.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T01:13:40.942 INFO:teuthology.orchestra.run.smithi103.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T01:13:40.942 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T01:13:40.942 INFO:teuthology.orchestra.run.smithi103.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T01:13:40.943 INFO:teuthology.orchestra.run.smithi103.stdout:Installing weak dependencies: 2024-04-25T01:13:40.943 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T01:13:40.943 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:13:40.943 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-25T01:13:40.943 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-04-25T01:13:40.943 INFO:teuthology.orchestra.run.smithi103.stdout:Install 148 Packages 2024-04-25T01:13:40.943 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrade 4 Packages 2024-04-25T01:13:40.943 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:13:40.943 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 220 M 2024-04-25T01:13:40.944 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-25T01:13:41.192 INFO:teuthology.orchestra.run.smithi090.stdout:(18/152): librgw2-19.0.0-2445.gd2c45d97.el9.x86 10 MB/s | 5.3 MB 00:00 2024-04-25T01:13:41.259 INFO:teuthology.orchestra.run.smithi090.stdout:(19/152): python3-ceph-argparse-19.0.0-2445.gd2 689 kB/s | 45 kB 00:00 2024-04-25T01:13:41.335 INFO:teuthology.orchestra.run.smithi090.stdout:(20/152): python3-ceph-common-19.0.0-2445.gd2c4 1.7 MB/s | 133 kB 00:00 2024-04-25T01:13:41.418 INFO:teuthology.orchestra.run.smithi090.stdout:(21/152): python3-cephfs-19.0.0-2445.gd2c45d97. 1.9 MB/s | 162 kB 00:00 2024-04-25T01:13:41.502 INFO:teuthology.orchestra.run.smithi090.stdout:(22/152): python3-rados-19.0.0-2445.gd2c45d97.e 3.8 MB/s | 321 kB 00:00 2024-04-25T01:13:41.577 INFO:teuthology.orchestra.run.smithi090.stdout:(23/152): python3-rbd-19.0.0-2445.gd2c45d97.el9 3.9 MB/s | 299 kB 00:00 2024-04-25T01:13:41.628 INFO:teuthology.orchestra.run.smithi090.stdout:(24/152): python3-rgw-19.0.0-2445.gd2c45d97.el9 2.0 MB/s | 99 kB 00:00 2024-04-25T01:13:41.803 INFO:teuthology.orchestra.run.smithi090.stdout:(25/152): ceph-radosgw-19.0.0-2445.gd2c45d97.el 4.2 MB/s | 10 MB 00:02 2024-04-25T01:13:41.837 INFO:teuthology.orchestra.run.smithi090.stdout:(26/152): rbd-fuse-19.0.0-2445.gd2c45d97.el9.x8 410 kB/s | 85 kB 00:00 2024-04-25T01:13:41.887 INFO:teuthology.orchestra.run.smithi090.stdout:(27/152): rbd-nbd-19.0.0-2445.gd2c45d97.el9.x86 3.4 MB/s | 171 kB 00:00 2024-04-25T01:13:41.946 INFO:teuthology.orchestra.run.smithi090.stdout:(28/152): ceph-grafana-dashboards-19.0.0-2445.g 553 kB/s | 32 kB 00:00 2024-04-25T01:13:42.021 INFO:teuthology.orchestra.run.smithi090.stdout:(29/152): ceph-mgr-cephadm-19.0.0-2445.gd2c45d9 1.9 MB/s | 141 kB 00:00 2024-04-25T01:13:42.396 INFO:teuthology.orchestra.run.smithi103.stdout:(1/152): ceph-19.0.0-2445.gd2c45d97.el9.x86_64. 14 kB/s | 6.6 kB 00:00 2024-04-25T01:13:42.438 INFO:teuthology.orchestra.run.smithi090.stdout:(30/152): ceph-mgr-dashboard-19.0.0-2445.gd2c45 8.4 MB/s | 3.5 MB 00:00 2024-04-25T01:13:42.538 INFO:teuthology.orchestra.run.smithi090.stdout:(31/152): rbd-mirror-19.0.0-2445.gd2c45d97.el9. 4.2 MB/s | 3.1 MB 00:00 2024-04-25T01:13:42.648 INFO:teuthology.orchestra.run.smithi090.stdout:(32/152): ceph-mgr-modules-core-19.0.0-2445.gd2 2.2 MB/s | 249 kB 00:00 2024-04-25T01:13:42.757 INFO:teuthology.orchestra.run.smithi090.stdout:(33/152): ceph-mgr-rook-19.0.0-2445.gd2c45d97.e 459 kB/s | 49 kB 00:00 2024-04-25T01:13:42.849 INFO:teuthology.orchestra.run.smithi090.stdout:(34/152): ceph-prometheus-alerts-19.0.0-2445.gd 168 kB/s | 15 kB 00:00 2024-04-25T01:13:42.974 INFO:teuthology.orchestra.run.smithi090.stdout:(35/152): ceph-volume-19.0.0-2445.gd2c45d97.el9 2.1 MB/s | 264 kB 00:00 2024-04-25T01:13:43.205 INFO:teuthology.orchestra.run.smithi103.stdout:(2/152): ceph-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.1 MB/s | 890 kB 00:00 2024-04-25T01:13:43.321 INFO:teuthology.orchestra.run.smithi103.stdout:(3/152): ceph-immutable-object-cache-19.0.0-244 1.2 MB/s | 145 kB 00:00 2024-04-25T01:13:43.633 INFO:teuthology.orchestra.run.smithi090.stdout:(36/152): ceph-test-19.0.0-2445.gd2c45d97.el9.x 13 MB/s | 48 MB 00:03 2024-04-25T01:13:43.692 INFO:teuthology.orchestra.run.smithi090.stdout:(37/152): cephadm-19.0.0-2445.gd2c45d97.el9.noa 1.0 MB/s | 762 kB 00:00 2024-04-25T01:13:43.718 INFO:teuthology.orchestra.run.smithi090.stdout:(38/152): cephfs-top-19.0.0-2445.gd2c45d97.el9. 221 kB/s | 18 kB 00:00 2024-04-25T01:13:43.722 INFO:teuthology.orchestra.run.smithi103.stdout:(4/152): ceph-base-19.0.0-2445.gd2c45d97.el9.x8 3.0 MB/s | 5.4 MB 00:01 2024-04-25T01:13:43.871 INFO:teuthology.orchestra.run.smithi090.stdout:(39/152): ceph-mgr-diskprediction-local-19.0.0- 5.2 MB/s | 7.4 MB 00:01 2024-04-25T01:13:43.944 INFO:teuthology.orchestra.run.smithi103.stdout:(5/152): ceph-mgr-19.0.0-2445.gd2c45d97.el9.x86 6.9 MB/s | 1.5 MB 00:00 2024-04-25T01:13:44.036 INFO:teuthology.orchestra.run.smithi090.stdout:(40/152): fuse-2.9.9-15.el9.x86_64.rpm 235 kB/s | 80 kB 00:00 2024-04-25T01:13:44.118 INFO:teuthology.orchestra.run.smithi090.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 782 kB/s | 195 kB 00:00 2024-04-25T01:13:44.133 INFO:teuthology.orchestra.run.smithi103.stdout:(6/152): ceph-mds-19.0.0-2445.gd2c45d97.el9.x86 2.9 MB/s | 2.4 MB 00:00 2024-04-25T01:13:44.269 INFO:teuthology.orchestra.run.smithi090.stdout:(42/152): libgfortran-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 803 kB 00:00 2024-04-25T01:13:44.302 INFO:teuthology.orchestra.run.smithi090.stdout:(43/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 950 kB/s | 253 kB 00:00 2024-04-25T01:13:44.498 INFO:teuthology.orchestra.run.smithi103.stdout:(7/152): ceph-mon-19.0.0-2445.gd2c45d97.el9.x86 8.4 MB/s | 4.7 MB 00:00 2024-04-25T01:13:44.527 INFO:teuthology.orchestra.run.smithi090.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 413 kB/s | 106 kB 00:00 2024-04-25T01:13:44.569 INFO:teuthology.orchestra.run.smithi090.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 508 kB/s | 135 kB 00:00 2024-04-25T01:13:44.782 INFO:teuthology.orchestra.run.smithi103.stdout:(8/152): ceph-common-19.0.0-2445.gd2c45d97.el9. 7.3 MB/s | 21 MB 00:02 2024-04-25T01:13:44.794 INFO:teuthology.orchestra.run.smithi090.stdout:(46/152): python3-cryptography-36.0.1-4.el9.x86 1.8 MB/s | 1.2 MB 00:00 2024-04-25T01:13:44.828 INFO:teuthology.orchestra.run.smithi090.stdout:(47/152): python3-requests-2.25.1-8.el9.noarch. 416 kB/s | 125 kB 00:00 2024-04-25T01:13:44.848 INFO:teuthology.orchestra.run.smithi103.stdout:(9/152): ceph-selinux-19.0.0-2445.gd2c45d97.el9 385 kB/s | 25 kB 00:00 2024-04-25T01:13:44.895 INFO:teuthology.orchestra.run.smithi090.stdout:(48/152): python3-urllib3-1.26.5-5.el9.noarch.r 660 kB/s | 215 kB 00:00 2024-04-25T01:13:45.146 INFO:teuthology.orchestra.run.smithi090.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 429 kB/s | 107 kB 00:00 2024-04-25T01:13:45.271 INFO:teuthology.orchestra.run.smithi090.stdout:(50/152): bison-3.7.4-5.el9.x86_64.rpm 2.1 MB/s | 943 kB 00:00 2024-04-25T01:13:45.305 INFO:teuthology.orchestra.run.smithi090.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 136 kB/s | 21 kB 00:00 2024-04-25T01:13:45.346 INFO:teuthology.orchestra.run.smithi090.stdout:(52/152): automake-1.16.2-8.el9.noarch.rpm 1.2 MB/s | 689 kB 00:00 2024-04-25T01:13:45.442 INFO:teuthology.orchestra.run.smithi103.stdout:(10/152): ceph-radosgw-19.0.0-2445.gd2c45d97.el 11 MB/s | 10 MB 00:00 2024-04-25T01:13:45.447 INFO:teuthology.orchestra.run.smithi090.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 231 kB/s | 33 kB 00:00 2024-04-25T01:13:45.524 INFO:teuthology.orchestra.run.smithi103.stdout:(11/152): cephfs-mirror-19.0.0-2445.gd2c45d97.e 2.6 MB/s | 222 kB 00:00 2024-04-25T01:13:45.572 INFO:teuthology.orchestra.run.smithi090.stdout:(54/152): flex-2.6.4-9.el9.x86_64.rpm 1.0 MB/s | 313 kB 00:00 2024-04-25T01:13:45.582 INFO:teuthology.orchestra.run.smithi103.stdout:(12/152): cephfs-shell-19.0.0-2445.gd2c45d97.el 471 kB/s | 27 kB 00:00 2024-04-25T01:13:45.597 INFO:teuthology.orchestra.run.smithi090.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 122 kB/s | 18 kB 00:00 2024-04-25T01:13:45.673 INFO:teuthology.orchestra.run.smithi090.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 112 kB/s | 11 kB 00:00 2024-04-25T01:13:45.683 INFO:teuthology.orchestra.run.smithi103.stdout:(13/152): libcephfs-devel-19.0.0-2445.gd2c45d97 319 kB/s | 32 kB 00:00 2024-04-25T01:13:45.825 INFO:teuthology.orchestra.run.smithi103.stdout:(14/152): libcephfs2-19.0.0-2445.gd2c45d97.el9. 5.2 MB/s | 745 kB 00:00 2024-04-25T01:13:45.858 INFO:teuthology.orchestra.run.smithi090.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 618 kB/s | 160 kB 00:00 2024-04-25T01:13:45.883 INFO:teuthology.orchestra.run.smithi090.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 216 kB/s | 45 kB 00:00 2024-04-25T01:13:45.917 INFO:teuthology.orchestra.run.smithi103.stdout:(15/152): libcephsqlite-19.0.0-2445.gd2c45d97.e 1.8 MB/s | 163 kB 00:00 2024-04-25T01:13:45.992 INFO:teuthology.orchestra.run.smithi103.stdout:(16/152): librados-devel-19.0.0-2445.gd2c45d97. 1.7 MB/s | 127 kB 00:00 2024-04-25T01:13:46.076 INFO:teuthology.orchestra.run.smithi103.stdout:(17/152): libradosstriper1-19.0.0-2445.gd2c45d9 5.9 MB/s | 501 kB 00:00 2024-04-25T01:13:46.267 INFO:teuthology.orchestra.run.smithi090.stdout:(59/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.3 MB/s | 3.0 MB 00:00 2024-04-25T01:13:46.350 INFO:teuthology.orchestra.run.smithi090.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 1.2 MB/s | 582 kB 00:00 2024-04-25T01:13:46.475 INFO:teuthology.orchestra.run.smithi090.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 122 kB/s | 25 kB 00:00 2024-04-25T01:13:46.517 INFO:teuthology.orchestra.run.smithi090.stdout:(62/152): librdkafka-1.6.1-102.el9.x86_64.rpm 1.0 MB/s | 662 kB 00:00 2024-04-25T01:13:46.593 INFO:teuthology.orchestra.run.smithi090.stdout:(63/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 196 kB/s | 47 kB 00:00 2024-04-25T01:13:46.593 INFO:teuthology.orchestra.run.smithi103.stdout:(18/152): ceph-osd-19.0.0-2445.gd2c45d97.el9.x8 6.8 MB/s | 17 MB 00:02 2024-04-25T01:13:46.735 INFO:teuthology.orchestra.run.smithi103.stdout:(19/152): librgw2-19.0.0-2445.gd2c45d97.el9.x86 8.1 MB/s | 5.3 MB 00:00 2024-04-25T01:13:46.760 INFO:teuthology.orchestra.run.smithi103.stdout:(20/152): python3-ceph-argparse-19.0.0-2445.gd2 272 kB/s | 45 kB 00:00 2024-04-25T01:13:46.793 INFO:teuthology.orchestra.run.smithi103.stdout:(21/152): python3-ceph-common-19.0.0-2445.gd2c4 2.3 MB/s | 133 kB 00:00 2024-04-25T01:13:46.810 INFO:teuthology.orchestra.run.smithi090.stdout:(64/152): openblas-0.3.21-2.el9.x86_64.rpm 163 kB/s | 35 kB 00:00 2024-04-25T01:13:46.835 INFO:teuthology.orchestra.run.smithi103.stdout:(22/152): python3-cephfs-19.0.0-2445.gd2c45d97. 2.1 MB/s | 162 kB 00:00 2024-04-25T01:13:46.860 INFO:teuthology.orchestra.run.smithi090.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 552 kB/s | 188 kB 00:00 2024-04-25T01:13:46.877 INFO:teuthology.orchestra.run.smithi103.stdout:(23/152): python3-rados-19.0.0-2445.gd2c45d97.e 3.8 MB/s | 321 kB 00:00 2024-04-25T01:13:46.894 INFO:teuthology.orchestra.run.smithi090.stdout:(66/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 701 kB/s | 292 kB 00:00 2024-04-25T01:13:46.928 INFO:teuthology.orchestra.run.smithi103.stdout:(24/152): python3-rgw-19.0.0-2445.gd2c45d97.el9 2.0 MB/s | 99 kB 00:00 2024-04-25T01:13:46.961 INFO:teuthology.orchestra.run.smithi103.stdout:(25/152): python3-rbd-19.0.0-2445.gd2c45d97.el9 2.3 MB/s | 299 kB 00:00 2024-04-25T01:13:46.995 INFO:teuthology.orchestra.run.smithi103.stdout:(26/152): rbd-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.3 MB/s | 85 kB 00:00 2024-04-25T01:13:47.045 INFO:teuthology.orchestra.run.smithi103.stdout:(27/152): rbd-nbd-19.0.0-2445.gd2c45d97.el9.x86 3.4 MB/s | 171 kB 00:00 2024-04-25T01:13:47.112 INFO:teuthology.orchestra.run.smithi103.stdout:(28/152): ceph-grafana-dashboards-19.0.0-2445.g 482 kB/s | 32 kB 00:00 2024-04-25T01:13:47.162 INFO:teuthology.orchestra.run.smithi103.stdout:(29/152): ceph-mgr-cephadm-19.0.0-2445.gd2c45d9 2.8 MB/s | 141 kB 00:00 2024-04-25T01:13:47.421 INFO:teuthology.orchestra.run.smithi103.stdout:(30/152): rbd-mirror-19.0.0-2445.gd2c45d97.el9. 6.7 MB/s | 3.1 MB 00:00 2024-04-25T01:13:47.571 INFO:teuthology.orchestra.run.smithi103.stdout:(31/152): ceph-mgr-dashboard-19.0.0-2445.gd2c45 8.6 MB/s | 3.5 MB 00:00 2024-04-25T01:13:47.663 INFO:teuthology.orchestra.run.smithi103.stdout:(32/152): ceph-mgr-modules-core-19.0.0-2445.gd2 2.7 MB/s | 249 kB 00:00 2024-04-25T01:13:47.730 INFO:teuthology.orchestra.run.smithi103.stdout:(33/152): ceph-mgr-rook-19.0.0-2445.gd2c45d97.e 750 kB/s | 49 kB 00:00 2024-04-25T01:13:47.797 INFO:teuthology.orchestra.run.smithi103.stdout:(34/152): ceph-prometheus-alerts-19.0.0-2445.gd 234 kB/s | 15 kB 00:00 2024-04-25T01:13:47.872 INFO:teuthology.orchestra.run.smithi103.stdout:(35/152): ceph-volume-19.0.0-2445.gd2c45d97.el9 3.5 MB/s | 264 kB 00:00 2024-04-25T01:13:47.953 INFO:teuthology.orchestra.run.smithi090.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 1.5 MB/s | 1.6 MB 00:01 2024-04-25T01:13:48.006 INFO:teuthology.orchestra.run.smithi103.stdout:(36/152): cephadm-19.0.0-2445.gd2c45d97.el9.noa 5.6 MB/s | 762 kB 00:00 2024-04-25T01:13:48.056 INFO:teuthology.orchestra.run.smithi103.stdout:(37/152): cephfs-top-19.0.0-2445.gd2c45d97.el9. 373 kB/s | 18 kB 00:00 2024-04-25T01:13:48.182 INFO:teuthology.orchestra.run.smithi103.stdout:(38/152): ceph-mgr-diskprediction-local-19.0.0- 9.7 MB/s | 7.4 MB 00:00 2024-04-25T01:13:48.228 INFO:teuthology.orchestra.run.smithi090.stdout:(68/152): openblas-openmp-0.3.21-2.el9.x86_64.r 3.3 MB/s | 4.7 MB 00:01 2024-04-25T01:13:48.341 INFO:teuthology.orchestra.run.smithi103.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 284 kB/s | 80 kB 00:00 2024-04-25T01:13:48.478 INFO:teuthology.orchestra.run.smithi090.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 1.7 MB/s | 911 kB 00:00 2024-04-25T01:13:48.511 INFO:teuthology.orchestra.run.smithi090.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 472 kB/s | 133 kB 00:00 2024-04-25T01:13:48.657 INFO:teuthology.orchestra.run.smithi103.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 619 kB/s | 195 kB 00:00 2024-04-25T01:13:48.738 INFO:teuthology.orchestra.run.smithi090.stdout:(71/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 2.4 MB/s | 4.4 MB 00:01 2024-04-25T01:13:48.775 INFO:teuthology.orchestra.run.smithi103.stdout:(41/152): libgfortran-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 803 kB 00:00 2024-04-25T01:13:48.925 INFO:teuthology.orchestra.run.smithi103.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 951 kB/s | 253 kB 00:00 2024-04-25T01:13:48.929 INFO:teuthology.orchestra.run.smithi090.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 462 kB/s | 87 kB 00:00 2024-04-25T01:13:49.254 INFO:teuthology.orchestra.run.smithi090.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 1.4 MB/s | 1.0 MB 00:00 2024-04-25T01:13:49.483 INFO:teuthology.orchestra.run.smithi103.stdout:(43/152): ceph-test-19.0.0-2445.gd2c45d97.el9.x 10 MB/s | 48 MB 00:04 2024-04-25T01:13:49.546 INFO:teuthology.orchestra.run.smithi090.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 838 kB/s | 244 kB 00:00 2024-04-25T01:13:49.600 INFO:teuthology.orchestra.run.smithi103.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 158 kB/s | 106 kB 00:00 2024-04-25T01:13:49.709 INFO:teuthology.orchestra.run.smithi103.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 603 kB/s | 135 kB 00:00 2024-04-25T01:13:49.818 INFO:teuthology.orchestra.run.smithi103.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 578 kB/s | 125 kB 00:00 2024-04-25T01:13:49.838 INFO:teuthology.orchestra.run.smithi090.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 852 kB/s | 248 kB 00:00 2024-04-25T01:13:49.968 INFO:teuthology.orchestra.run.smithi103.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 833 kB/s | 215 kB 00:00 2024-04-25T01:13:50.005 INFO:teuthology.orchestra.run.smithi090.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 155 kB/s | 26 kB 00:00 2024-04-25T01:13:50.018 INFO:teuthology.orchestra.run.smithi103.stdout:(48/152): python3-cryptography-36.0.1-4.el9.x86 1.0 MB/s | 1.2 MB 00:01 2024-04-25T01:13:50.172 INFO:teuthology.orchestra.run.smithi090.stdout:(77/152): python3-babel-2.9.1-2.el9.noarch.rpm 4.8 MB/s | 6.0 MB 00:01 2024-04-25T01:13:50.198 INFO:teuthology.orchestra.run.smithi090.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 100 kB/s | 19 kB 00:00 2024-04-25T01:13:50.260 INFO:teuthology.orchestra.run.smithi103.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 444 kB/s | 107 kB 00:00 2024-04-25T01:13:50.302 INFO:teuthology.orchestra.run.smithi103.stdout:(50/152): automake-1.16.2-8.el9.noarch.rpm 1.4 MB/s | 689 kB 00:00 2024-04-25T01:13:50.381 INFO:teuthology.orchestra.run.smithi090.stdout:(79/152): postgresql-server-13.14-1.el9.x86_64. 3.1 MB/s | 5.8 MB 00:01 2024-04-25T01:13:50.402 INFO:teuthology.orchestra.run.smithi103.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 152 kB/s | 21 kB 00:00 2024-04-25T01:13:50.406 INFO:teuthology.orchestra.run.smithi090.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 167 kB/s | 35 kB 00:00 2024-04-25T01:13:50.482 INFO:teuthology.orchestra.run.smithi090.stdout:(81/152): python3-mako-1.1.4-6.el9.noarch.rpm 558 kB/s | 172 kB 00:00 2024-04-25T01:13:50.536 INFO:teuthology.orchestra.run.smithi103.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 247 kB/s | 33 kB 00:00 2024-04-25T01:13:50.569 INFO:teuthology.orchestra.run.smithi103.stdout:(53/152): flex-2.6.4-9.el9.x86_64.rpm 1.1 MB/s | 313 kB 00:00 2024-04-25T01:13:50.620 INFO:teuthology.orchestra.run.smithi103.stdout:(54/152): bison-3.7.4-5.el9.x86_64.rpm 1.4 MB/s | 943 kB 00:00 2024-04-25T01:13:50.700 INFO:teuthology.orchestra.run.smithi090.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 857 kB/s | 250 kB 00:00 2024-04-25T01:13:50.712 INFO:teuthology.orchestra.run.smithi103.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 129 kB/s | 18 kB 00:00 2024-04-25T01:13:50.729 INFO:teuthology.orchestra.run.smithi103.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 103 kB/s | 11 kB 00:00 2024-04-25T01:13:50.775 INFO:teuthology.orchestra.run.smithi090.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 759 kB/s | 222 kB 00:00 2024-04-25T01:13:50.884 INFO:teuthology.orchestra.run.smithi090.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 422 kB/s | 77 kB 00:00 2024-04-25T01:13:50.937 INFO:teuthology.orchestra.run.smithi103.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 218 kB/s | 45 kB 00:00 2024-04-25T01:13:50.954 INFO:teuthology.orchestra.run.smithi103.stdout:(58/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 664 kB/s | 160 kB 00:00 2024-04-25T01:13:50.968 INFO:teuthology.orchestra.run.smithi090.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 221 kB/s | 42 kB 00:00 2024-04-25T01:13:51.185 INFO:teuthology.orchestra.run.smithi090.stdout:(86/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 735 kB/s | 159 kB 00:00 2024-04-25T01:13:51.218 INFO:teuthology.orchestra.run.smithi090.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 808 kB/s | 269 kB 00:00 2024-04-25T01:13:51.296 INFO:teuthology.orchestra.run.smithi103.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 1.8 MB/s | 662 kB 00:00 2024-04-25T01:13:51.386 INFO:teuthology.orchestra.run.smithi090.stdout:(88/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 5.1 MB/s | 5.1 MB 00:01 2024-04-25T01:13:51.419 INFO:teuthology.orchestra.run.smithi090.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 255 kB/s | 51 kB 00:00 2024-04-25T01:13:51.446 INFO:teuthology.orchestra.run.smithi103.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 1.2 MB/s | 582 kB 00:00 2024-04-25T01:13:51.472 INFO:teuthology.orchestra.run.smithi103.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 146 kB/s | 25 kB 00:00 2024-04-25T01:13:51.561 INFO:teuthology.orchestra.run.smithi090.stdout:(90/152): python3-pyasn1-modules-0.4.8-6.el9.no 743 kB/s | 279 kB 00:00 2024-04-25T01:13:51.595 INFO:teuthology.orchestra.run.smithi090.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 258 kB/s | 54 kB 00:00 2024-04-25T01:13:51.622 INFO:teuthology.orchestra.run.smithi103.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 272 kB/s | 47 kB 00:00 2024-04-25T01:13:51.737 INFO:teuthology.orchestra.run.smithi090.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 238 kB/s | 42 kB 00:00 2024-04-25T01:13:51.747 INFO:teuthology.orchestra.run.smithi103.stdout:(63/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.5 MB/s | 3.0 MB 00:01 2024-04-25T01:13:51.804 INFO:teuthology.orchestra.run.smithi090.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 212 kB/s | 44 kB 00:00 2024-04-25T01:13:51.881 INFO:teuthology.orchestra.run.smithi103.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 716 kB/s | 292 kB 00:00 2024-04-25T01:13:51.950 INFO:teuthology.orchestra.run.smithi103.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 576 kB/s | 188 kB 00:00 2024-04-25T01:13:51.962 INFO:teuthology.orchestra.run.smithi090.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 228 kB/s | 36 kB 00:00 2024-04-25T01:13:51.992 INFO:teuthology.orchestra.run.smithi103.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 144 kB/s | 35 kB 00:00 2024-04-25T01:13:52.054 INFO:teuthology.orchestra.run.smithi090.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 697 kB/s | 221 kB 00:00 2024-04-25T01:13:52.163 INFO:teuthology.orchestra.run.smithi090.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 238 kB/s | 47 kB 00:00 2024-04-25T01:13:52.355 INFO:teuthology.orchestra.run.smithi090.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 1.0 MB/s | 305 kB 00:00 2024-04-25T01:13:52.380 INFO:teuthology.orchestra.run.smithi090.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 267 kB/s | 58 kB 00:00 2024-04-25T01:13:52.572 INFO:teuthology.orchestra.run.smithi090.stdout:(99/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 333 kB/s | 64 kB 00:00 2024-04-25T01:13:52.598 INFO:teuthology.orchestra.run.smithi090.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 388 kB/s | 94 kB 00:00 2024-04-25T01:13:52.723 INFO:teuthology.orchestra.run.smithi090.stdout:(101/152): lua-devel-5.4.4-4.el9.x86_64.rpm 179 kB/s | 22 kB 00:00 2024-04-25T01:13:52.782 INFO:teuthology.orchestra.run.smithi090.stdout:(102/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 215 kB/s | 45 kB 00:00 2024-04-25T01:13:52.817 INFO:teuthology.orchestra.run.smithi103.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 1.9 MB/s | 1.6 MB 00:00 2024-04-25T01:13:52.918 INFO:teuthology.orchestra.run.smithi103.stdout:(68/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 4.6 MB/s | 4.4 MB 00:00 2024-04-25T01:13:52.932 INFO:teuthology.orchestra.run.smithi090.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 4.1 MB/s | 863 kB 00:00 2024-04-25T01:13:53.193 INFO:teuthology.orchestra.run.smithi103.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 486 kB/s | 133 kB 00:00 2024-04-25T01:13:53.258 INFO:teuthology.orchestra.run.smithi090.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-04-25T01:13:53.291 INFO:teuthology.orchestra.run.smithi090.stdout:(105/152): gperftools-libs-2.9.1-2.el9.x86_64.r 863 kB/s | 309 kB 00:00 2024-04-25T01:13:53.477 INFO:teuthology.orchestra.run.smithi103.stdout:(70/152): openblas-openmp-0.3.21-2.el9.x86_64.r 3.0 MB/s | 4.7 MB 00:01 2024-04-25T01:13:53.533 INFO:teuthology.orchestra.run.smithi090.stdout:(106/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 7.7 MB/s | 16 MB 00:02 2024-04-25T01:13:53.568 INFO:teuthology.orchestra.run.smithi090.stdout:(107/152): grpc-data-1.46.7-10.el9.noarch.rpm 63 kB/s | 19 kB 00:00 2024-04-25T01:13:53.635 INFO:teuthology.orchestra.run.smithi090.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 735 kB/s | 49 kB 00:00 2024-04-25T01:13:53.635 INFO:teuthology.orchestra.run.smithi103.stdout:(71/152): postgresql-contrib-13.14-1.el9.x86_64 1.1 MB/s | 911 kB 00:00 2024-04-25T01:13:53.677 INFO:teuthology.orchestra.run.smithi090.stdout:(109/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 182 kB/s | 26 kB 00:00 2024-04-25T01:13:53.702 INFO:teuthology.orchestra.run.smithi090.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-04-25T01:13:53.852 INFO:teuthology.orchestra.run.smithi090.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 865 kB/s | 151 kB 00:00 2024-04-25T01:13:53.869 INFO:teuthology.orchestra.run.smithi103.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 376 kB/s | 87 kB 00:00 2024-04-25T01:13:53.977 INFO:teuthology.orchestra.run.smithi103.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 2.0 MB/s | 1.0 MB 00:00 2024-04-25T01:13:54.020 INFO:teuthology.orchestra.run.smithi090.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 3.2 MB/s | 548 kB 00:00 2024-04-25T01:13:54.095 INFO:teuthology.orchestra.run.smithi090.stdout:(113/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.1 MB/s | 838 kB 00:00 2024-04-25T01:13:54.120 INFO:teuthology.orchestra.run.smithi090.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 437 kB/s | 43 kB 00:00 2024-04-25T01:13:54.229 INFO:teuthology.orchestra.run.smithi090.stdout:(115/152): libarrow-9.0.0-10.el9.x86_64.rpm 4.7 MB/s | 4.4 MB 00:00 2024-04-25T01:13:54.244 INFO:teuthology.orchestra.run.smithi103.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 919 kB/s | 244 kB 00:00 2024-04-25T01:13:54.254 INFO:teuthology.orchestra.run.smithi090.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 113 kB/s | 15 kB 00:00 2024-04-25T01:13:54.279 INFO:teuthology.orchestra.run.smithi090.stdout:(117/152): python3-cachetools-4.2.4-1.el9.noarc 175 kB/s | 32 kB 00:00 2024-04-25T01:13:54.313 INFO:teuthology.orchestra.run.smithi090.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-04-25T01:13:54.381 INFO:teuthology.orchestra.run.smithi090.stdout:(119/152): python3-colorama-0.4.4-12.el9.noarch 531 kB/s | 35 kB 00:00 2024-04-25T01:13:54.414 INFO:teuthology.orchestra.run.smithi090.stdout:(120/152): python3-cherrypy-18.6.1-2.el9.noarch 2.2 MB/s | 358 kB 00:00 2024-04-25T01:13:54.515 INFO:teuthology.orchestra.run.smithi090.stdout:(121/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.3 MB/s | 303 kB 00:00 2024-04-25T01:13:54.557 INFO:teuthology.orchestra.run.smithi090.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 1.1 MB/s | 206 kB 00:00 2024-04-25T01:13:54.561 INFO:teuthology.orchestra.run.smithi103.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 785 kB/s | 248 kB 00:00 2024-04-25T01:13:54.590 INFO:teuthology.orchestra.run.smithi090.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-04-25T01:13:54.624 INFO:teuthology.orchestra.run.smithi090.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 162 kB/s | 11 kB 00:00 2024-04-25T01:13:54.649 INFO:teuthology.orchestra.run.smithi090.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 305 kB/s | 18 kB 00:00 2024-04-25T01:13:54.683 INFO:teuthology.orchestra.run.smithi090.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 400 kB/s | 23 kB 00:00 2024-04-25T01:13:54.703 INFO:teuthology.orchestra.run.smithi103.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 183 kB/s | 26 kB 00:00 2024-04-25T01:13:54.716 INFO:teuthology.orchestra.run.smithi090.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 293 kB/s | 19 kB 00:00 2024-04-25T01:13:54.742 INFO:teuthology.orchestra.run.smithi090.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 339 kB/s | 20 kB 00:00 2024-04-25T01:13:54.809 INFO:teuthology.orchestra.run.smithi090.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 702 kB/s | 46 kB 00:00 2024-04-25T01:13:54.862 INFO:teuthology.orchestra.run.smithi103.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 122 kB/s | 19 kB 00:00 2024-04-25T01:13:54.884 INFO:teuthology.orchestra.run.smithi090.stdout:(130/152): python3-grpcio-1.46.7-10.el9.x86_64. 4.3 MB/s | 2.0 MB 00:00 2024-04-25T01:13:54.919 INFO:teuthology.orchestra.run.smithi090.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 723 kB/s | 79 kB 00:00 2024-04-25T01:13:54.953 INFO:teuthology.orchestra.run.smithi090.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 856 kB/s | 58 kB 00:00 2024-04-25T01:13:54.995 INFO:teuthology.orchestra.run.smithi090.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 3.6 MB/s | 272 kB 00:00 2024-04-25T01:13:55.020 INFO:teuthology.orchestra.run.smithi090.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 247 kB/s | 16 kB 00:00 2024-04-25T01:13:55.062 INFO:teuthology.orchestra.run.smithi090.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-04-25T01:13:55.112 INFO:teuthology.orchestra.run.smithi103.stdout:(78/152): python3-babel-2.9.1-2.el9.noarch.rpm 4.8 MB/s | 6.0 MB 00:01 2024-04-25T01:13:55.147 INFO:teuthology.orchestra.run.smithi090.stdout:(136/152): python3-kubernetes-26.1.0-2.el9.noar 2.4 MB/s | 1.0 MB 00:00 2024-04-25T01:13:55.179 INFO:teuthology.orchestra.run.smithi090.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 178 kB/s | 28 kB 00:00 2024-04-25T01:13:55.205 INFO:teuthology.orchestra.run.smithi090.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 217 kB/s | 31 kB 00:00 2024-04-25T01:13:55.220 INFO:teuthology.orchestra.run.smithi103.stdout:(79/152): postgresql-server-13.14-1.el9.x86_64. 2.9 MB/s | 5.8 MB 00:02 2024-04-25T01:13:55.238 INFO:teuthology.orchestra.run.smithi090.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-04-25T01:13:55.246 INFO:teuthology.orchestra.run.smithi103.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 450 kB/s | 172 kB 00:00 2024-04-25T01:13:55.272 INFO:teuthology.orchestra.run.smithi090.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 898 kB/s | 59 kB 00:00 2024-04-25T01:13:55.297 INFO:teuthology.orchestra.run.smithi090.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 394 kB/s | 36 kB 00:00 2024-04-25T01:13:55.322 INFO:teuthology.orchestra.run.smithi090.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 613 kB/s | 51 kB 00:00 2024-04-25T01:13:55.346 INFO:teuthology.orchestra.run.smithi103.stdout:(81/152): python3-markupsafe-1.1.1-12.el9.x86_6 149 kB/s | 35 kB 00:00 2024-04-25T01:13:55.364 INFO:teuthology.orchestra.run.smithi090.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-04-25T01:13:55.406 INFO:teuthology.orchestra.run.smithi090.stdout:(144/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-04-25T01:13:55.440 INFO:teuthology.orchestra.run.smithi090.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-04-25T01:13:55.465 INFO:teuthology.orchestra.run.smithi090.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 201 kB/s | 20 kB 00:00 2024-04-25T01:13:55.496 INFO:teuthology.orchestra.run.smithi103.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.0 MB/s | 250 kB 00:00 2024-04-25T01:13:55.499 INFO:teuthology.orchestra.run.smithi090.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-04-25T01:13:55.621 INFO:teuthology.orchestra.run.smithi103.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 810 kB/s | 222 kB 00:00 2024-04-25T01:13:55.683 INFO:teuthology.orchestra.run.smithi090.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 6.5 MB/s | 1.6 MB 00:00 2024-04-25T01:13:55.722 INFO:teuthology.orchestra.run.smithi103.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 344 kB/s | 77 kB 00:00 2024-04-25T01:13:55.822 INFO:teuthology.orchestra.run.smithi103.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 211 kB/s | 42 kB 00:00 2024-04-25T01:13:55.867 INFO:teuthology.orchestra.run.smithi090.stdout:(149/152): librbd1-19.0.0-2445.gd2c45d97.el9.x8 8.5 MB/s | 3.1 MB 00:00 2024-04-25T01:13:56.039 INFO:teuthology.orchestra.run.smithi103.stdout:(86/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 6.2 MB/s | 5.1 MB 00:00 2024-04-25T01:13:56.073 INFO:teuthology.orchestra.run.smithi103.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 770 kB/s | 269 kB 00:00 2024-04-25T01:13:56.106 INFO:teuthology.orchestra.run.smithi103.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 561 kB/s | 159 kB 00:00 2024-04-25T01:13:56.142 INFO:teuthology.orchestra.run.smithi090.stdout:(150/152): librados2-19.0.0-2445.gd2c45d97.el9. 5.0 MB/s | 3.4 MB 00:00 2024-04-25T01:13:56.242 INFO:teuthology.orchestra.run.smithi090.stdout:(151/152): openssl-3.2.1-1.el9.x86_64.rpm 2.4 MB/s | 1.3 MB 00:00 2024-04-25T01:13:56.282 INFO:teuthology.orchestra.run.smithi103.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 245 kB/s | 51 kB 00:00 2024-04-25T01:13:56.332 INFO:teuthology.orchestra.run.smithi103.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 239 kB/s | 54 kB 00:00 2024-04-25T01:13:56.365 INFO:teuthology.orchestra.run.smithi103.stdout:(91/152): python3-pyasn1-modules-0.4.8-6.el9.no 857 kB/s | 279 kB 00:00 2024-04-25T01:13:56.566 INFO:teuthology.orchestra.run.smithi103.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 179 kB/s | 42 kB 00:00 2024-04-25T01:13:56.616 INFO:teuthology.orchestra.run.smithi103.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 177 kB/s | 44 kB 00:00 2024-04-25T01:13:56.617 INFO:teuthology.orchestra.run.smithi090.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 3.2 MB/s | 2.4 MB 00:00 2024-04-25T01:13:56.625 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:13:56.625 INFO:teuthology.orchestra.run.smithi090.stdout:Total 11 MB/s | 220 MB 00:20 2024-04-25T01:13:56.800 INFO:teuthology.orchestra.run.smithi103.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 196 kB/s | 36 kB 00:00 2024-04-25T01:13:56.867 INFO:teuthology.orchestra.run.smithi103.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 736 kB/s | 221 kB 00:00 2024-04-25T01:13:57.084 INFO:teuthology.orchestra.run.smithi103.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 167 kB/s | 47 kB 00:00 2024-04-25T01:13:57.159 INFO:teuthology.orchestra.run.smithi103.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 1.0 MB/s | 305 kB 00:00 2024-04-25T01:13:57.335 INFO:teuthology.orchestra.run.smithi103.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 233 kB/s | 58 kB 00:00 2024-04-25T01:13:57.360 INFO:teuthology.orchestra.run.smithi103.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 470 kB/s | 94 kB 00:00 2024-04-25T01:13:57.577 INFO:teuthology.orchestra.run.smithi103.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 264 kB/s | 64 kB 00:00 2024-04-25T01:13:57.619 INFO:teuthology.orchestra.run.smithi103.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 174 kB/s | 45 kB 00:00 2024-04-25T01:13:57.894 INFO:teuthology.orchestra.run.smithi103.stdout:(102/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 10 MB/s | 16 MB 00:01 2024-04-25T01:13:57.911 INFO:teuthology.orchestra.run.smithi103.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 67 kB/s | 22 kB 00:00 2024-04-25T01:13:58.094 INFO:teuthology.orchestra.run.smithi103.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 1.8 MB/s | 863 kB 00:00 2024-04-25T01:13:58.378 INFO:teuthology.orchestra.run.smithi103.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 69 kB/s | 19 kB 00:00 2024-04-25T01:13:58.442 INFO:teuthology.orchestra.run.smithi103.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 583 kB/s | 309 kB 00:00 2024-04-25T01:13:58.562 INFO:teuthology.orchestra.run.smithi103.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 822 kB/s | 548 kB 00:00 2024-04-25T01:13:58.729 INFO:teuthology.orchestra.run.smithi103.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 91 kB/s | 26 kB 00:00 2024-04-25T01:13:58.930 INFO:teuthology.orchestra.run.smithi103.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 133 kB/s | 49 kB 00:00 2024-04-25T01:13:59.080 INFO:teuthology.orchestra.run.smithi103.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 193 kB/s | 67 kB 00:00 2024-04-25T01:13:59.380 INFO:teuthology.orchestra.run.smithi103.stdout:(111/152): libarrow-9.0.0-10.el9.x86_64.rpm 4.4 MB/s | 4.4 MB 00:01 2024-04-25T01:13:59.405 INFO:teuthology.orchestra.run.smithi103.stdout:(112/152): luarocks-3.9.2-1.el9.noarch.rpm 318 kB/s | 151 kB 00:00 2024-04-25T01:13:59.518 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-04-25T01:13:59.772 INFO:teuthology.orchestra.run.smithi103.stdout:(113/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.2 MB/s | 838 kB 00:00 2024-04-25T01:13:59.814 INFO:teuthology.orchestra.run.smithi103.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 106 kB/s | 43 kB 00:00 2024-04-25T01:13:59.988 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-04-25T01:13:59.988 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-04-25T01:14:00.015 INFO:teuthology.orchestra.run.smithi103.stdout:(115/152): python3-asyncssh-2.13.2-5.el9.noarch 865 kB/s | 548 kB 00:00 2024-04-25T01:14:00.056 INFO:teuthology.orchestra.run.smithi103.stdout:(116/152): python3-cachetools-4.2.4-1.el9.noarc 114 kB/s | 32 kB 00:00 2024-04-25T01:14:00.099 INFO:teuthology.orchestra.run.smithi103.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 53 kB/s | 15 kB 00:00 2024-04-25T01:14:00.491 INFO:teuthology.orchestra.run.smithi103.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 361 kB/s | 172 kB 00:00 2024-04-25T01:14:00.566 INFO:teuthology.orchestra.run.smithi103.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 704 kB/s | 358 kB 00:00 2024-04-25T01:14:00.608 INFO:teuthology.orchestra.run.smithi103.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 596 kB/s | 303 kB 00:00 2024-04-25T01:14:00.767 INFO:teuthology.orchestra.run.smithi103.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 128 kB/s | 35 kB 00:00 2024-04-25T01:14:01.075 INFO:teuthology.orchestra.run.smithi103.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 405 kB/s | 206 kB 00:00 2024-04-25T01:14:01.192 INFO:teuthology.orchestra.run.smithi103.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 340 kB/s | 144 kB 00:00 2024-04-25T01:14:01.259 INFO:teuthology.orchestra.run.smithi103.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 58 kB/s | 11 kB 00:00 2024-04-25T01:14:01.476 INFO:teuthology.orchestra.run.smithi103.stdout:(125/152): python3-grpcio-1.46.7-10.el9.x86_64. 2.4 MB/s | 2.0 MB 00:00 2024-04-25T01:14:01.510 INFO:teuthology.orchestra.run.smithi103.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 56 kB/s | 18 kB 00:00 2024-04-25T01:14:01.535 INFO:teuthology.orchestra.run.smithi103.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 84 kB/s | 23 kB 00:00 2024-04-25T01:14:01.752 INFO:teuthology.orchestra.run.smithi103.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 71 kB/s | 19 kB 00:00 2024-04-25T01:14:01.778 INFO:teuthology.orchestra.run.smithi103.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 74 kB/s | 20 kB 00:00 2024-04-25T01:14:02.153 INFO:teuthology.orchestra.run.smithi103.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 116 kB/s | 46 kB 00:00 2024-04-25T01:14:02.178 INFO:teuthology.orchestra.run.smithi103.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 197 kB/s | 79 kB 00:00 2024-04-25T01:14:02.337 INFO:teuthology.orchestra.run.smithi103.stdout:(132/152): python3-kubernetes-26.1.0-2.el9.noar 1.3 MB/s | 1.0 MB 00:00 2024-04-25T01:14:02.504 INFO:teuthology.orchestra.run.smithi103.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 165 kB/s | 58 kB 00:00 2024-04-25T01:14:02.612 INFO:teuthology.orchestra.run.smithi103.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 60 kB/s | 16 kB 00:00 2024-04-25T01:14:02.696 INFO:teuthology.orchestra.run.smithi103.stdout:(135/152): python3-pecan-1.4.2-3.el9.noarch.rpm 527 kB/s | 272 kB 00:00 2024-04-25T01:14:02.776 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-04-25T01:14:02.777 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-04-25T01:14:02.888 INFO:teuthology.orchestra.run.smithi103.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 103 kB/s | 28 kB 00:00 2024-04-25T01:14:02.929 INFO:teuthology.orchestra.run.smithi103.stdout:(137/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 212 kB/s | 90 kB 00:00 2024-04-25T01:14:02.963 INFO:teuthology.orchestra.run.smithi103.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 115 kB/s | 31 kB 00:00 2024-04-25T01:14:03.230 INFO:teuthology.orchestra.run.smithi103.stdout:(139/152): python3-tempora-5.0.0-2.el9.noarch.r 134 kB/s | 36 kB 00:00 2024-04-25T01:14:03.272 INFO:teuthology.orchestra.run.smithi103.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 173 kB/s | 59 kB 00:00 2024-04-25T01:14:03.314 INFO:teuthology.orchestra.run.smithi103.stdout:(141/152): python3-routes-2.5.1-5.el9.noarch.rp 443 kB/s | 188 kB 00:00 2024-04-25T01:14:03.609 INFO:teuthology.orchestra.run.smithi103.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 135 kB/s | 51 kB 00:00 2024-04-25T01:14:03.757 INFO:teuthology.orchestra.run.smithi103.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 203 kB/s | 90 kB 00:00 2024-04-25T01:14:03.782 INFO:teuthology.orchestra.run.smithi103.stdout:(144/152): python3-webob-1.8.7-6.el9.noarch.rpm 451 kB/s | 230 kB 00:00 2024-04-25T01:14:04.057 INFO:teuthology.orchestra.run.smithi103.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 67 kB/s | 20 kB 00:00 2024-04-25T01:14:04.216 INFO:teuthology.orchestra.run.smithi103.stdout:(146/152): python3-werkzeug-2.0.3-3.el9.1.noarc 706 kB/s | 427 kB 00:00 2024-04-25T01:14:04.291 INFO:teuthology.orchestra.run.smithi103.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 377 kB/s | 192 kB 00:00 2024-04-25T01:14:04.642 INFO:teuthology.orchestra.run.smithi103.stdout:(148/152): librbd1-19.0.0-2445.gd2c45d97.el9.x8 9.0 MB/s | 3.1 MB 00:00 2024-04-25T01:14:04.791 INFO:teuthology.orchestra.run.smithi103.stdout:(149/152): librados2-19.0.0-2445.gd2c45d97.el9. 5.9 MB/s | 3.4 MB 00:00 2024-04-25T01:14:04.850 INFO:teuthology.orchestra.run.smithi103.stdout:(150/152): thrift-0.15.0-2.el9.x86_64.rpm 2.0 MB/s | 1.6 MB 00:00 2024-04-25T01:14:05.442 INFO:teuthology.orchestra.run.smithi103.stdout:(151/152): openssl-3.2.1-1.el9.x86_64.rpm 1.7 MB/s | 1.3 MB 00:00 2024-04-25T01:14:05.634 INFO:teuthology.orchestra.run.smithi103.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 2.8 MB/s | 2.4 MB 00:00 2024-04-25T01:14:05.641 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:14:05.642 INFO:teuthology.orchestra.run.smithi103.stdout:Total 8.9 MB/s | 220 MB 00:24 2024-04-25T01:14:08.536 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-25T01:14:08.782 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-25T01:14:08.782 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-25T01:14:09.419 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-04-25T01:14:09.527 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T01:14:09.616 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T01:14:09.734 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T01:14:09.824 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T01:14:09.863 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T01:14:09.996 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T01:14:10.419 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T01:14:10.553 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:14:10.739 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:14:10.772 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:14:10.906 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:14:10.946 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:14:11.314 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T01:14:11.377 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T01:14:11.410 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:14:11.553 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-25T01:14:11.553 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-25T01:14:11.562 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:14:11.594 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:14:11.717 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:14:11.754 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:14:11.825 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:14:12.133 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:14:12.195 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:14:12.408 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T01:14:12.536 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T01:14:12.573 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T01:14:12.621 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T01:14:12.723 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T01:14:12.768 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T01:14:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T01:14:12.885 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T01:14:12.922 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T01:14:13.077 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T01:14:13.153 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T01:14:13.197 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T01:14:13.261 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T01:14:13.368 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T01:14:13.412 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 31/156 2024-04-25T01:14:13.470 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 32/156 2024-04-25T01:14:13.511 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 33/156 2024-04-25T01:14:13.629 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T01:14:13.689 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T01:14:13.789 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T01:14:13.958 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T01:14:14.019 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T01:14:14.706 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T01:14:14.816 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T01:14:14.853 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T01:14:14.935 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T01:14:14.985 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 43/156 2024-04-25T01:14:15.064 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T01:14:15.409 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T01:14:15.469 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T01:14:15.593 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T01:14:15.701 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T01:14:15.741 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T01:14:15.780 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T01:14:15.848 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T01:14:15.889 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T01:14:15.951 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T01:14:16.017 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T01:14:16.225 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T01:14:16.280 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T01:14:16.317 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T01:14:16.375 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T01:14:16.515 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T01:14:16.566 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T01:14:16.629 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T01:14:16.668 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T01:14:16.701 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T01:14:16.909 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T01:14:17.087 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T01:14:17.132 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T01:14:17.162 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T01:14:17.205 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T01:14:17.258 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T01:14:17.312 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T01:14:17.354 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T01:14:17.399 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T01:14:17.484 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T01:14:17.561 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T01:14:17.616 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T01:14:17.768 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T01:14:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T01:14:17.873 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T01:14:17.936 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T01:14:17.967 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-25T01:14:18.066 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T01:14:18.079 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T01:14:18.154 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T01:14:18.155 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T01:14:18.262 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T01:14:18.318 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T01:14:18.351 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T01:14:18.479 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T01:14:18.941 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T01:14:19.077 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:14:19.105 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T01:14:19.171 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T01:14:19.204 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T01:14:19.263 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:14:19.292 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:14:19.340 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T01:14:19.417 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T01:14:19.419 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:14:19.474 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:14:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T01:14:19.566 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T01:14:19.802 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T01:14:19.815 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T01:14:19.830 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T01:14:19.871 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T01:14:19.896 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:14:19.974 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T01:14:20.054 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:14:20.089 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:14:20.174 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:14:20.207 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:14:20.278 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:14:20.684 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:14:20.722 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:14:20.825 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T01:14:20.924 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T01:14:20.970 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T01:14:21.050 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T01:14:21.091 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T01:14:21.138 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T01:14:21.240 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T01:14:21.302 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T01:14:21.367 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T01:14:21.410 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T01:14:21.448 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T01:14:21.563 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T01:14:21.634 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T01:14:21.674 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T01:14:21.733 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T01:14:21.840 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T01:14:21.878 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 31/156 2024-04-25T01:14:21.919 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 32/156 2024-04-25T01:14:21.951 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 33/156 2024-04-25T01:14:22.061 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T01:14:22.121 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T01:14:22.216 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T01:14:22.383 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T01:14:22.447 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T01:14:22.620 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T01:14:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T01:14:22.775 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T01:14:22.823 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T01:14:23.081 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T01:14:23.181 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T01:14:23.217 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T01:14:23.219 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T01:14:23.260 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T01:14:23.264 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:14:23.301 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 43/156 2024-04-25T01:14:23.360 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T01:14:23.437 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:14:23.472 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 100/156 2024-04-25T01:14:23.529 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T01:14:23.621 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T01:14:23.695 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T01:14:23.777 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T01:14:23.877 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T01:14:23.919 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T01:14:23.958 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T01:14:24.025 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T01:14:24.067 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T01:14:24.071 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T01:14:24.109 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:14:24.129 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T01:14:24.186 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T01:14:24.210 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:14:24.238 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 104/156 2024-04-25T01:14:24.385 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T01:14:24.440 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T01:14:24.506 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T01:14:24.563 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T01:14:24.702 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T01:14:24.743 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T01:14:24.792 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T01:14:24.831 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T01:14:24.859 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T01:14:25.045 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T01:14:25.221 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T01:14:25.265 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T01:14:25.304 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T01:14:25.341 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T01:14:25.424 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T01:14:25.488 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T01:14:25.538 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T01:14:25.599 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T01:14:25.651 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T01:14:25.725 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T01:14:25.771 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T01:14:25.915 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T01:14:25.978 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T01:14:26.021 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T01:14:26.091 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T01:14:26.239 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T01:14:26.325 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T01:14:26.983 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:14:27.036 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:14:27.285 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T01:14:27.342 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T01:14:27.375 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T01:14:27.497 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T01:14:27.569 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T01:14:27.667 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T01:14:27.710 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T01:14:27.727 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:14:27.774 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:14:27.928 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:14:27.929 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T01:14:27.929 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T01:14:27.929 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:14:27.940 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T01:14:27.974 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T01:14:27.975 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:14:28.089 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T01:14:28.925 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T01:14:29.081 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T01:14:30.687 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T01:14:30.823 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T01:14:30.872 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T01:14:30.922 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T01:14:31.305 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T01:14:31.352 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:14:31.475 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:14:31.509 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 100/156 2024-04-25T01:14:31.566 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T01:14:32.098 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T01:14:32.130 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:14:32.222 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:14:32.249 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 104/156 2024-04-25T01:14:35.081 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:14:35.131 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:14:35.779 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:14:35.826 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:14:35.999 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:14:35.999 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T01:14:36.000 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T01:14:36.000 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:14:36.044 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:14:42.451 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:14:42.452 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /sys 2024-04-25T01:14:42.452 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /proc 2024-04-25T01:14:42.452 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /mnt 2024-04-25T01:14:42.452 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /var/tmp 2024-04-25T01:14:42.452 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /home 2024-04-25T01:14:42.452 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /root 2024-04-25T01:14:42.452 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /tmp 2024-04-25T01:14:42.452 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:14:42.681 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:14:42.723 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:14:42.723 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:14:42.723 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T01:14:42.723 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:14:44.414 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:14:44.495 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:14:44.495 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:14:44.495 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T01:14:44.495 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:14:44.496 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:14:44.496 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:14:44.757 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:14:44.825 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:14:44.826 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:14:44.826 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T01:14:44.826 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:14:44.826 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:14:44.826 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:14:45.317 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:14:45.382 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:14:45.383 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:14:45.383 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T01:14:45.383 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:14:45.383 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:14:45.383 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:14:45.509 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T01:14:45.570 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T01:14:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T01:14:45.896 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T01:14:45.965 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T01:14:46.058 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T01:14:46.167 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:14:46.332 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:14:46.400 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T01:14:46.450 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T01:14:46.482 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T01:14:47.218 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:14:47.262 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:14:47.326 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 124/156 2024-04-25T01:14:47.447 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 125/156 2024-04-25T01:14:47.481 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:14:48.792 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:14:48.842 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:14:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:14:49.086 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /sys 2024-04-25T01:14:49.086 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /proc 2024-04-25T01:14:49.086 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /mnt 2024-04-25T01:14:49.086 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /var/tmp 2024-04-25T01:14:49.086 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /home 2024-04-25T01:14:49.086 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /root 2024-04-25T01:14:49.086 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /tmp 2024-04-25T01:14:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:14:49.292 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:14:49.334 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:14:49.334 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:14:49.335 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T01:14:49.335 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:14:49.932 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:14:49.958 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:14:50.104 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:14:50.295 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 129/156 2024-04-25T01:14:50.327 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:14:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:14:50.394 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:14:50.395 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T01:14:50.395 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:14:50.395 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:14:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:14:50.453 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:14:50.492 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:14:50.610 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 132/156 2024-04-25T01:14:50.677 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 133/156 2024-04-25T01:14:50.927 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:14:51.007 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:14:51.007 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:14:51.007 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T01:14:51.007 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:14:51.007 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:14:51.007 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:14:51.261 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:14:51.332 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:14:51.332 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:14:51.332 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T01:14:51.332 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:14:51.332 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:14:51.332 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:14:51.654 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T01:14:51.699 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:14:51.769 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:14:51.769 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:14:51.769 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T01:14:51.769 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:14:51.769 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:14:51.769 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:14:51.812 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:14:51.842 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:14:51.881 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:14:51.881 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:14:51.881 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T01:14:51.881 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:14:51.881 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:14:51.881 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:14:51.910 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:14:51.910 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:14:51.911 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T01:14:51.911 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:14:51.976 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:14:52.020 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T01:14:52.050 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:14:52.050 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:14:52.050 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T01:14:52.050 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:14:52.051 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:14:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:14:52.084 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T01:14:52.158 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T01:14:52.388 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T01:14:52.410 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:14:52.468 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T01:14:52.486 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:14:52.486 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:14:52.486 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T01:14:52.486 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:14:52.486 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:14:52.486 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:14:52.552 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T01:14:52.659 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:14:52.824 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:14:52.891 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T01:14:52.933 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T01:14:52.965 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T01:14:53.570 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:14:53.611 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:14:53.666 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 124/156 2024-04-25T01:14:53.776 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 125/156 2024-04-25T01:14:53.812 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:14:55.114 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:14:55.156 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:14:56.252 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:14:56.281 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:14:56.432 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:14:56.614 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 129/156 2024-04-25T01:14:56.641 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:14:56.712 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:14:56.712 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:14:56.712 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T01:14:56.713 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:14:56.713 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:14:56.713 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:14:56.776 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:14:56.815 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:14:56.957 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 132/156 2024-04-25T01:14:57.026 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 133/156 2024-04-25T01:14:57.410 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 139/156 2024-04-25T01:14:57.480 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 140/156 2024-04-25T01:14:57.600 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 141/156 2024-04-25T01:14:57.650 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T01:14:57.706 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T01:14:58.062 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 144/156 2024-04-25T01:14:58.066 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T01:14:58.095 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:14:58.125 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T01:14:58.153 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 146/156 2024-04-25T01:14:58.165 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:14:58.165 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:14:58.165 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T01:14:58.166 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:14:58.166 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:14:58.166 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:14:58.237 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:14:58.306 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:14:58.307 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:14:58.307 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T01:14:58.307 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:14:58.380 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:14:58.453 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:14:58.453 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:14:58.453 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T01:14:58.453 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:14:58.453 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:14:58.453 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:14:58.814 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:14:58.886 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:14:58.887 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:14:58.887 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T01:14:58.887 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:14:58.887 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:14:58.887 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:14:59.079 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:14:59.137 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:14:59.192 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:15:00.513 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 148/156 2024-04-25T01:15:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T01:15:00.705 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T01:15:00.892 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T01:15:00.939 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T01:15:00.939 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:15:01.073 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:15:01.121 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:15:01.121 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:15:01.261 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:15:01.392 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:15:01.691 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:15:01.691 INFO:teuthology.orchestra.run.smithi090.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-25T01:15:01.691 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:03.794 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 139/156 2024-04-25T01:15:03.844 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 140/156 2024-04-25T01:15:03.912 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 141/156 2024-04-25T01:15:03.957 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T01:15:04.002 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T01:15:04.326 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 144/156 2024-04-25T01:15:04.383 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T01:15:04.411 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 146/156 2024-04-25T01:15:05.329 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:15:05.401 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:15:05.458 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:15:06.775 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 148/156 2024-04-25T01:15:06.892 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T01:15:06.970 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T01:15:07.139 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T01:15:07.198 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T01:15:07.198 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:15:07.316 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:15:07.363 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:15:07.364 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:15:07.503 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:15:07.637 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:15:07.911 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:15:07.911 INFO:teuthology.orchestra.run.smithi103.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-25T01:15:07.911 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:09.064 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 1/156 2024-04-25T01:15:09.064 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2/156 2024-04-25T01:15:09.064 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 3/156 2024-04-25T01:15:09.064 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 4/156 2024-04-25T01:15:09.064 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 5/156 2024-04-25T01:15:09.064 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 6/156 2024-04-25T01:15:09.064 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 7/156 2024-04-25T01:15:09.064 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:15:09.064 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:15:09.064 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:15:09.064 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 11/156 2024-04-25T01:15:09.065 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 12/156 2024-04-25T01:15:09.065 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:15:09.065 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 14/156 2024-04-25T01:15:09.065 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 15/156 2024-04-25T01:15:09.065 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 16/156 2024-04-25T01:15:09.065 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 17/156 2024-04-25T01:15:09.065 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 18/156 2024-04-25T01:15:09.065 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 19/156 2024-04-25T01:15:09.065 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 20/156 2024-04-25T01:15:09.065 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 21/156 2024-04-25T01:15:09.065 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 22/156 2024-04-25T01:15:09.065 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 23/156 2024-04-25T01:15:09.065 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 24/156 2024-04-25T01:15:09.065 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 25/156 2024-04-25T01:15:09.065 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 26/156 2024-04-25T01:15:09.066 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 27/156 2024-04-25T01:15:09.077 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 28/156 2024-04-25T01:15:09.078 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 29/156 2024-04-25T01:15:09.078 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 30/156 2024-04-25T01:15:09.078 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 31/156 2024-04-25T01:15:09.078 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 32/156 2024-04-25T01:15:09.078 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 33/156 2024-04-25T01:15:09.078 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 34/156 2024-04-25T01:15:09.078 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 35/156 2024-04-25T01:15:09.078 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 36/156 2024-04-25T01:15:09.078 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 37/156 2024-04-25T01:15:09.078 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 38/156 2024-04-25T01:15:09.078 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 39/156 2024-04-25T01:15:09.078 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T01:15:09.078 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T01:15:09.079 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T01:15:09.079 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T01:15:09.079 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T01:15:09.079 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T01:15:09.079 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T01:15:09.079 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T01:15:09.079 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T01:15:09.079 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T01:15:09.079 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T01:15:09.079 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T01:15:09.079 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T01:15:09.079 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T01:15:09.079 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T01:15:09.080 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T01:15:09.080 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T01:15:09.080 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T01:15:09.080 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T01:15:09.080 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T01:15:09.080 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T01:15:09.080 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T01:15:09.080 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T01:15:09.080 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T01:15:09.080 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T01:15:09.080 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T01:15:09.080 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T01:15:09.080 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T01:15:09.081 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T01:15:09.081 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T01:15:09.081 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T01:15:09.081 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T01:15:09.081 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T01:15:09.081 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T01:15:09.081 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T01:15:09.081 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T01:15:09.081 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T01:15:09.081 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T01:15:09.081 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T01:15:09.081 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T01:15:09.081 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T01:15:09.082 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T01:15:09.082 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T01:15:09.082 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T01:15:09.082 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T01:15:09.082 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T01:15:09.082 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T01:15:09.082 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T01:15:09.082 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T01:15:09.082 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T01:15:09.082 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T01:15:09.082 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T01:15:09.082 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T01:15:09.082 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T01:15:09.082 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T01:15:09.083 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T01:15:09.083 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T01:15:09.083 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T01:15:09.083 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T01:15:09.083 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T01:15:09.083 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T01:15:09.083 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T01:15:09.083 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T01:15:09.083 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T01:15:09.083 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T01:15:09.083 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T01:15:09.083 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T01:15:09.083 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T01:15:09.083 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T01:15:09.084 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T01:15:09.084 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T01:15:09.084 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T01:15:09.084 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T01:15:09.084 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T01:15:09.084 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T01:15:09.084 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T01:15:09.084 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T01:15:09.084 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T01:15:09.084 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:15:09.084 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:15:09.084 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T01:15:09.084 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T01:15:09.084 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T01:15:09.085 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T01:15:09.085 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T01:15:09.085 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T01:15:09.085 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T01:15:09.085 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T01:15:09.085 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T01:15:09.085 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T01:15:09.085 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T01:15:09.085 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T01:15:09.085 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T01:15:09.085 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T01:15:09.085 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T01:15:09.085 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T01:15:09.086 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T01:15:09.086 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T01:15:09.086 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T01:15:09.086 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T01:15:09.086 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T01:15:09.086 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T01:15:09.086 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T01:15:09.086 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T01:15:09.086 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T01:15:09.086 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T01:15:09.086 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T01:15:09.086 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T01:15:09.086 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T01:15:09.086 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 149/156 2024-04-25T01:15:09.087 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T01:15:09.087 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 151/156 2024-04-25T01:15:09.087 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T01:15:09.087 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T01:15:09.087 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:15:09.087 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T01:15:11.464 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:15:11.465 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:11.465 INFO:teuthology.orchestra.run.smithi090.stdout:Upgraded: 2024-04-25T01:15:11.465 INFO:teuthology.orchestra.run.smithi090.stdout: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.465 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.465 INFO:teuthology.orchestra.run.smithi090.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T01:15:11.465 INFO:teuthology.orchestra.run.smithi090.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T01:15:11.465 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-04-25T01:15:11.465 INFO:teuthology.orchestra.run.smithi090.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T01:15:11.465 INFO:teuthology.orchestra.run.smithi090.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T01:15:11.465 INFO:teuthology.orchestra.run.smithi090.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T01:15:11.465 INFO:teuthology.orchestra.run.smithi090.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T01:15:11.465 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.465 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.466 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.466 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.466 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:11.466 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.466 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.466 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.466 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:11.466 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:11.466 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:11.466 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:11.466 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:11.466 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.466 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.467 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:11.467 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.467 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.467 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.467 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:11.467 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:11.467 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.467 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.467 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:11.467 INFO:teuthology.orchestra.run.smithi090.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T01:15:11.467 INFO:teuthology.orchestra.run.smithi090.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T01:15:11.467 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T01:15:11.467 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T01:15:11.467 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T01:15:11.468 INFO:teuthology.orchestra.run.smithi090.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T01:15:11.468 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T01:15:11.468 INFO:teuthology.orchestra.run.smithi090.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T01:15:11.468 INFO:teuthology.orchestra.run.smithi090.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T01:15:11.468 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T01:15:11.468 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T01:15:11.468 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.468 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.468 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.468 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T01:15:11.468 INFO:teuthology.orchestra.run.smithi090.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T01:15:11.468 INFO:teuthology.orchestra.run.smithi090.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T01:15:11.468 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T01:15:11.468 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T01:15:11.469 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.469 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.469 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T01:15:11.469 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.469 INFO:teuthology.orchestra.run.smithi090.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T01:15:11.469 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T01:15:11.469 INFO:teuthology.orchestra.run.smithi090.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T01:15:11.469 INFO:teuthology.orchestra.run.smithi090.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T01:15:11.469 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T01:15:11.469 INFO:teuthology.orchestra.run.smithi090.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T01:15:11.469 INFO:teuthology.orchestra.run.smithi090.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T01:15:11.469 INFO:teuthology.orchestra.run.smithi090.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T01:15:11.469 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T01:15:11.469 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T01:15:11.470 INFO:teuthology.orchestra.run.smithi090.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T01:15:11.470 INFO:teuthology.orchestra.run.smithi090.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T01:15:11.470 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T01:15:11.470 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T01:15:11.470 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T01:15:11.470 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T01:15:11.470 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T01:15:11.470 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T01:15:11.470 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T01:15:11.470 INFO:teuthology.orchestra.run.smithi090.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T01:15:11.470 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T01:15:11.470 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T01:15:11.470 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T01:15:11.471 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.471 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.471 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.471 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T01:15:11.471 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T01:15:11.471 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T01:15:11.471 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T01:15:11.471 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T01:15:11.471 INFO:teuthology.orchestra.run.smithi090.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T01:15:11.471 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T01:15:11.471 INFO:teuthology.orchestra.run.smithi090.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T01:15:11.471 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T01:15:11.471 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T01:15:11.471 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T01:15:11.472 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T01:15:11.472 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T01:15:11.472 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T01:15:11.472 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T01:15:11.472 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T01:15:11.472 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T01:15:11.472 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T01:15:11.472 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T01:15:11.472 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T01:15:11.473 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T01:15:11.473 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T01:15:11.473 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T01:15:11.473 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T01:15:11.473 INFO:teuthology.orchestra.run.smithi090.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T01:15:11.473 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T01:15:11.473 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T01:15:11.473 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T01:15:11.474 INFO:teuthology.orchestra.run.smithi090.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T01:15:11.474 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T01:15:11.474 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T01:15:11.474 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T01:15:11.474 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T01:15:11.474 INFO:teuthology.orchestra.run.smithi090.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T01:15:11.474 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T01:15:11.474 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T01:15:11.474 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T01:15:11.474 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T01:15:11.474 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T01:15:11.474 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T01:15:11.474 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.475 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.475 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T01:15:11.475 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T01:15:11.475 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T01:15:11.475 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.475 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T01:15:11.475 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T01:15:11.475 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T01:15:11.475 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T01:15:11.475 INFO:teuthology.orchestra.run.smithi090.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T01:15:11.475 INFO:teuthology.orchestra.run.smithi090.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T01:15:11.475 INFO:teuthology.orchestra.run.smithi090.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T01:15:11.475 INFO:teuthology.orchestra.run.smithi090.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T01:15:11.475 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T01:15:11.476 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T01:15:11.476 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T01:15:11.476 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T01:15:11.476 INFO:teuthology.orchestra.run.smithi090.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T01:15:11.476 INFO:teuthology.orchestra.run.smithi090.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T01:15:11.476 INFO:teuthology.orchestra.run.smithi090.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T01:15:11.476 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.476 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.476 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:11.476 INFO:teuthology.orchestra.run.smithi090.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T01:15:11.476 INFO:teuthology.orchestra.run.smithi090.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T01:15:11.476 INFO:teuthology.orchestra.run.smithi090.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T01:15:11.476 INFO:teuthology.orchestra.run.smithi090.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T01:15:11.476 INFO:teuthology.orchestra.run.smithi090.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T01:15:11.477 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T01:15:11.477 INFO:teuthology.orchestra.run.smithi090.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T01:15:11.477 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:11.477 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-25T01:15:12.123 DEBUG:teuthology.parallel:result is None 2024-04-25T01:15:13.622 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 1/156 2024-04-25T01:15:13.622 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2/156 2024-04-25T01:15:13.622 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 3/156 2024-04-25T01:15:13.622 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 4/156 2024-04-25T01:15:13.623 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 5/156 2024-04-25T01:15:13.623 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 6/156 2024-04-25T01:15:13.623 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 7/156 2024-04-25T01:15:13.623 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:15:13.623 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:15:13.623 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:15:13.623 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 11/156 2024-04-25T01:15:13.623 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 12/156 2024-04-25T01:15:13.623 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:15:13.623 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 14/156 2024-04-25T01:15:13.623 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 15/156 2024-04-25T01:15:13.623 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 16/156 2024-04-25T01:15:13.623 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 17/156 2024-04-25T01:15:13.623 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 18/156 2024-04-25T01:15:13.624 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 19/156 2024-04-25T01:15:13.624 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 20/156 2024-04-25T01:15:13.624 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 21/156 2024-04-25T01:15:13.624 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 22/156 2024-04-25T01:15:13.624 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 23/156 2024-04-25T01:15:13.624 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 24/156 2024-04-25T01:15:13.624 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 25/156 2024-04-25T01:15:13.624 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 26/156 2024-04-25T01:15:13.624 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 27/156 2024-04-25T01:15:13.624 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 28/156 2024-04-25T01:15:13.624 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 29/156 2024-04-25T01:15:13.624 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 30/156 2024-04-25T01:15:13.637 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 31/156 2024-04-25T01:15:13.637 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 32/156 2024-04-25T01:15:13.638 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 33/156 2024-04-25T01:15:13.638 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 34/156 2024-04-25T01:15:13.638 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 35/156 2024-04-25T01:15:13.638 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 36/156 2024-04-25T01:15:13.638 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 37/156 2024-04-25T01:15:13.638 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 38/156 2024-04-25T01:15:13.638 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 39/156 2024-04-25T01:15:13.638 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T01:15:13.639 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T01:15:13.639 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T01:15:13.639 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T01:15:13.639 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T01:15:13.639 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T01:15:13.639 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T01:15:13.639 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T01:15:13.639 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T01:15:13.639 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T01:15:13.639 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T01:15:13.640 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T01:15:13.640 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T01:15:13.640 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T01:15:13.640 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T01:15:13.640 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T01:15:13.640 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T01:15:13.640 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T01:15:13.640 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T01:15:13.640 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T01:15:13.640 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T01:15:13.640 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T01:15:13.640 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T01:15:13.640 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T01:15:13.640 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T01:15:13.641 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T01:15:13.641 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T01:15:13.641 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T01:15:13.641 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T01:15:13.641 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T01:15:13.641 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T01:15:13.641 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T01:15:13.641 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T01:15:13.641 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T01:15:13.641 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T01:15:13.641 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T01:15:13.641 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T01:15:13.641 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T01:15:13.642 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T01:15:13.642 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T01:15:13.642 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T01:15:13.642 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T01:15:13.642 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T01:15:13.642 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T01:15:13.642 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T01:15:13.642 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T01:15:13.642 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T01:15:13.642 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T01:15:13.642 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T01:15:13.643 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T01:15:13.643 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T01:15:13.643 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T01:15:13.643 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T01:15:13.643 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T01:15:13.643 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T01:15:13.643 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T01:15:13.643 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T01:15:13.643 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T01:15:13.643 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T01:15:13.644 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T01:15:13.644 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T01:15:13.644 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T01:15:13.644 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T01:15:13.644 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T01:15:13.644 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T01:15:13.644 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T01:15:13.644 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T01:15:13.644 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T01:15:13.644 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T01:15:13.644 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T01:15:13.644 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T01:15:13.644 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T01:15:13.644 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T01:15:13.645 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T01:15:13.645 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T01:15:13.645 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T01:15:13.645 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T01:15:13.645 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T01:15:13.645 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:15:13.645 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:15:13.645 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T01:15:13.645 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T01:15:13.645 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T01:15:13.645 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T01:15:13.645 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T01:15:13.645 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T01:15:13.645 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T01:15:13.646 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T01:15:13.646 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T01:15:13.646 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T01:15:13.646 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T01:15:13.646 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T01:15:13.646 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T01:15:13.646 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T01:15:13.646 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T01:15:13.646 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T01:15:13.646 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T01:15:13.646 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T01:15:13.646 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T01:15:13.646 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T01:15:13.646 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T01:15:13.647 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T01:15:13.647 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T01:15:13.647 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T01:15:13.647 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T01:15:13.647 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T01:15:13.647 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T01:15:13.647 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T01:15:13.647 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T01:15:13.647 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 149/156 2024-04-25T01:15:13.647 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T01:15:13.647 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 151/156 2024-04-25T01:15:13.647 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T01:15:13.647 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T01:15:13.647 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:15:13.648 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T01:15:16.045 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:15:16.045 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:16.046 INFO:teuthology.orchestra.run.smithi103.stdout:Upgraded: 2024-04-25T01:15:16.046 INFO:teuthology.orchestra.run.smithi103.stdout: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.046 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.046 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T01:15:16.046 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T01:15:16.046 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-25T01:15:16.046 INFO:teuthology.orchestra.run.smithi103.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T01:15:16.046 INFO:teuthology.orchestra.run.smithi103.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T01:15:16.046 INFO:teuthology.orchestra.run.smithi103.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T01:15:16.046 INFO:teuthology.orchestra.run.smithi103.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T01:15:16.046 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.046 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.047 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.047 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.047 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:16.047 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.047 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.047 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.047 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:16.047 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:16.047 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:16.047 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:16.047 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:16.047 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.047 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.047 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:16.048 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.048 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.048 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.048 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:16.048 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:16.048 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.048 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.048 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:15:16.048 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T01:15:16.048 INFO:teuthology.orchestra.run.smithi103.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T01:15:16.048 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T01:15:16.048 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T01:15:16.048 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T01:15:16.048 INFO:teuthology.orchestra.run.smithi103.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T01:15:16.049 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T01:15:16.049 INFO:teuthology.orchestra.run.smithi103.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T01:15:16.049 INFO:teuthology.orchestra.run.smithi103.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T01:15:16.049 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T01:15:16.049 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T01:15:16.049 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.049 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.049 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.049 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T01:15:16.049 INFO:teuthology.orchestra.run.smithi103.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T01:15:16.049 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T01:15:16.049 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T01:15:16.049 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T01:15:16.049 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.050 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.050 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T01:15:16.050 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.050 INFO:teuthology.orchestra.run.smithi103.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T01:15:16.050 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T01:15:16.050 INFO:teuthology.orchestra.run.smithi103.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T01:15:16.051 INFO:teuthology.orchestra.run.smithi103.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T01:15:16.051 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T01:15:16.051 INFO:teuthology.orchestra.run.smithi103.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T01:15:16.051 INFO:teuthology.orchestra.run.smithi103.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T01:15:16.051 INFO:teuthology.orchestra.run.smithi103.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T01:15:16.051 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T01:15:16.051 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T01:15:16.051 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T01:15:16.051 INFO:teuthology.orchestra.run.smithi103.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T01:15:16.051 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T01:15:16.051 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T01:15:16.051 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T01:15:16.051 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T01:15:16.051 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T01:15:16.052 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T01:15:16.052 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T01:15:16.052 INFO:teuthology.orchestra.run.smithi103.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T01:15:16.052 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T01:15:16.052 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T01:15:16.052 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T01:15:16.052 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.052 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.052 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.052 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T01:15:16.052 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T01:15:16.052 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T01:15:16.052 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T01:15:16.053 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T01:15:16.053 INFO:teuthology.orchestra.run.smithi103.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T01:15:16.053 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T01:15:16.053 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T01:15:16.053 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T01:15:16.053 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T01:15:16.053 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T01:15:16.053 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T01:15:16.053 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T01:15:16.053 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T01:15:16.053 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T01:15:16.053 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T01:15:16.053 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T01:15:16.053 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T01:15:16.053 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T01:15:16.054 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T01:15:16.054 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T01:15:16.054 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T01:15:16.054 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T01:15:16.054 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T01:15:16.054 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T01:15:16.054 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T01:15:16.054 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T01:15:16.054 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T01:15:16.054 INFO:teuthology.orchestra.run.smithi103.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T01:15:16.054 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T01:15:16.054 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T01:15:16.054 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T01:15:16.055 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T01:15:16.055 INFO:teuthology.orchestra.run.smithi103.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T01:15:16.055 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T01:15:16.055 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T01:15:16.055 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T01:15:16.055 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T01:15:16.055 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T01:15:16.055 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T01:15:16.055 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.055 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.055 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T01:15:16.055 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T01:15:16.055 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T01:15:16.055 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.056 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T01:15:16.056 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T01:15:16.056 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T01:15:16.056 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T01:15:16.056 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T01:15:16.056 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T01:15:16.056 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T01:15:16.056 INFO:teuthology.orchestra.run.smithi103.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T01:15:16.056 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T01:15:16.056 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T01:15:16.056 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T01:15:16.056 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T01:15:16.056 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T01:15:16.056 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T01:15:16.057 INFO:teuthology.orchestra.run.smithi103.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T01:15:16.057 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.057 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.057 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:15:16.057 INFO:teuthology.orchestra.run.smithi103.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T01:15:16.057 INFO:teuthology.orchestra.run.smithi103.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T01:15:16.057 INFO:teuthology.orchestra.run.smithi103.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T01:15:16.057 INFO:teuthology.orchestra.run.smithi103.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T01:15:16.057 INFO:teuthology.orchestra.run.smithi103.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T01:15:16.057 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T01:15:16.058 INFO:teuthology.orchestra.run.smithi103.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T01:15:16.058 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:16.058 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-25T01:15:16.612 DEBUG:teuthology.parallel:result is None 2024-04-25T01:15:16.613 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:15:16.851 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T01:15:16.908 INFO:teuthology.orchestra.run.smithi090.stdout:19.0.0-2445.gd2c45d97.el9 2024-04-25T01:15:16.909 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445.gd2c45d97.el9 2024-04-25T01:15:16.909 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445.gd2c45d97 is installed. 2024-04-25T01:15:16.911 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:15:17.055 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T01:15:17.111 INFO:teuthology.orchestra.run.smithi103.stdout:19.0.0-2445.gd2c45d97.el9 2024-04-25T01:15:17.112 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445.gd2c45d97.el9 2024-04-25T01:15:17.112 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445.gd2c45d97 is installed. 2024-04-25T01:15:17.114 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T01:15:17.114 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:15:17.114 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T01:15:17.186 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:15:17.186 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T01:15:17.276 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T01:15:17.277 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:15:17.277 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T01:15:17.339 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T01:15:17.447 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:15:17.448 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T01:15:17.516 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T01:15:17.624 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T01:15:17.625 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:15:17.625 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T01:15:17.692 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T01:15:17.797 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:15:17.797 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T01:15:17.863 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T01:15:17.972 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T01:15:17.973 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:15:17.974 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T01:15:18.046 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T01:15:18.164 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:15:18.164 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T01:15:18.235 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T01:15:18.343 INFO:teuthology.run_tasks:Running task ceph... 2024-04-25T01:15:18.440 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-25T01:15:18.440 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 777 /var/log/ceph 2024-04-25T01:15:18.446 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 777 /var/log/ceph 2024-04-25T01:15:18.517 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-25T01:15:18.518 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-25T01:15:18.522 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-25T01:15:18.593 INFO:tasks.ceph:Creating extra log directories... 2024-04-25T01:15:18.593 DEBUG:teuthology.orchestra.run.smithi090:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-25T01:15:18.597 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-25T01:15:18.665 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-04-25T01:15:18.665 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-04-25T01:15:18.665 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-04-25T01:15:18.666 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:15:18.666 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-25T01:15:18.702 DEBUG:teuthology.orchestra.run.smithi090:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-25T01:15:18.898 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-25T01:15:18.971 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:15:18.972 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-25T01:15:19.011 DEBUG:teuthology.orchestra.run.smithi103:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-25T01:15:19.191 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-25T01:15:19.258 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-25T01:15:19.258 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'bad backtrace', 'object missing on disk', 'error reading table object', 'error reading sessionmap', 'unmatched fragstat', 'unmatched rstat', 'was unreadable, recreating it now', 'Scrub error on inode', 'Metadata damage detected', 'MDS_FAILED', 'MDS_DAMAGE'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-25T01:15:19.258 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672309', 'branch': 'squid', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/alternate-pool}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7672309', 'kernel': {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'bad backtrace', 'object missing on disk', 'error reading table object', 'error reading sessionmap', 'unmatched fragstat', 'unmatched rstat', 'was unreadable, recreating it now', 'Scrub error on inode', 'Metadata damage detected', 'MDS_FAILED', 'MDS_DAMAGE'], 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}}, 'kclient': {'syntax': 'v1'}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 8528, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'sleep_before_teardown': 0, 'subset': '122/512', 'suite': 'fs', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi090.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHhKKIWdeKmaEvW2s6Px2CB2wu5+nf+EUnjoYA/bxEDczaUNHNtgxqJdt4l2JR7Wl3nhZNluZNuKdwPXdezSumc=', 'smithi103.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEA6BtYIgnGNys70WvzPevHdJk4Kw8F56Dgwf+M3Il5OcQdD9/MBCQPcBUO8YG/5tH6Ao817SNhuxBOyPkv6YYo='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'bad backtrace', 'object missing on disk', 'error reading table object', 'error reading sessionmap', 'unmatched fragstat', 'unmatched rstat', 'was unreadable, recreating it now', 'Scrub error on inode', 'Metadata damage detected', 'MDS_FAILED', 'MDS_DAMAGE'], 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_recovery_pool']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-24_21:24:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-25T01:15:19.259 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-25T01:15:19.265 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-25T01:15:19.303 DEBUG:teuthology.orchestra.run.smithi090:> sudo install -d -m0777 -- /var/run/ceph 2024-04-25T01:15:19.349 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m0777 -- /var/run/ceph 2024-04-25T01:15:19.413 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:15:19.413 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T01:15:19.466 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T01:15:19.467 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_1 2024-04-25T01:15:19.516 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T01:15:19.516 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:15:19.516 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 6h/6d Inode: 633 Links: 1 2024-04-25T01:15:19.516 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:15:19.516 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:15:19.516 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-04-25 01:15:02.178198779 +0000 2024-04-25T01:15:19.516 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-04-25 01:12:45.679524220 +0000 2024-04-25T01:15:19.516 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-04-25 01:12:45.679524220 +0000 2024-04-25T01:15:19.516 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-04-25 01:12:45.679524220 +0000 2024-04-25T01:15:19.517 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T01:15:19.591 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-04-25T01:15:19.591 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-04-25T01:15:19.591 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.00049159 s, 1.0 MB/s 2024-04-25T01:15:19.595 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T01:15:19.646 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_2 2024-04-25T01:15:19.692 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T01:15:19.693 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:15:19.693 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 6h/6d Inode: 651 Links: 1 2024-04-25T01:15:19.693 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:15:19.693 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:15:19.693 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-04-25 01:15:02.179198764 +0000 2024-04-25T01:15:19.693 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-04-25 01:12:46.285514741 +0000 2024-04-25T01:15:19.693 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-04-25 01:12:46.285514741 +0000 2024-04-25T01:15:19.693 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-04-25 01:12:46.285514741 +0000 2024-04-25T01:15:19.694 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T01:15:19.807 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-04-25T01:15:19.807 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-04-25T01:15:19.807 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000377211 s, 1.4 MB/s 2024-04-25T01:15:19.810 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T01:15:19.865 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_3 2024-04-25T01:15:19.917 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T01:15:19.917 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:15:19.917 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 6h/6d Inode: 672 Links: 1 2024-04-25T01:15:19.918 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:15:19.918 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:15:19.918 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-04-25 01:15:02.179198764 +0000 2024-04-25T01:15:19.918 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-04-25 01:12:46.881505417 +0000 2024-04-25T01:15:19.918 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-04-25 01:12:46.881505417 +0000 2024-04-25T01:15:19.918 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-04-25 01:12:46.881505417 +0000 2024-04-25T01:15:19.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T01:15:20.045 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-04-25T01:15:20.046 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-04-25T01:15:20.046 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000403656 s, 1.3 MB/s 2024-04-25T01:15:20.050 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T01:15:20.121 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_4 2024-04-25T01:15:20.168 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T01:15:20.169 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:15:20.169 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 6h/6d Inode: 688 Links: 1 2024-04-25T01:15:20.169 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:15:20.169 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:15:20.169 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-04-25 01:15:02.180198748 +0000 2024-04-25T01:15:20.169 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-04-25 01:12:47.472496172 +0000 2024-04-25T01:15:20.170 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-04-25 01:12:47.472496172 +0000 2024-04-25T01:15:20.170 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-04-25 01:12:47.472496172 +0000 2024-04-25T01:15:20.170 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T01:15:20.293 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-04-25T01:15:20.294 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-04-25T01:15:20.294 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000364249 s, 1.4 MB/s 2024-04-25T01:15:20.298 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T01:15:20.353 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T01:15:20.353 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:15:20.353 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T01:15:20.406 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T01:15:20.407 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_1 2024-04-25T01:15:20.447 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T01:15:20.448 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:15:20.448 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 631 Links: 1 2024-04-25T01:15:20.448 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:15:20.448 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:15:20.448 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-25 01:15:08.415582874 +0000 2024-04-25T01:15:20.448 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-25 01:12:44.935091361 +0000 2024-04-25T01:15:20.449 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-25 01:12:44.935091361 +0000 2024-04-25T01:15:20.449 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-04-25 01:12:44.935091361 +0000 2024-04-25T01:15:20.449 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T01:15:20.556 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-25T01:15:20.556 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-25T01:15:20.557 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000357515 s, 1.4 MB/s 2024-04-25T01:15:20.563 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T01:15:20.615 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_2 2024-04-25T01:15:20.703 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T01:15:20.703 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:15:20.703 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 652 Links: 1 2024-04-25T01:15:20.703 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:15:20.703 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:15:20.704 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-25 01:15:08.416582855 +0000 2024-04-25T01:15:20.704 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-25 01:12:45.528080002 +0000 2024-04-25T01:15:20.704 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-25 01:12:45.528080002 +0000 2024-04-25T01:15:20.704 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-04-25 01:12:45.528080002 +0000 2024-04-25T01:15:20.704 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T01:15:20.807 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-25T01:15:20.807 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-25T01:15:20.807 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000317611 s, 1.6 MB/s 2024-04-25T01:15:20.809 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T01:15:20.850 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_3 2024-04-25T01:15:20.934 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T01:15:20.934 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:15:20.934 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 670 Links: 1 2024-04-25T01:15:20.934 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:15:20.934 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:15:20.934 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-25 01:15:08.416582855 +0000 2024-04-25T01:15:20.935 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-25 01:12:46.169067723 +0000 2024-04-25T01:15:20.935 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-25 01:12:46.169067723 +0000 2024-04-25T01:15:20.935 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-04-25 01:12:46.169067723 +0000 2024-04-25T01:15:20.935 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T01:15:21.039 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-25T01:15:21.039 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-25T01:15:21.039 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000287788 s, 1.8 MB/s 2024-04-25T01:15:21.042 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T01:15:21.085 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_4 2024-04-25T01:15:21.166 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T01:15:21.167 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:15:21.167 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 689 Links: 1 2024-04-25T01:15:21.167 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:15:21.167 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:15:21.167 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-25 01:15:08.417582836 +0000 2024-04-25T01:15:21.167 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-25 01:12:46.757056460 +0000 2024-04-25T01:15:21.167 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-25 01:12:46.757056460 +0000 2024-04-25T01:15:21.167 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-04-25 01:12:46.757056460 +0000 2024-04-25T01:15:21.168 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T01:15:21.268 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-25T01:15:21.268 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-25T01:15:21.268 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000302489 s, 1.7 MB/s 2024-04-25T01:15:21.271 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T01:15:21.319 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T01:15:21.319 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi090.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi103.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-25T01:15:21.319 INFO:tasks.ceph:Generating config... 2024-04-25T01:15:21.321 INFO:tasks.ceph:[client] client mount timeout = 600 2024-04-25T01:15:21.321 INFO:tasks.ceph:[client] debug client = 20 2024-04-25T01:15:21.321 INFO:tasks.ceph:[client] debug ms = 1 2024-04-25T01:15:21.321 INFO:tasks.ceph:[client] pid file = 2024-04-25T01:15:21.321 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-04-25T01:15:21.321 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-04-25T01:15:21.322 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-04-25T01:15:21.322 INFO:tasks.ceph:[mds] debug mds = 20 2024-04-25T01:15:21.322 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-04-25T01:15:21.322 INFO:tasks.ceph:[mds] debug ms = 1 2024-04-25T01:15:21.322 INFO:tasks.ceph:[mds] mds debug frag = True 2024-04-25T01:15:21.322 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-04-25T01:15:21.322 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-04-25T01:15:21.322 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-04-25T01:15:21.322 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-04-25T01:15:21.322 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-04-25T01:15:21.323 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-04-25T01:15:21.323 INFO:tasks.ceph:[mgr] debug client = 20 2024-04-25T01:15:21.323 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-25T01:15:21.323 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-25T01:15:21.323 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-25T01:15:21.323 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-25T01:15:21.323 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-25T01:15:21.323 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-04-25T01:15:21.323 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-04-25T01:15:21.323 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-25T01:15:21.323 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-25T01:15:21.324 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-25T01:15:21.324 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-25T01:15:21.324 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-25T01:15:21.324 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-25T01:15:21.324 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-25T01:15:21.324 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-25T01:15:21.324 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-25T01:15:21.324 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-25T01:15:21.324 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-25T01:15:21.324 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-25T01:15:21.325 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-04-25T01:15:21.325 INFO:tasks.ceph:Setting up mon.a... 2024-04-25T01:15:21.325 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-25T01:15:21.480 INFO:teuthology.orchestra.run.smithi090.stdout:creating /etc/ceph/ceph.keyring 2024-04-25T01:15:21.489 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-25T01:15:21.615 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T01:15:21.698 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.90'), ('mon.b', '172.21.15.103'), ('mon.c', '[v2:172.21.15.103:3301,v1:172.21.15.103:6790]')] 2024-04-25T01:15:21.699 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.90,172.21.15.103,[v2:172.21.15.103:3301,v1:172.21.15.103:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-04-25T01:15:21.700 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:15:21.700 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-25T01:15:21.757 DEBUG:teuthology.orchestra.run.smithi090:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.90 --add b 172.21.15.103 --addv c '[v2:172.21.15.103:3301,v1:172.21.15.103:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-25T01:15:21.900 INFO:teuthology.orchestra.run.smithi090.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-25T01:15:21.901 INFO:teuthology.orchestra.run.smithi090.stdout:monmaptool: generated fsid e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21 2024-04-25T01:15:21.901 INFO:teuthology.orchestra.run.smithi090.stdout:setting min_mon_release = quincy 2024-04-25T01:15:21.901 INFO:teuthology.orchestra.run.smithi090.stdout:epoch 0 2024-04-25T01:15:21.901 INFO:teuthology.orchestra.run.smithi090.stdout:fsid e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21 2024-04-25T01:15:21.901 INFO:teuthology.orchestra.run.smithi090.stdout:last_changed 2024-04-25T01:15:21.916142+0000 2024-04-25T01:15:21.901 INFO:teuthology.orchestra.run.smithi090.stdout:created 2024-04-25T01:15:21.916142+0000 2024-04-25T01:15:21.902 INFO:teuthology.orchestra.run.smithi090.stdout:min_mon_release 17 (quincy) 2024-04-25T01:15:21.902 INFO:teuthology.orchestra.run.smithi090.stdout:election_strategy: 1 2024-04-25T01:15:21.902 INFO:teuthology.orchestra.run.smithi090.stdout:0: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.a 2024-04-25T01:15:21.902 INFO:teuthology.orchestra.run.smithi090.stdout:1: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.b 2024-04-25T01:15:21.902 INFO:teuthology.orchestra.run.smithi090.stdout:2: [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] mon.c 2024-04-25T01:15:21.903 INFO:teuthology.orchestra.run.smithi090.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-25T01:15:21.904 DEBUG:teuthology.orchestra.run.smithi090:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-25T01:15:21.951 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21... 2024-04-25T01:15:21.953 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T01:15:21.958 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T01:15:22.075 INFO:teuthology.orchestra.run.smithi103.stdout:[global] 2024-04-25T01:15:22.075 INFO:teuthology.orchestra.run.smithi103.stdout: chdir = "" 2024-04-25T01:15:22.076 INFO:teuthology.orchestra.run.smithi103.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T01:15:22.076 INFO:teuthology.orchestra.run.smithi103.stdout: auth supported = cephx 2024-04-25T01:15:22.076 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:22.076 INFO:teuthology.orchestra.run.smithi103.stdout: filestore xattr use omap = true 2024-04-25T01:15:22.076 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:22.076 INFO:teuthology.orchestra.run.smithi103.stdout: mon clock drift allowed = 1.000 2024-04-25T01:15:22.076 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:22.076 INFO:teuthology.orchestra.run.smithi103.stdout: osd crush chooseleaf type = 0 2024-04-25T01:15:22.076 INFO:teuthology.orchestra.run.smithi103.stdout: auth debug = true 2024-04-25T01:15:22.076 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:22.076 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on old message = true 2024-04-25T01:15:22.077 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on bug = true 2024-04-25T01:15:22.077 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:22.077 INFO:teuthology.orchestra.run.smithi103.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T01:15:22.077 INFO:teuthology.orchestra.run.smithi103.stdout: mon pg warn max object skew = 0 2024-04-25T01:15:22.077 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:22.077 INFO:teuthology.orchestra.run.smithi103.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T01:15:22.077 INFO:teuthology.orchestra.run.smithi103.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T01:15:22.077 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:22.077 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default size = 2 2024-04-25T01:15:22.077 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:22.077 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd allow primary affinity = true 2024-04-25T01:15:22.077 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd allow pg remap = true 2024-04-25T01:15:22.077 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on legacy crush tunables = false 2024-04-25T01:15:22.077 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on crush straw calc version zero = false 2024-04-25T01:15:22.078 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on no sortbitwise = false 2024-04-25T01:15:22.078 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on osd down out interval zero = false 2024-04-25T01:15:22.078 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on too few osds = false 2024-04-25T01:15:22.078 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T01:15:22.078 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T01:15:22.078 INFO:teuthology.orchestra.run.smithi103.stdout: mon_allow_pool_size_one = true 2024-04-25T01:15:22.078 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:22.078 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T01:15:22.078 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:22.078 INFO:teuthology.orchestra.run.smithi103.stdout: osd default data pool replay window = 5 2024-04-25T01:15:22.078 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:22.078 INFO:teuthology.orchestra.run.smithi103.stdout: mon allow pool delete = true 2024-04-25T01:15:22.079 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:22.079 INFO:teuthology.orchestra.run.smithi103.stdout: mon cluster log file level = debug 2024-04-25T01:15:22.079 INFO:teuthology.orchestra.run.smithi103.stdout: debug asserts on shutdown = true 2024-04-25T01:15:22.079 INFO:teuthology.orchestra.run.smithi103.stdout: mon health detail to clog = false 2024-04-25T01:15:22.079 INFO:teuthology.orchestra.run.smithi103.stdout: mon host = "172.21.15.90,172.21.15.103,[v2:172.21.15.103:3301,v1:172.21.15.103:6790]" 2024-04-25T01:15:22.079 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on skipped message = False 2024-04-25T01:15:22.079 INFO:teuthology.orchestra.run.smithi103.stdout: fsid = e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21 2024-04-25T01:15:22.079 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:22.079 INFO:teuthology.orchestra.run.smithi103.stdout:[osd] 2024-04-25T01:15:22.079 INFO:teuthology.orchestra.run.smithi103.stdout: osd journal size = 100 2024-04-25T01:15:22.079 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:22.079 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub load threshold = 5.0 2024-04-25T01:15:22.080 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub max interval = 600 2024-04-25T01:15:22.080 INFO:teuthology.orchestra.run.smithi103.stdout: osd mclock profile = high_recovery_ops 2024-04-25T01:15:22.080 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:22.080 INFO:teuthology.orchestra.run.smithi103.stdout: osd recover clone overlap = true 2024-04-25T01:15:22.080 INFO:teuthology.orchestra.run.smithi103.stdout: osd recovery max chunk = 1048576 2024-04-25T01:15:22.080 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:22.080 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug shutdown = true 2024-04-25T01:15:22.080 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug op order = true 2024-04-25T01:15:22.080 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug verify stray on activate = true 2024-04-25T01:15:22.080 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug trim objects = true 2024-04-25T01:15:22.080 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:22.080 INFO:teuthology.orchestra.run.smithi103.stdout: osd open classes on start = true 2024-04-25T01:15:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug pg log writeout = true 2024-04-25T01:15:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: osd deep scrub update digest min age = 30 2024-04-25T01:15:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: osd map max advance = 10 2024-04-25T01:15:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: journal zero on create = true 2024-04-25T01:15:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: filestore ondisk finisher threads = 3 2024-04-25T01:15:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: filestore apply finisher threads = 3 2024-04-25T01:15:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: bdev debug aio = true 2024-04-25T01:15:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug misdirected ops = true 2024-04-25T01:15:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore block size = 96636764160 2024-04-25T01:15:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore fsck on mount = True 2024-04-25T01:15:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: debug bluefs = 20 2024-04-25T01:15:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: debug bluestore = 20 2024-04-25T01:15:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-25T01:15:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: debug osd = 20 2024-04-25T01:15:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: debug rocksdb = 10 2024-04-25T01:15:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T01:15:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd full ratio = 0.9 2024-04-25T01:15:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd nearfull ratio = 0.8 2024-04-25T01:15:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: osd failsafe full ratio = 0.95 2024-04-25T01:15:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: osd objectstore = bluestore 2024-04-25T01:15:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: osd op complaint time = 180 2024-04-25T01:15:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:22.082 INFO:teuthology.orchestra.run.smithi103.stdout:[mgr] 2024-04-25T01:15:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-25T01:15:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: debug mgr = 20 2024-04-25T01:15:22.083 INFO:teuthology.orchestra.run.smithi103.stdout: debug mon = 20 2024-04-25T01:15:22.083 INFO:teuthology.orchestra.run.smithi103.stdout: debug auth = 20 2024-04-25T01:15:22.083 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min pgs per osd = 4 2024-04-25T01:15:22.083 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min bytes per osd = 10 2024-04-25T01:15:22.083 INFO:teuthology.orchestra.run.smithi103.stdout: mgr/telemetry/nag = false 2024-04-25T01:15:22.083 INFO:teuthology.orchestra.run.smithi103.stdout: debug client = 20 2024-04-25T01:15:22.083 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:22.083 INFO:teuthology.orchestra.run.smithi103.stdout:[mon] 2024-04-25T01:15:22.083 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-25T01:15:22.083 INFO:teuthology.orchestra.run.smithi103.stdout: debug mon = 20 2024-04-25T01:15:22.083 INFO:teuthology.orchestra.run.smithi103.stdout: debug paxos = 20 2024-04-25T01:15:22.083 INFO:teuthology.orchestra.run.smithi103.stdout: debug auth = 20 2024-04-25T01:15:22.083 INFO:teuthology.orchestra.run.smithi103.stdout: mon data avail warn = 5 2024-04-25T01:15:22.083 INFO:teuthology.orchestra.run.smithi103.stdout: mon mgr mkfs grace = 240 2024-04-25T01:15:22.083 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min pgs per osd = 4 2024-04-25T01:15:22.084 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd reporter subtree level = osd 2024-04-25T01:15:22.084 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd prime pg temp = true 2024-04-25T01:15:22.084 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min bytes per osd = 10 2024-04-25T01:15:22.084 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:22.084 INFO:teuthology.orchestra.run.smithi103.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T01:15:22.084 INFO:teuthology.orchestra.run.smithi103.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T01:15:22.084 INFO:teuthology.orchestra.run.smithi103.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T01:15:22.084 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:22.084 INFO:teuthology.orchestra.run.smithi103.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T01:15:22.084 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T01:15:22.084 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T01:15:22.084 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:22.085 INFO:teuthology.orchestra.run.smithi103.stdout: # 1m isn't quite enough 2024-04-25T01:15:22.085 INFO:teuthology.orchestra.run.smithi103.stdout: mon_down_mkfs_grace = 2m 2024-04-25T01:15:22.085 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:22.085 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_filestore_osds = false 2024-04-25T01:15:22.085 INFO:teuthology.orchestra.run.smithi103.stdout: mon down mkfs grace = 300 2024-04-25T01:15:22.085 INFO:teuthology.orchestra.run.smithi103.stdout: mon op complaint time = 120 2024-04-25T01:15:22.085 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:15:22.085 INFO:teuthology.orchestra.run.smithi103.stdout:[client] 2024-04-25T01:15:22.085 INFO:teuthology.orchestra.run.smithi103.stdout: rgw cache enabled = true 2024-04-25T01:15:22.085 INFO:teuthology.orchestra.run.smithi103.stdout: rgw enable ops log = true 2024-04-25T01:15:22.085 INFO:teuthology.orchestra.run.smithi103.stdout: rgw enable usage log = true 2024-04-25T01:15:22.085 INFO:teuthology.orchestra.run.smithi103.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T01:15:22.085 INFO:teuthology.orchestra.run.smithi103.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T01:15:22.085 INFO:teuthology.orchestra.run.smithi103.stdout: client mount timeout = 600 2024-04-25T01:15:22.085 INFO:teuthology.orchestra.run.smithi103.stdout: debug client = 20 2024-04-25T01:15:22.086 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-25T01:15:22.086 INFO:teuthology.orchestra.run.smithi103.stdout: pid file = "" 2024-04-25T01:15:22.086 INFO:teuthology.orchestra.run.smithi103.stdout: rados mon op timeout = 900 2024-04-25T01:15:22.086 INFO:teuthology.orchestra.run.smithi103.stdout: rados osd op timeout = 900 2024-04-25T01:15:22.086 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.a] 2024-04-25T01:15:22.086 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.b] 2024-04-25T01:15:22.086 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.c] 2024-04-25T01:15:22.086 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.a] 2024-04-25T01:15:22.086 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.c] 2024-04-25T01:15:22.086 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.b] 2024-04-25T01:15:22.086 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.d] 2024-04-25T01:15:22.086 INFO:teuthology.orchestra.run.smithi103.stdout:[mds] 2024-04-25T01:15:22.086 INFO:teuthology.orchestra.run.smithi103.stdout: debug mds = 20 2024-04-25T01:15:22.087 INFO:teuthology.orchestra.run.smithi103.stdout: debug mds balancer = 20 2024-04-25T01:15:22.087 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-25T01:15:22.087 INFO:teuthology.orchestra.run.smithi103.stdout: mds debug frag = True 2024-04-25T01:15:22.087 INFO:teuthology.orchestra.run.smithi103.stdout: mds debug scatterstat = True 2024-04-25T01:15:22.087 INFO:teuthology.orchestra.run.smithi103.stdout: mds op complaint time = 180 2024-04-25T01:15:22.087 INFO:teuthology.orchestra.run.smithi103.stdout: mds verify scatter = True 2024-04-25T01:15:22.087 INFO:teuthology.orchestra.run.smithi103.stdout: osd op complaint time = 180 2024-04-25T01:15:22.087 INFO:teuthology.orchestra.run.smithi103.stdout: rados mon op timeout = 900 2024-04-25T01:15:22.087 INFO:teuthology.orchestra.run.smithi103.stdout: rados osd op timeout = 900 2024-04-25T01:15:22.088 INFO:teuthology.orchestra.run.smithi090.stdout:[global] 2024-04-25T01:15:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: chdir = "" 2024-04-25T01:15:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T01:15:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: auth supported = cephx 2024-04-25T01:15:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: filestore xattr use omap = true 2024-04-25T01:15:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: mon clock drift allowed = 1.000 2024-04-25T01:15:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: osd crush chooseleaf type = 0 2024-04-25T01:15:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: auth debug = true 2024-04-25T01:15:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: ms die on old message = true 2024-04-25T01:15:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: ms die on bug = true 2024-04-25T01:15:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T01:15:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: mon pg warn max object skew = 0 2024-04-25T01:15:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T01:15:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T01:15:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: osd pool default size = 2 2024-04-25T01:15:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd allow primary affinity = true 2024-04-25T01:15:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd allow pg remap = true 2024-04-25T01:15:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on legacy crush tunables = false 2024-04-25T01:15:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on crush straw calc version zero = false 2024-04-25T01:15:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on no sortbitwise = false 2024-04-25T01:15:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on osd down out interval zero = false 2024-04-25T01:15:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on too few osds = false 2024-04-25T01:15:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T01:15:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T01:15:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: mon_allow_pool_size_one = true 2024-04-25T01:15:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T01:15:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: osd default data pool replay window = 5 2024-04-25T01:15:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: mon allow pool delete = true 2024-04-25T01:15:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: mon cluster log file level = debug 2024-04-25T01:15:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: debug asserts on shutdown = true 2024-04-25T01:15:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: mon health detail to clog = false 2024-04-25T01:15:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: mon host = "172.21.15.90,172.21.15.103,[v2:172.21.15.103:3301,v1:172.21.15.103:6790]" 2024-04-25T01:15:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: ms die on skipped message = False 2024-04-25T01:15:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: fsid = e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21 2024-04-25T01:15:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:22.091 INFO:teuthology.orchestra.run.smithi090.stdout:[osd] 2024-04-25T01:15:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: osd journal size = 100 2024-04-25T01:15:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: osd scrub load threshold = 5.0 2024-04-25T01:15:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: osd scrub max interval = 600 2024-04-25T01:15:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: osd mclock profile = high_recovery_ops 2024-04-25T01:15:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: osd recover clone overlap = true 2024-04-25T01:15:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: osd recovery max chunk = 1048576 2024-04-25T01:15:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug shutdown = true 2024-04-25T01:15:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug op order = true 2024-04-25T01:15:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug verify stray on activate = true 2024-04-25T01:15:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug trim objects = true 2024-04-25T01:15:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: osd open classes on start = true 2024-04-25T01:15:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug pg log writeout = true 2024-04-25T01:15:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: osd deep scrub update digest min age = 30 2024-04-25T01:15:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: osd map max advance = 10 2024-04-25T01:15:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: journal zero on create = true 2024-04-25T01:15:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: filestore ondisk finisher threads = 3 2024-04-25T01:15:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: filestore apply finisher threads = 3 2024-04-25T01:15:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: bdev debug aio = true 2024-04-25T01:15:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug misdirected ops = true 2024-04-25T01:15:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: bluestore block size = 96636764160 2024-04-25T01:15:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: bluestore fsck on mount = True 2024-04-25T01:15:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: debug bluefs = 20 2024-04-25T01:15:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: debug bluestore = 20 2024-04-25T01:15:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-04-25T01:15:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: debug osd = 20 2024-04-25T01:15:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: debug rocksdb = 10 2024-04-25T01:15:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T01:15:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd full ratio = 0.9 2024-04-25T01:15:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd nearfull ratio = 0.8 2024-04-25T01:15:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: osd failsafe full ratio = 0.95 2024-04-25T01:15:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: osd objectstore = bluestore 2024-04-25T01:15:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: osd op complaint time = 180 2024-04-25T01:15:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:22.094 INFO:teuthology.orchestra.run.smithi090.stdout:[mgr] 2024-04-25T01:15:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-04-25T01:15:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: debug mgr = 20 2024-04-25T01:15:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: debug mon = 20 2024-04-25T01:15:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: debug auth = 20 2024-04-25T01:15:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min pgs per osd = 4 2024-04-25T01:15:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min bytes per osd = 10 2024-04-25T01:15:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: mgr/telemetry/nag = false 2024-04-25T01:15:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: debug client = 20 2024-04-25T01:15:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:22.095 INFO:teuthology.orchestra.run.smithi090.stdout:[mon] 2024-04-25T01:15:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-04-25T01:15:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: debug mon = 20 2024-04-25T01:15:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: debug paxos = 20 2024-04-25T01:15:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: debug auth = 20 2024-04-25T01:15:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: mon data avail warn = 5 2024-04-25T01:15:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: mon mgr mkfs grace = 240 2024-04-25T01:15:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min pgs per osd = 4 2024-04-25T01:15:22.096 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd reporter subtree level = osd 2024-04-25T01:15:22.096 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd prime pg temp = true 2024-04-25T01:15:22.096 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min bytes per osd = 10 2024-04-25T01:15:22.096 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T01:15:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T01:15:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T01:15:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T01:15:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T01:15:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T01:15:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: # 1m isn't quite enough 2024-04-25T01:15:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: mon_down_mkfs_grace = 2m 2024-04-25T01:15:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_filestore_osds = false 2024-04-25T01:15:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: mon down mkfs grace = 300 2024-04-25T01:15:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: mon op complaint time = 120 2024-04-25T01:15:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:15:22.099 INFO:teuthology.orchestra.run.smithi090.stdout:[client] 2024-04-25T01:15:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: rgw cache enabled = true 2024-04-25T01:15:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: rgw enable ops log = true 2024-04-25T01:15:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: rgw enable usage log = true 2024-04-25T01:15:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T01:15:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T01:15:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: client mount timeout = 600 2024-04-25T01:15:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: debug client = 20 2024-04-25T01:15:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-04-25T01:15:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: pid file = "" 2024-04-25T01:15:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: rados mon op timeout = 900 2024-04-25T01:15:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: rados osd op timeout = 900 2024-04-25T01:15:22.100 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.a] 2024-04-25T01:15:22.100 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.b] 2024-04-25T01:15:22.100 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.c] 2024-04-25T01:15:22.100 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.a] 2024-04-25T01:15:22.100 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.c] 2024-04-25T01:15:22.100 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.b] 2024-04-25T01:15:22.100 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.d] 2024-04-25T01:15:22.100 INFO:teuthology.orchestra.run.smithi090.stdout:[mds] 2024-04-25T01:15:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: debug mds = 20 2024-04-25T01:15:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: debug mds balancer = 20 2024-04-25T01:15:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-04-25T01:15:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: mds debug frag = True 2024-04-25T01:15:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: mds debug scatterstat = True 2024-04-25T01:15:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: mds op complaint time = 180 2024-04-25T01:15:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: mds verify scatter = True 2024-04-25T01:15:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: osd op complaint time = 180 2024-04-25T01:15:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: rados mon op timeout = 900 2024-04-25T01:15:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: rados osd op timeout = 900 2024-04-25T01:15:22.117 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-25T01:15:22.117 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-25T01:15:22.268 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-25T01:15:22.268 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:15:22.268 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-25T01:15:22.321 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:15:22.321 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-25T01:15:22.376 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi090.front.sepia.ceph.com 2024-04-25T01:15:22.376 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:15:22.377 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-25T01:15:22.377 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T01:15:22.534 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:15:22.534 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-25T01:15:22.634 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi103.front.sepia.ceph.com 2024-04-25T01:15:22.634 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:15:22.634 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-25T01:15:22.635 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T01:15:22.729 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:15:22.729 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-25T01:15:22.812 INFO:tasks.ceph:Setting up mon nodes... 2024-04-25T01:15:22.812 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-25T01:15:22.812 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-25T01:15:22.977 INFO:teuthology.orchestra.run.smithi090.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-25T01:15:22.988 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-25T01:15:23.162 INFO:teuthology.orchestra.run.smithi103.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-25T01:15:23.172 INFO:tasks.ceph:Setting up mds nodes... 2024-04-25T01:15:23.172 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-04-25T01:15:23.326 INFO:teuthology.orchestra.run.smithi090.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-04-25T01:15:23.335 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-04-25T01:15:23.411 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-04-25T01:15:23.558 INFO:teuthology.orchestra.run.smithi090.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-04-25T01:15:23.567 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-04-25T01:15:23.642 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-04-25T01:15:23.804 INFO:teuthology.orchestra.run.smithi103.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-04-25T01:15:23.811 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-04-25T01:15:23.877 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-04-25T01:15:24.007 INFO:teuthology.orchestra.run.smithi103.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-04-25T01:15:24.014 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-04-25T01:15:24.102 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-25T01:15:24.102 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-25T01:15:24.222 INFO:teuthology.orchestra.run.smithi090.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-25T01:15:24.261 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-04-25T01:15:24.376 INFO:teuthology.orchestra.run.smithi090.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-04-25T01:15:24.419 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-25T01:15:24.538 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-25T01:15:24.581 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-04-25T01:15:24.694 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-04-25T01:15:24.743 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-25T01:15:24.743 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi090.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi103.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-25T01:15:24.744 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-25T01:15:24.831 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T01:15:24.831 INFO:tasks.ceph:role: osd.0 2024-04-25T01:15:24.831 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi090.front.sepia.ceph.com 2024-04-25T01:15:24.831 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-25T01:15:24.917 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T01:15:24.917 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T01:15:24.917 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T01:15:24.917 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T01:15:24.917 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T01:15:24.917 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2024-04-25T01:15:24.917 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T01:15:24.917 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T01:15:24.917 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T01:15:24.917 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T01:15:25.331 INFO:teuthology.orchestra.run.smithi090.stdout:Discarding blocks...Done. 2024-04-25T01:15:25.337 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2024-04-25T01:15:25.337 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-25T01:15:25.519 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-25T01:15:25.615 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-25T01:15:25.684 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T01:15:25.685 INFO:tasks.ceph:role: osd.1 2024-04-25T01:15:25.685 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi090.front.sepia.ceph.com 2024-04-25T01:15:25.685 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-25T01:15:25.766 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T01:15:25.767 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T01:15:25.767 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T01:15:25.767 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T01:15:25.767 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T01:15:25.767 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2024-04-25T01:15:25.767 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T01:15:25.767 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T01:15:25.767 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T01:15:25.767 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T01:15:26.120 INFO:teuthology.orchestra.run.smithi090.stdout:Discarding blocks...Done. 2024-04-25T01:15:26.125 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2024-04-25T01:15:26.126 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-25T01:15:26.234 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-25T01:15:26.307 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-25T01:15:26.374 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T01:15:26.374 INFO:tasks.ceph:role: osd.2 2024-04-25T01:15:26.374 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi090.front.sepia.ceph.com 2024-04-25T01:15:26.374 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-25T01:15:26.446 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T01:15:26.447 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T01:15:26.447 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T01:15:26.447 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T01:15:26.447 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T01:15:26.447 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2024-04-25T01:15:26.447 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T01:15:26.447 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T01:15:26.447 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T01:15:26.447 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T01:15:26.837 INFO:teuthology.orchestra.run.smithi090.stdout:Discarding blocks...Done. 2024-04-25T01:15:26.843 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2024-04-25T01:15:26.843 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-25T01:15:26.970 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-25T01:15:27.039 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-25T01:15:27.101 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T01:15:27.102 INFO:tasks.ceph:role: osd.3 2024-04-25T01:15:27.102 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi090.front.sepia.ceph.com 2024-04-25T01:15:27.102 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-25T01:15:27.172 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T01:15:27.172 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T01:15:27.172 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T01:15:27.172 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T01:15:27.172 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T01:15:27.172 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2024-04-25T01:15:27.172 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T01:15:27.172 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T01:15:27.172 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T01:15:27.172 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T01:15:27.574 INFO:teuthology.orchestra.run.smithi090.stdout:Discarding blocks...Done. 2024-04-25T01:15:27.580 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2024-04-25T01:15:27.580 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-25T01:15:27.729 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-25T01:15:27.814 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T01:15:27.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:15:27.971+0000 7f757d0e8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-25T01:15:27.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:15:27.971+0000 7f757d0e8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-25T01:15:27.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:15:27.971+0000 7f757d0e8640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T01:15:27.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:15:27.972+0000 7f757d0e8640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T01:15:27.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:15:27.972+0000 7f757d0e8640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T01:15:27.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:15:27.976+0000 7f757d0e8640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-25T01:15:31.212 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-25T01:15:31.299 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T01:15:31.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:15:31.446+0000 7ffb3f433640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-25T01:15:31.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:15:31.447+0000 7ffb3f433640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-25T01:15:31.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:15:31.447+0000 7ffb3f433640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T01:15:31.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:15:31.447+0000 7ffb3f433640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T01:15:31.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:15:31.447+0000 7ffb3f433640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T01:15:31.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:15:31.449+0000 7ffb3f433640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-25T01:15:34.659 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-25T01:15:34.739 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T01:15:34.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:15:34.876+0000 7fb10ef33640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-25T01:15:34.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:15:34.877+0000 7fb10ef33640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-25T01:15:34.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:15:34.877+0000 7fb10ef33640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T01:15:34.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:15:34.877+0000 7fb10ef33640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T01:15:34.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:15:34.877+0000 7fb10ef33640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T01:15:34.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:15:34.879+0000 7fb10ef33640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-25T01:15:37.823 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-25T01:15:37.913 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T01:15:38.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:15:38.056+0000 7f4c2b40d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-25T01:15:38.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:15:38.057+0000 7f4c2b40d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-25T01:15:38.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:15:38.057+0000 7f4c2b40d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T01:15:38.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:15:38.057+0000 7f4c2b40d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T01:15:38.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:15:38.057+0000 7f4c2b40d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T01:15:38.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:15:38.060+0000 7f4c2b40d640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-25T01:15:41.016 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-25T01:15:41.101 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-25T01:15:41.193 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T01:15:41.193 INFO:tasks.ceph:role: osd.4 2024-04-25T01:15:41.193 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-25T01:15:41.193 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-25T01:15:41.284 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T01:15:41.284 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T01:15:41.284 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T01:15:41.284 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T01:15:41.284 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T01:15:41.284 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-04-25T01:15:41.284 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T01:15:41.284 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T01:15:41.285 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T01:15:41.285 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T01:15:41.702 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-04-25T01:15:41.708 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-04-25T01:15:41.708 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-25T01:15:41.863 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-25T01:15:41.940 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-25T01:15:42.010 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T01:15:42.010 INFO:tasks.ceph:role: osd.5 2024-04-25T01:15:42.010 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-25T01:15:42.010 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-25T01:15:42.082 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T01:15:42.082 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T01:15:42.082 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T01:15:42.082 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T01:15:42.082 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T01:15:42.083 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-04-25T01:15:42.083 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T01:15:42.083 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T01:15:42.083 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T01:15:42.083 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T01:15:42.435 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-04-25T01:15:42.440 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-04-25T01:15:42.440 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-25T01:15:42.533 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-25T01:15:42.598 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-25T01:15:42.657 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T01:15:42.658 INFO:tasks.ceph:role: osd.6 2024-04-25T01:15:42.658 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-25T01:15:42.658 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-25T01:15:42.732 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T01:15:42.732 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T01:15:42.732 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T01:15:42.732 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T01:15:42.732 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T01:15:42.732 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-04-25T01:15:42.733 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T01:15:42.733 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T01:15:42.733 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T01:15:42.733 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T01:15:43.118 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-04-25T01:15:43.124 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-04-25T01:15:43.125 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-25T01:15:43.236 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-25T01:15:43.306 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-25T01:15:43.366 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T01:15:43.366 INFO:tasks.ceph:role: osd.7 2024-04-25T01:15:43.366 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-25T01:15:43.366 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-25T01:15:43.438 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T01:15:43.439 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T01:15:43.439 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T01:15:43.439 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T01:15:43.439 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T01:15:43.439 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-04-25T01:15:43.439 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T01:15:43.439 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T01:15:43.439 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T01:15:43.439 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T01:15:43.849 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-04-25T01:15:43.854 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-04-25T01:15:43.855 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-25T01:15:43.997 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-25T01:15:44.078 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T01:15:44.207 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:15:44.222+0000 7f09ee618640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-25T01:15:44.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:15:44.226+0000 7f09ee618640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-25T01:15:44.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:15:44.226+0000 7f09ee618640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-25T01:15:44.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:15:44.226+0000 7f09ee618640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-25T01:15:44.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:15:44.226+0000 7f09ee618640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-25T01:15:44.222 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:15:44.237+0000 7f09ee618640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-25T01:15:47.188 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-25T01:15:47.278 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T01:15:47.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:15:47.421+0000 7f64c309b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-25T01:15:47.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:15:47.422+0000 7f64c309b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-25T01:15:47.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:15:47.422+0000 7f64c309b640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-25T01:15:47.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:15:47.422+0000 7f64c309b640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-25T01:15:47.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:15:47.422+0000 7f64c309b640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-25T01:15:47.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:15:47.424+0000 7f64c309b640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-25T01:15:49.260 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:15:49.268 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:15:50.664 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-25T01:15:50.746 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T01:15:50.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:15:50.889+0000 7fd120d01640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-25T01:15:50.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:15:50.889+0000 7fd120d01640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-25T01:15:50.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:15:50.889+0000 7fd120d01640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-25T01:15:50.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:15:50.889+0000 7fd120d01640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-25T01:15:50.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:15:50.889+0000 7fd120d01640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-25T01:15:50.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:15:50.891+0000 7fd120d01640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-25T01:15:53.863 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-25T01:15:53.950 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T01:15:54.083 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:15:54.098+0000 7fbaa4d68640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-25T01:15:54.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:15:54.098+0000 7fbaa4d68640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-25T01:15:54.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:15:54.098+0000 7fbaa4d68640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-25T01:15:54.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:15:54.098+0000 7fbaa4d68640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-25T01:15:54.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:15:54.098+0000 7fbaa4d68640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-25T01:15:54.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:15:54.101+0000 7fbaa4d68640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-25T01:15:57.079 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-25T01:15:57.164 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-25T01:15:57.164 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:15:57.164 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-25T01:15:57.254 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:15:57.254 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-04-25T01:15:57.338 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:15:57.338 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-04-25T01:15:57.422 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:15:57.423 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-25T01:15:57.505 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:15:57.505 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-25T01:15:57.586 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:15:57.587 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-25T01:15:57.714 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:15:57.714 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-25T01:15:57.798 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:15:57.798 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-25T01:15:57.893 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:15:57.894 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-04-25T01:15:58.030 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:15:58.030 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-04-25T01:15:58.110 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:15:58.110 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-25T01:15:58.189 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:15:58.189 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-25T01:15:58.316 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:15:58.316 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-25T01:15:58.396 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:15:58.396 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-25T01:15:58.474 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:15:58.474 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-25T01:15:58.530 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:15:58.530 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-04-25T01:15:58.582 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:15:58.582 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-25T01:15:58.640 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:15:58.640 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-04-25T01:15:58.691 INFO:tasks.ceph:Adding keys to all mons... 2024-04-25T01:15:58.691 DEBUG:teuthology.orchestra.run.smithi090:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-25T01:15:58.699 DEBUG:teuthology.orchestra.run.smithi103:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-25T01:15:58.773 INFO:teuthology.orchestra.run.smithi090.stdout:[mgr.y] 2024-04-25T01:15:58.773 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQCqrilm53U1OxAAV/IYzYKURT+G44pUIEkejg== 2024-04-25T01:15:58.773 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.a] 2024-04-25T01:15:58.773 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQCrrilmjb9qFBAAlnPjHsgFagy1je4+qyjyOQ== 2024-04-25T01:15:58.773 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.c] 2024-04-25T01:15:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQCrrilmmi9BIhAA2s5UuqgNa1mTnzELh6kg1Q== 2024-04-25T01:15:58.774 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.0] 2024-04-25T01:15:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQCvrilmONAAOhAAMfwg6+ChCofN/OS6Jp7DLg== 2024-04-25T01:15:58.774 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.1] 2024-04-25T01:15:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQCzrilmaKqpGhAAMFnbJ9tBiOgP7aUcxHxdAQ== 2024-04-25T01:15:58.774 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.2] 2024-04-25T01:15:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQC2rilm9bRPNBAAh9uf5qeKxKVGZjeURZN2ng== 2024-04-25T01:15:58.774 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.3] 2024-04-25T01:15:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQC6rilm335uAxAAzL5VZalis0ydEKCAc9p20w== 2024-04-25T01:15:58.774 INFO:teuthology.orchestra.run.smithi090.stdout:[mgr.x] 2024-04-25T01:15:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQCrrilm5IGgChAALBQG+sk4Pr+UuJWnHXh/ZA== 2024-04-25T01:15:58.774 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.b] 2024-04-25T01:15:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQCrrilmUZHfMBAAbh51emKl6HSLVHIaUOllXA== 2024-04-25T01:15:58.774 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.d] 2024-04-25T01:15:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQCsrilm/zZiARAAP8D+OSj0K9A3NGjnI+NOLA== 2024-04-25T01:15:58.775 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.4] 2024-04-25T01:15:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDArilmkRNZDRAAPLVSDr1+ERL09nIchRCNVw== 2024-04-25T01:15:58.775 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.5] 2024-04-25T01:15:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDDrilmM3k5GRAA6toW36myIctvYTYY5kzayA== 2024-04-25T01:15:58.775 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.6] 2024-04-25T01:15:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDGrilmZJEXNRAARn43EOUZQzSdNDmifLDQpg== 2024-04-25T01:15:58.775 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.7] 2024-04-25T01:15:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDKrilm60XzBRAAz37R48OIHSB4petorwdLDQ== 2024-04-25T01:15:58.775 INFO:teuthology.orchestra.run.smithi090.stdout:[client.0] 2024-04-25T01:15:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQCsrilmTsovDhAAVqcIHshdGrSHncSLyuueoQ== 2024-04-25T01:15:58.775 INFO:teuthology.orchestra.run.smithi090.stdout:[client.2] 2024-04-25T01:15:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQCsrilmLMVnFxAAQv8hYmJ1f8WsFc4+C6QFWg== 2024-04-25T01:15:58.775 INFO:teuthology.orchestra.run.smithi090.stdout:[client.1] 2024-04-25T01:15:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQCsrilmUIgRIRAAmTYrl9YlFG/l6hAHrMhYAQ== 2024-04-25T01:15:58.776 INFO:teuthology.orchestra.run.smithi090.stdout:[client.3] 2024-04-25T01:15:58.776 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQCsrilmYhNVKhAAsFW+T2gQAEsOl8EFhBicyg== 2024-04-25T01:15:58.815 INFO:teuthology.orchestra.run.smithi103.stdout:[mgr.y] 2024-04-25T01:15:58.815 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQCqrilm53U1OxAAV/IYzYKURT+G44pUIEkejg== 2024-04-25T01:15:58.815 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.a] 2024-04-25T01:15:58.815 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQCrrilmjb9qFBAAlnPjHsgFagy1je4+qyjyOQ== 2024-04-25T01:15:58.815 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.c] 2024-04-25T01:15:58.815 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQCrrilmmi9BIhAA2s5UuqgNa1mTnzELh6kg1Q== 2024-04-25T01:15:58.815 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.0] 2024-04-25T01:15:58.816 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQCvrilmONAAOhAAMfwg6+ChCofN/OS6Jp7DLg== 2024-04-25T01:15:58.816 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.1] 2024-04-25T01:15:58.816 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQCzrilmaKqpGhAAMFnbJ9tBiOgP7aUcxHxdAQ== 2024-04-25T01:15:58.816 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.2] 2024-04-25T01:15:58.816 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQC2rilm9bRPNBAAh9uf5qeKxKVGZjeURZN2ng== 2024-04-25T01:15:58.816 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.3] 2024-04-25T01:15:58.816 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQC6rilm335uAxAAzL5VZalis0ydEKCAc9p20w== 2024-04-25T01:15:58.816 INFO:teuthology.orchestra.run.smithi103.stdout:[mgr.x] 2024-04-25T01:15:58.816 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQCrrilm5IGgChAALBQG+sk4Pr+UuJWnHXh/ZA== 2024-04-25T01:15:58.816 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.b] 2024-04-25T01:15:58.816 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQCrrilmUZHfMBAAbh51emKl6HSLVHIaUOllXA== 2024-04-25T01:15:58.816 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.d] 2024-04-25T01:15:58.816 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQCsrilm/zZiARAAP8D+OSj0K9A3NGjnI+NOLA== 2024-04-25T01:15:58.816 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.4] 2024-04-25T01:15:58.816 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQDArilmkRNZDRAAPLVSDr1+ERL09nIchRCNVw== 2024-04-25T01:15:58.817 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.5] 2024-04-25T01:15:58.817 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQDDrilmM3k5GRAA6toW36myIctvYTYY5kzayA== 2024-04-25T01:15:58.817 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.6] 2024-04-25T01:15:58.817 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQDGrilmZJEXNRAARn43EOUZQzSdNDmifLDQpg== 2024-04-25T01:15:58.817 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.7] 2024-04-25T01:15:58.817 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQDKrilm60XzBRAAz37R48OIHSB4petorwdLDQ== 2024-04-25T01:15:58.817 INFO:teuthology.orchestra.run.smithi103.stdout:[client.0] 2024-04-25T01:15:58.817 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQCsrilmTsovDhAAVqcIHshdGrSHncSLyuueoQ== 2024-04-25T01:15:58.817 INFO:teuthology.orchestra.run.smithi103.stdout:[client.2] 2024-04-25T01:15:58.817 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQCsrilmLMVnFxAAQv8hYmJ1f8WsFc4+C6QFWg== 2024-04-25T01:15:58.817 INFO:teuthology.orchestra.run.smithi103.stdout:[client.1] 2024-04-25T01:15:58.817 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQCsrilmUIgRIRAAmTYrl9YlFG/l6hAHrMhYAQ== 2024-04-25T01:15:58.817 INFO:teuthology.orchestra.run.smithi103.stdout:[client.3] 2024-04-25T01:15:58.817 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQCsrilmYhNVKhAAsFW+T2gQAEsOl8EFhBicyg== 2024-04-25T01:15:58.819 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T01:15:58.828 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T01:15:58.994 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T01:15:59.002 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T01:15:59.135 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T01:15:59.142 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T01:15:59.276 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:15:59.281 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:15:59.414 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:15:59.422 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:15:59.554 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:15:59.560 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:15:59.703 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:15:59.710 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:15:59.840 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T01:15:59.845 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T01:15:59.979 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T01:15:59.984 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T01:16:00.116 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T01:16:00.121 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T01:16:00.249 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:16:00.255 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:16:00.373 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:16:00.380 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:16:00.514 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:16:00.519 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:16:00.645 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:16:00.649 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T01:16:00.778 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T01:16:00.783 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T01:16:00.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T01:16:00.926 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T01:16:01.060 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T01:16:01.086 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T01:16:01.223 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T01:16:01.230 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T01:16:01.366 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-25T01:16:01.366 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-25T01:16:01.453 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T01:16:01.603 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-25T01:16:01.672 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-25T01:16:01.761 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T01:16:01.908 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-25T01:16:01.985 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-25T01:16:02.068 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T01:16:02.216 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-25T01:16:02.304 DEBUG:teuthology.orchestra.run.smithi090:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-25T01:16:02.313 DEBUG:teuthology.orchestra.run.smithi103:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-25T01:16:02.367 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-25T01:16:02.368 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-25T01:16:02.368 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-25T01:16:02.375 INFO:tasks.ceph.mon.a:Started 2024-04-25T01:16:02.375 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-25T01:16:02.375 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-25T01:16:02.381 INFO:tasks.ceph.mon.b:Started 2024-04-25T01:16:02.381 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-25T01:16:02.382 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-25T01:16:02.387 INFO:tasks.ceph.mon.c:Started 2024-04-25T01:16:02.387 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-25T01:16:02.388 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-25T01:16:02.388 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-25T01:16:02.396 INFO:tasks.ceph.mgr.y:Started 2024-04-25T01:16:02.396 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-25T01:16:02.396 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-25T01:16:02.402 INFO:tasks.ceph.mgr.x:Started 2024-04-25T01:16:02.403 INFO:tasks.ceph:Setting crush tunables to default 2024-04-25T01:16:02.403 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd crush tunables default 2024-04-25T01:16:02.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.590+0000 7ff743800640 1 Processor -- start 2024-04-25T01:16:02.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.590+0000 7ff743800640 1 -- start start 2024-04-25T01:16:02.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.592+0000 7ff743800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff73c0722e0 0x7ff73c06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:02.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.592+0000 7ff743800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff73c0cf4d0 0x7ff73c1600c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:02.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.592+0000 7ff743800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff73c06e410 con 0x7ff73c15fc40 2024-04-25T01:16:02.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.592+0000 7ff743800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7ff73c160aa0 con 0x7ff73c0722e0 2024-04-25T01:16:02.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.592+0000 7ff743800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff73c06da40 con 0x7ff73c0cf4d0 2024-04-25T01:16:02.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.592+0000 7ff740e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff73c0722e0 0x7ff73c06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:02.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.592+0000 7ff740e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff73c0722e0 0x7ff73c06ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54006/0 (socket says 172.21.15.90:54006) 2024-04-25T01:16:02.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.592+0000 7ff740e00640 1 -- 172.21.15.90:0/1759497843 learned_addr learned my addr 172.21.15.90:0/1759497843 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:02.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.592+0000 7ff740e00640 1 -- 172.21.15.90:0/1759497843 >> v2:172.21.15.90:3300/0 conn(0x7ff73c0722e0 msgr2=0x7ff73c06ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-25T01:16:02.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.592+0000 7ff740e00640 1 -- 172.21.15.90:0/1759497843 >> v2:172.21.15.90:3300/0 conn(0x7ff73c0722e0 msgr2=0x7ff73c06ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T01:16:02.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.592+0000 7ff740e00640 1 --2- 172.21.15.90:0/1759497843 >> v2:172.21.15.90:3300/0 conn(0x7ff73c0722e0 0x7ff73c06ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T01:16:02.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.592+0000 7ff740e00640 1 --2- 172.21.15.90:0/1759497843 >> v2:172.21.15.90:3300/0 conn(0x7ff73c0722e0 0x7ff73c06ded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:16:02.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.593+0000 7ff741800640 1 --2- 172.21.15.90:0/1759497843 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff73c0cf4d0 0x7ff73c1600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:02.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.593+0000 7ff741800640 1 -- 172.21.15.90:0/1759497843 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff73c0cf4d0 msgr2=0x7ff73c1600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-25T01:16:02.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.593+0000 7ff741800640 1 -- 172.21.15.90:0/1759497843 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff73c0cf4d0 msgr2=0x7ff73c1600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T01:16:02.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.593+0000 7ff741800640 1 --2- 172.21.15.90:0/1759497843 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff73c0cf4d0 0x7ff73c1600c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T01:16:02.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.593+0000 7ff741800640 1 --2- 172.21.15.90:0/1759497843 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff73c0cf4d0 0x7ff73c1600c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:16:02.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.793+0000 7ff740e00640 1 --2- 172.21.15.90:0/1759497843 >> v2:172.21.15.90:3300/0 conn(0x7ff73c0722e0 0x7ff73c06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:02.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.793+0000 7ff740e00640 1 -- 172.21.15.90:0/1759497843 >> v2:172.21.15.90:3300/0 conn(0x7ff73c0722e0 msgr2=0x7ff73c06ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-25T01:16:02.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.793+0000 7ff740e00640 1 -- 172.21.15.90:0/1759497843 >> v2:172.21.15.90:3300/0 conn(0x7ff73c0722e0 msgr2=0x7ff73c06ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T01:16:02.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.793+0000 7ff740e00640 1 --2- 172.21.15.90:0/1759497843 >> v2:172.21.15.90:3300/0 conn(0x7ff73c0722e0 0x7ff73c06ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T01:16:02.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.793+0000 7ff740e00640 1 --2- 172.21.15.90:0/1759497843 >> v2:172.21.15.90:3300/0 conn(0x7ff73c0722e0 0x7ff73c06ded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:16:02.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.794+0000 7ff741800640 1 --2- 172.21.15.90:0/1759497843 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff73c0cf4d0 0x7ff73c1600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:02.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.795+0000 7ff741800640 1 -- 172.21.15.90:0/1759497843 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff73c0cf4d0 msgr2=0x7ff73c1600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-25T01:16:02.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.795+0000 7ff741800640 1 -- 172.21.15.90:0/1759497843 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff73c0cf4d0 msgr2=0x7ff73c1600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T01:16:02.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.795+0000 7ff741800640 1 --2- 172.21.15.90:0/1759497843 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff73c0cf4d0 0x7ff73c1600c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T01:16:02.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.795+0000 7ff741800640 1 --2- 172.21.15.90:0/1759497843 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff73c0cf4d0 0x7ff73c1600c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:16:02.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.830+0000 7ff73be00640 1 -- 172.21.15.90:0/1759497843 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2918072942 0 0) 0x7ff73c06e410 con 0x7ff73c15fc40 2024-04-25T01:16:02.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.830+0000 7ff73be00640 1 -- 172.21.15.90:0/1759497843 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff720003520 con 0x7ff73c15fc40 2024-04-25T01:16:02.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.830+0000 7ff73be00640 1 -- 172.21.15.90:0/1759497843 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 45338007 0 0) 0x7ff720003520 con 0x7ff73c15fc40 2024-04-25T01:16:02.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.830+0000 7ff73be00640 1 -- 172.21.15.90:0/1759497843 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff73c0cf4d0 msgr2=0x7ff73c1600c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:16:02.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.830+0000 7ff73be00640 1 --2- 172.21.15.90:0/1759497843 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff73c0cf4d0 0x7ff73c1600c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:02.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.830+0000 7ff73be00640 1 -- 172.21.15.90:0/1759497843 >> v2:172.21.15.90:3300/0 conn(0x7ff73c0722e0 msgr2=0x7ff73c06ded0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:16:02.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.830+0000 7ff73be00640 1 --2- 172.21.15.90:0/1759497843 >> v2:172.21.15.90:3300/0 conn(0x7ff73c0722e0 0x7ff73c06ded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:02.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.830+0000 7ff73be00640 1 -- 172.21.15.90:0/1759497843 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff73c06d110 con 0x7ff73c15fc40 2024-04-25T01:16:02.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.830+0000 7ff73be00640 1 -- 172.21.15.90:0/1759497843 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7ff724001670 con 0x7ff73c15fc40 2024-04-25T01:16:02.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.831+0000 7ff73be00640 1 -- 172.21.15.90:0/1759497843 >> v1:172.21.15.90:6789/0 conn(0x7ff73c15fc40 legacy=0x7ff73c0cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:02.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.831+0000 7ff73be00640 1 --2- 172.21.15.90:0/1759497843 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7200039b0 0x7ff720003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:02.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.831+0000 7ff73be00640 1 --2- 172.21.15.90:0/1759497843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff720004800 0x7ff720004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:02.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.831+0000 7ff73be00640 1 --2- 172.21.15.90:0/1759497843 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff73c0cf4d0 0x7ff7200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:02.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.831+0000 7ff73be00640 1 -- 172.21.15.90:0/1759497843 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff73c160aa0 con 0x7ff7200039b0 2024-04-25T01:16:02.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.831+0000 7ff73be00640 1 -- 172.21.15.90:0/1759497843 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff72000baa0 con 0x7ff720004800 2024-04-25T01:16:02.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.831+0000 7ff73be00640 1 -- 172.21.15.90:0/1759497843 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff7200100b0 con 0x7ff73c0cf4d0 2024-04-25T01:16:02.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.832+0000 7ff733e00640 1 --2- 172.21.15.90:0/1759497843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff720004800 0x7ff720004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:02.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.832+0000 7ff740e00640 1 --2- 172.21.15.90:0/1759497843 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7200039b0 0x7ff720003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:02.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.832+0000 7ff741800640 1 --2- 172.21.15.90:0/1759497843 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff73c0cf4d0 0x7ff7200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:02.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.833+0000 7ff740e00640 1 -- 172.21.15.90:0/1759497843 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff73c0cf4d0 msgr2=0x7ff7200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:02.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.833+0000 7ff740e00640 1 --2- 172.21.15.90:0/1759497843 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff73c0cf4d0 0x7ff7200051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:02.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.833+0000 7ff740e00640 1 -- 172.21.15.90:0/1759497843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff720004800 msgr2=0x7ff720004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:02.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.833+0000 7ff740e00640 1 --2- 172.21.15.90:0/1759497843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff720004800 0x7ff720004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:02.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.833+0000 7ff740e00640 1 -- 172.21.15.90:0/1759497843 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff73c06d110 con 0x7ff7200039b0 2024-04-25T01:16:02.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.834+0000 7ff740e00640 1 --2- 172.21.15.90:0/1759497843 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7200039b0 0x7ff720003dd0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff72c009100 tx=0x7ff720005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=769224070eafc0e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:02.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.835+0000 7ff73be00640 1 -- 172.21.15.90:0/1759497843 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72c038070 con 0x7ff7200039b0 2024-04-25T01:16:02.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.835+0000 7ff73be00640 1 -- 172.21.15.90:0/1759497843 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff72c033410 con 0x7ff7200039b0 2024-04-25T01:16:02.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.835+0000 7ff73be00640 1 -- 172.21.15.90:0/1759497843 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72c00f040 con 0x7ff7200039b0 2024-04-25T01:16:02.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.836+0000 7ff743800640 1 -- 172.21.15.90:0/1759497843 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7200039b0 msgr2=0x7ff720003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:02.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.836+0000 7ff743800640 1 --2- 172.21.15.90:0/1759497843 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7200039b0 0x7ff720003dd0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff72c009100 tx=0x7ff720005830 comp rx=0 tx=0).stop 2024-04-25T01:16:02.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.836+0000 7ff741800640 1 -- 172.21.15.90:0/1759497843 reap_dead start 2024-04-25T01:16:02.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.837+0000 7ff743800640 1 -- 172.21.15.90:0/1759497843 shutdown_connections 2024-04-25T01:16:02.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.837+0000 7ff743800640 1 -- 172.21.15.90:0/1759497843 >> 172.21.15.90:0/1759497843 conn(0x7ff73c06b720 msgr2=0x7ff73c0b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:02.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.837+0000 7ff743800640 1 -- 172.21.15.90:0/1759497843 shutdown_connections 2024-04-25T01:16:02.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.837+0000 7ff743800640 1 -- 172.21.15.90:0/1759497843 wait complete. 2024-04-25T01:16:02.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.842+0000 7ff743800640 1 Processor -- start 2024-04-25T01:16:02.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.842+0000 7ff743800640 1 -- start start 2024-04-25T01:16:02.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.844+0000 7ff743800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7200039b0 0x7ff73c13e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:02.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.844+0000 7ff743800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff73c13f720 0x7ff73c0b36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:02.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.844+0000 7ff743800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff73c13fb80 0x7ff73c15b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:02.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.844+0000 7ff743800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff73c06d6d0 con 0x7ff73c13f720 2024-04-25T01:16:02.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.844+0000 7ff743800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff73c06cec0 con 0x7ff7200039b0 2024-04-25T01:16:02.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.844+0000 7ff743800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff73c1600c0 con 0x7ff73c13fb80 2024-04-25T01:16:02.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.844+0000 7ff733e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff73c13f720 0x7ff73c0b36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:02.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.844+0000 7ff733e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff73c13f720 0x7ff73c0b36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54046/0 (socket says 172.21.15.90:54046) 2024-04-25T01:16:02.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.844+0000 7ff733e00640 1 -- 172.21.15.90:0/59805396 learned_addr learned my addr 172.21.15.90:0/59805396 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:02.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.844+0000 7ff741800640 1 --2- 172.21.15.90:0/59805396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff73c13fb80 0x7ff73c15b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:02.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.845+0000 7ff740e00640 1 --2- 172.21.15.90:0/59805396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7200039b0 0x7ff73c13e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:02.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.845+0000 7ff733e00640 1 -- 172.21.15.90:0/59805396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff73c13fb80 msgr2=0x7ff73c15b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:02.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.845+0000 7ff733e00640 1 --2- 172.21.15.90:0/59805396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff73c13fb80 0x7ff73c15b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:02.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.845+0000 7ff733e00640 1 -- 172.21.15.90:0/59805396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7200039b0 msgr2=0x7ff73c13e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:02.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.845+0000 7ff733e00640 1 --2- 172.21.15.90:0/59805396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7200039b0 0x7ff73c13e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:02.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.845+0000 7ff733e00640 1 -- 172.21.15.90:0/59805396 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff73c1400e0 con 0x7ff73c13f720 2024-04-25T01:16:02.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.846+0000 7ff733e00640 1 --2- 172.21.15.90:0/59805396 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff73c13f720 0x7ff73c0b36e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff72400f9e0 tx=0x7ff72400fea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:02.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.847+0000 7ff739600640 1 -- 172.21.15.90:0/59805396 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7240053a0 con 0x7ff73c13f720 2024-04-25T01:16:02.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.847+0000 7ff739600640 1 -- 172.21.15.90:0/59805396 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff724018070 con 0x7ff73c13f720 2024-04-25T01:16:02.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.847+0000 7ff743800640 1 -- 172.21.15.90:0/59805396 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff73c1403c0 con 0x7ff73c13f720 2024-04-25T01:16:02.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.847+0000 7ff739600640 1 -- 172.21.15.90:0/59805396 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72400da60 con 0x7ff73c13f720 2024-04-25T01:16:02.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.847+0000 7ff743800640 1 -- 172.21.15.90:0/59805396 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff73c13ed30 con 0x7ff73c13f720 2024-04-25T01:16:02.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.847+0000 7ff739600640 1 -- 172.21.15.90:0/59805396 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff72400dc00 con 0x7ff73c13f720 2024-04-25T01:16:02.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.847+0000 7ff739600640 1 -- 172.21.15.90:0/59805396 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff72401d570 con 0x7ff73c13f720 2024-04-25T01:16:02.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.848+0000 7ff743800640 1 -- 172.21.15.90:0/59805396 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff73c0679d0 con 0x7ff73c13f720 2024-04-25T01:16:02.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.856+0000 7ff739600640 1 -- 172.21.15.90:0/59805396 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff73c0679d0 con 0x7ff73c13f720 2024-04-25T01:16:02.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.930+0000 7ff743800640 1 -- 172.21.15.90:0/59805396 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7ff73c068fb0 con 0x7ff73c13f720 2024-04-25T01:16:02.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.933+0000 7ff739600640 1 -- 172.21.15.90:0/59805396 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7ff7240035f0 con 0x7ff73c13f720 2024-04-25T01:16:02.918 INFO:teuthology.orchestra.run.smithi090.stderr:adjusted tunables profile to default 2024-04-25T01:16:02.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.934+0000 7ff743800640 1 -- 172.21.15.90:0/59805396 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff73c13f720 msgr2=0x7ff73c0b36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:02.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.934+0000 7ff743800640 1 --2- 172.21.15.90:0/59805396 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff73c13f720 0x7ff73c0b36e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff72400f9e0 tx=0x7ff72400fea0 comp rx=0 tx=0).stop 2024-04-25T01:16:02.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.934+0000 7ff743800640 1 -- 172.21.15.90:0/59805396 shutdown_connections 2024-04-25T01:16:02.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.934+0000 7ff743800640 1 --2- 172.21.15.90:0/59805396 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff73c13fb80 0x7ff73c15b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:02.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.934+0000 7ff743800640 1 --2- 172.21.15.90:0/59805396 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff73c13f720 0x7ff73c0b36e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:02.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.934+0000 7ff743800640 1 --2- 172.21.15.90:0/59805396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7200039b0 0x7ff73c13e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:02.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.934+0000 7ff743800640 1 -- 172.21.15.90:0/59805396 >> 172.21.15.90:0/59805396 conn(0x7ff73c06b720 msgr2=0x7ff73c141be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:02.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.934+0000 7ff743800640 1 -- 172.21.15.90:0/59805396 shutdown_connections 2024-04-25T01:16:02.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:02.934+0000 7ff743800640 1 -- 172.21.15.90:0/59805396 wait complete. 2024-04-25T01:16:02.936 INFO:tasks.ceph:check_enable_crimson: False 2024-04-25T01:16:02.936 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-25T01:16:02.937 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:16:02.937 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-25T01:16:02.973 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-25T01:16:02.988+0000 7f8b2b323040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:16:02.977 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-04-25T01:16:02.992+0000 7fa6dff67040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:16:03.010 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:16:03.010 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-25T01:16:03.128 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:16:03.129 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-25T01:16:03.158 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-25T01:16:03.174+0000 7f8b2b323040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:16:03.169 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-04-25T01:16:03.184+0000 7fa6dff67040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:16:03.224 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-25T01:16:03.240+0000 7f8b2b323040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:16:03.236 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-04-25T01:16:03.252+0000 7fa6dff67040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:16:03.244 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:16:03.245 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-25T01:16:03.356 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:16:03.356 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-25T01:16:03.434 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:16:03.435 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-25T01:16:03.519 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-25T01:16:03.534+0000 7f8b2b323040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:16:03.550 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-04-25T01:16:03.565+0000 7fa6dff67040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:16:03.556 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:16:03.556 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-25T01:16:03.575 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-25T01:16:03.590+0000 7f8b2b323040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:16:03.602 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-04-25T01:16:03.618+0000 7fa6dff67040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:16:03.646 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:16:03.646 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-25T01:16:03.761 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 1ad7c9e8-f014-464d-9354-a71afeaf587c 0 2024-04-25T01:16:03.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.011+0000 7fc2f9e00640 1 Processor -- start 2024-04-25T01:16:03.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.011+0000 7fc2f9e00640 1 -- start start 2024-04-25T01:16:03.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.012+0000 7fc2f9e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc2f415f250 0x7fc2f40b1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:03.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.012+0000 7fc2f9e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2f406b830 0x7fc2f4157dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:03.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.012+0000 7fc2f9e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2f40ce430 con 0x7fc2f406d300 2024-04-25T01:16:03.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.012+0000 7fc2f9e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc2f40b27e0 con 0x7fc2f415f250 2024-04-25T01:16:03.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.012+0000 7fc2f9e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc2f40b2ac0 con 0x7fc2f406b830 2024-04-25T01:16:03.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.013+0000 7fc2f8a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc2f415f250 0x7fc2f40b1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:03.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.013+0000 7fc2f8a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc2f415f250 0x7fc2f40b1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49832/0 (socket says 172.21.15.103:49832) 2024-04-25T01:16:03.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.013+0000 7fc2f3e00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fc2f406d300 0x7fc2f406b120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55292/0 (socket says 172.21.15.103:55292) 2024-04-25T01:16:03.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.013+0000 7fc2f8a00640 1 -- 172.21.15.103:0/1337291394 learned_addr learned my addr 172.21.15.103:0/1337291394 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-25T01:16:03.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.013+0000 7fc2f9400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2f406b830 0x7fc2f4157dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:03.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.013+0000 7fc2f8a00640 1 -- 172.21.15.103:0/1337291394 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2f406b830 msgr2=0x7fc2f4157dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:03.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.013+0000 7fc2f8a00640 1 --2- 172.21.15.103:0/1337291394 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2f406b830 0x7fc2f4157dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:03.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.013+0000 7fc2f8a00640 1 -- 172.21.15.103:0/1337291394 >> v1:172.21.15.103:6789/0 conn(0x7fc2f406d300 legacy=0x7fc2f406b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:03.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.013+0000 7fc2f8a00640 1 -- 172.21.15.103:0/1337291394 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f40ce6d0 con 0x7fc2f415f250 2024-04-25T01:16:03.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.013+0000 7fc2f9400640 1 --2- 172.21.15.103:0/1337291394 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2f406b830 0x7fc2f4157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:16:03.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.013+0000 7fc2f3400640 1 -- 172.21.15.103:0/1337291394 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3408247417 0 0) 0x7fc2f40ce430 con 0x7fc2f406d300 2024-04-25T01:16:03.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.015+0000 7fc2f8a00640 1 --2- 172.21.15.103:0/1337291394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f415f250 0x7fc2f40b1d80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8009870 tx=0x7fc2e802a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47a9ff9e34f30dc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:03.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.015+0000 7fc2f3400640 1 -- 172.21.15.103:0/1337291394 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8037070 con 0x7fc2f415f250 2024-04-25T01:16:04.000 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.015+0000 7fc2f3400640 1 -- 172.21.15.103:0/1337291394 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2e8032440 con 0x7fc2f415f250 2024-04-25T01:16:04.000 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.015+0000 7fc2f3400640 1 -- 172.21.15.103:0/1337291394 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8032810 con 0x7fc2f415f250 2024-04-25T01:16:04.000 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.016+0000 7fc2f9e00640 1 -- 172.21.15.103:0/1337291394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f415f250 msgr2=0x7fc2f40b1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:04.000 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.016+0000 7fc2f9e00640 1 --2- 172.21.15.103:0/1337291394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f415f250 0x7fc2f40b1d80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8009870 tx=0x7fc2e802a250 comp rx=0 tx=0).stop 2024-04-25T01:16:04.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.016+0000 7fc2f9e00640 1 -- 172.21.15.103:0/1337291394 shutdown_connections 2024-04-25T01:16:04.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.016+0000 7fc2f9e00640 1 --2- 172.21.15.103:0/1337291394 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2f406b830 0x7fc2f4157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:04.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.016+0000 7fc2f9e00640 1 --2- 172.21.15.103:0/1337291394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f415f250 0x7fc2f40b1d80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:04.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.016+0000 7fc2f9e00640 1 -- 172.21.15.103:0/1337291394 >> 172.21.15.103:0/1337291394 conn(0x7fc2f40717a0 msgr2=0x7fc2f40b4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:04.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.016+0000 7fc2f9e00640 1 -- 172.21.15.103:0/1337291394 shutdown_connections 2024-04-25T01:16:04.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.016+0000 7fc2f9e00640 1 -- 172.21.15.103:0/1337291394 wait complete. 2024-04-25T01:16:04.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.018+0000 7fc2f9e00640 1 Processor -- start 2024-04-25T01:16:04.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.018+0000 7fc2f9e00640 1 -- start start 2024-04-25T01:16:04.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.019+0000 7fc2f9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2f406b830 0x7fc2f414d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:04.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.019+0000 7fc2f9e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f406d300 0x7fc2f40ca480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:04.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.019+0000 7fc2f9e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2f415f250 0x7fc2f40ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:04.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.019+0000 7fc2f9e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f415e950 con 0x7fc2f406b830 2024-04-25T01:16:04.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.019+0000 7fc2f9e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f415f630 con 0x7fc2f406d300 2024-04-25T01:16:04.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.019+0000 7fc2f9e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc2f40b22c0 con 0x7fc2f415f250 2024-04-25T01:16:04.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.019+0000 7fc2f3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f406d300 0x7fc2f40ca480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:04.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.019+0000 7fc2f9400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2f415f250 0x7fc2f40ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:04.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.019+0000 7fc2f9400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2f415f250 0x7fc2f40ccb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.103:46242/0 (socket says 172.21.15.103:46242) 2024-04-25T01:16:04.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.019+0000 7fc2f9400640 1 -- 172.21.15.103:0/2310610604 learned_addr learned my addr 172.21.15.103:0/2310610604 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-25T01:16:04.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.019+0000 7fc2f8a00640 1 --2- 172.21.15.103:0/2310610604 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2f406b830 0x7fc2f414d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:04.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.020+0000 7fc2f3e00640 1 -- 172.21.15.103:0/2310610604 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2f415f250 msgr2=0x7fc2f40ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:04.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.020+0000 7fc2f3e00640 1 --2- 172.21.15.103:0/2310610604 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2f415f250 0x7fc2f40ccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:04.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.020+0000 7fc2f3e00640 1 -- 172.21.15.103:0/2310610604 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2f406b830 msgr2=0x7fc2f414d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:04.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.020+0000 7fc2f3e00640 1 --2- 172.21.15.103:0/2310610604 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2f406b830 0x7fc2f414d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:04.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.020+0000 7fc2f3e00640 1 -- 172.21.15.103:0/2310610604 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f40c96c0 con 0x7fc2f406d300 2024-04-25T01:16:04.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.020+0000 7fc2f3e00640 1 --2- 172.21.15.103:0/2310610604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f406d300 0x7fc2f40ca480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc2e000c450 tx=0x7fc2e000c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:04.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.020+0000 7fc2f0c00640 1 -- 172.21.15.103:0/2310610604 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e0018070 con 0x7fc2f406d300 2024-04-25T01:16:04.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.020+0000 7fc2f0c00640 1 -- 172.21.15.103:0/2310610604 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2e00074a0 con 0x7fc2f406d300 2024-04-25T01:16:04.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.020+0000 7fc2f9e00640 1 -- 172.21.15.103:0/2310610604 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2f40c99a0 con 0x7fc2f406d300 2024-04-25T01:16:04.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.020+0000 7fc2f0c00640 1 -- 172.21.15.103:0/2310610604 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e0004a20 con 0x7fc2f406d300 2024-04-25T01:16:04.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.020+0000 7fc2f9e00640 1 -- 172.21.15.103:0/2310610604 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2f40c9f60 con 0x7fc2f406d300 2024-04-25T01:16:04.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.020+0000 7fc2f0c00640 1 -- 172.21.15.103:0/2310610604 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc2e001b070 con 0x7fc2f406d300 2024-04-25T01:16:04.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.021+0000 7fc2f0c00640 1 -- 172.21.15.103:0/2310610604 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7fc2e0012440 con 0x7fc2f406d300 2024-04-25T01:16:04.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.021+0000 7fc2f9e00640 1 -- 172.21.15.103:0/2310610604 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2f4068fb0 con 0x7fc2f406d300 2024-04-25T01:16:04.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.024+0000 7fc2f0c00640 1 -- 172.21.15.103:0/2310610604 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fc2f4068fb0 con 0x7fc2f406d300 2024-04-25T01:16:04.087 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.103+0000 7fc2f9e00640 1 -- 172.21.15.103:0/2310610604 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1ad7c9e8-f014-464d-9354-a71afeaf587c", "id": 0} v 0) -- 0x7fc2f4075350 con 0x7fc2f406d300 2024-04-25T01:16:04.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.109+0000 7fc2f0c00640 1 -- 172.21.15.103:0/2310610604 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1ad7c9e8-f014-464d-9354-a71afeaf587c", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7fc2e0004bc0 con 0x7fc2f406d300 2024-04-25T01:16:04.094 INFO:teuthology.orchestra.run.smithi103.stdout:0 2024-04-25T01:16:04.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.111+0000 7fc2f9e00640 1 -- 172.21.15.103:0/2310610604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f406d300 msgr2=0x7fc2f40ca480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:04.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.111+0000 7fc2f9e00640 1 --2- 172.21.15.103:0/2310610604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f406d300 0x7fc2f40ca480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc2e000c450 tx=0x7fc2e000c910 comp rx=0 tx=0).stop 2024-04-25T01:16:04.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.111+0000 7fc2f9e00640 1 -- 172.21.15.103:0/2310610604 shutdown_connections 2024-04-25T01:16:04.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.111+0000 7fc2f9e00640 1 --2- 172.21.15.103:0/2310610604 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc2f415f250 0x7fc2f40ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:04.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.111+0000 7fc2f9e00640 1 --2- 172.21.15.103:0/2310610604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f406d300 0x7fc2f40ca480 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:04.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.111+0000 7fc2f9e00640 1 --2- 172.21.15.103:0/2310610604 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2f406b830 0x7fc2f414d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:04.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.111+0000 7fc2f9e00640 1 -- 172.21.15.103:0/2310610604 >> 172.21.15.103:0/2310610604 conn(0x7fc2f40717a0 msgr2=0x7fc2f406b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:04.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.111+0000 7fc2f9e00640 1 -- 172.21.15.103:0/2310610604 shutdown_connections 2024-04-25T01:16:04.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.111+0000 7fc2f9e00640 1 -- 172.21.15.103:0/2310610604 wait complete. 2024-04-25T01:16:04.110 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new eae07c14-363b-4f7f-8682-d1bd39111e41 1 2024-04-25T01:16:04.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.293+0000 7fbc31e00640 1 Processor -- start 2024-04-25T01:16:04.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.293+0000 7fbc31e00640 1 -- start start 2024-04-25T01:16:04.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.295+0000 7fbc31e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbc2c161560 0x7fbc2c0d0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:04.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.295+0000 7fbc31e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbc2c0b4d30 0x7fbc2c0d0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:04.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.295+0000 7fbc31e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc2c0d4c50 con 0x7fbc2c161940 2024-04-25T01:16:04.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.295+0000 7fbc31e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fbc2c0d1430 con 0x7fbc2c161560 2024-04-25T01:16:04.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.295+0000 7fbc31e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fbc2c0d4a00 con 0x7fbc2c0b4d30 2024-04-25T01:16:04.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.295+0000 7fbc22000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbc2c0b4d30 0x7fbc2c0d0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:04.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.295+0000 7fbc22000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbc2c0b4d30 0x7fbc2c0d0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49850/0 (socket says 172.21.15.103:49850) 2024-04-25T01:16:04.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.295+0000 7fbc22000640 1 -- 172.21.15.103:0/1054848617 learned_addr learned my addr 172.21.15.103:0/1054848617 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-25T01:16:04.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.295+0000 7fbc2aa00640 1 --2- 172.21.15.103:0/1054848617 >> v2:172.21.15.90:3300/0 conn(0x7fbc2c161560 0x7fbc2c0d0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:04.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.296+0000 7fbc22000640 1 -- 172.21.15.103:0/1054848617 >> v2:172.21.15.90:3300/0 conn(0x7fbc2c161560 msgr2=0x7fbc2c0d0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:04.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.296+0000 7fbc22000640 1 --2- 172.21.15.103:0/1054848617 >> v2:172.21.15.90:3300/0 conn(0x7fbc2c161560 0x7fbc2c0d0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:04.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.296+0000 7fbc22000640 1 -- 172.21.15.103:0/1054848617 >> v1:172.21.15.90:6789/0 conn(0x7fbc2c161940 legacy=0x7fbc2c161e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:04.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.296+0000 7fbc22000640 1 -- 172.21.15.103:0/1054848617 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc2c0cf730 con 0x7fbc2c0b4d30 2024-04-25T01:16:04.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.297+0000 7fbc22000640 1 --2- 172.21.15.103:0/1054848617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc2c0b4d30 0x7fbc2c0d0c40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c0b9f60 tx=0x7fbc1c02a1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=517c118e8ada58df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:04.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.297+0000 7fbc2a000640 1 -- 172.21.15.103:0/1054848617 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c037070 con 0x7fbc2c0b4d30 2024-04-25T01:16:04.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.297+0000 7fbc2a000640 1 -- 172.21.15.103:0/1054848617 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc1c032440 con 0x7fbc2c0b4d30 2024-04-25T01:16:04.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.297+0000 7fbc2a000640 1 -- 172.21.15.103:0/1054848617 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c032810 con 0x7fbc2c0b4d30 2024-04-25T01:16:04.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.298+0000 7fbc31e00640 1 -- 172.21.15.103:0/1054848617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc2c0b4d30 msgr2=0x7fbc2c0d0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:04.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.298+0000 7fbc31e00640 1 --2- 172.21.15.103:0/1054848617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc2c0b4d30 0x7fbc2c0d0c40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c0b9f60 tx=0x7fbc1c02a1e0 comp rx=0 tx=0).stop 2024-04-25T01:16:04.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.298+0000 7fbc31e00640 1 -- 172.21.15.103:0/1054848617 shutdown_connections 2024-04-25T01:16:04.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.298+0000 7fbc31e00640 1 --2- 172.21.15.103:0/1054848617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc2c0b4d30 0x7fbc2c0d0c40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:04.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.298+0000 7fbc31e00640 1 --2- 172.21.15.103:0/1054848617 >> v2:172.21.15.90:3300/0 conn(0x7fbc2c161560 0x7fbc2c0d0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:04.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.298+0000 7fbc31e00640 1 -- 172.21.15.103:0/1054848617 >> 172.21.15.103:0/1054848617 conn(0x7fbc2c0d3ee0 msgr2=0x7fbc2c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:04.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.298+0000 7fbc31e00640 1 -- 172.21.15.103:0/1054848617 shutdown_connections 2024-04-25T01:16:04.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.298+0000 7fbc31e00640 1 -- 172.21.15.103:0/1054848617 wait complete. 2024-04-25T01:16:04.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.299+0000 7fbc31e00640 1 Processor -- start 2024-04-25T01:16:04.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.300+0000 7fbc31e00640 1 -- start start 2024-04-25T01:16:04.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.300+0000 7fbc31e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc2c0b4d30 0x7fbc2c152cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:04.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.301+0000 7fbc31e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc2c161560 0x7fbc2c1f8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:04.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.301+0000 7fbc31e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc2c161940 0x7fbc2c15fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:04.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.301+0000 7fbc31e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2c0b52f0 con 0x7fbc2c161560 2024-04-25T01:16:04.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.301+0000 7fbc31e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2c0cf4e0 con 0x7fbc2c161940 2024-04-25T01:16:04.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.301+0000 7fbc31e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbc2c0b5470 con 0x7fbc2c0b4d30 2024-04-25T01:16:04.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.301+0000 7fbc2aa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc2c0b4d30 0x7fbc2c152cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:04.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.301+0000 7fbc2b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc2c161940 0x7fbc2c15fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:04.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.301+0000 7fbc2aa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc2c0b4d30 0x7fbc2c152cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.103:46258/0 (socket says 172.21.15.103:46258) 2024-04-25T01:16:04.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.301+0000 7fbc2aa00640 1 -- 172.21.15.103:0/1677593553 learned_addr learned my addr 172.21.15.103:0/1677593553 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-25T01:16:04.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.301+0000 7fbc22000640 1 --2- 172.21.15.103:0/1677593553 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc2c161560 0x7fbc2c1f8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:04.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.301+0000 7fbc2aa00640 1 -- 172.21.15.103:0/1677593553 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc2c161940 msgr2=0x7fbc2c15fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:04.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.301+0000 7fbc2aa00640 1 --2- 172.21.15.103:0/1677593553 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc2c161940 0x7fbc2c15fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:04.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.301+0000 7fbc2aa00640 1 -- 172.21.15.103:0/1677593553 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc2c161560 msgr2=0x7fbc2c1f8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:04.287 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.301+0000 7fbc2aa00640 1 --2- 172.21.15.103:0/1677593553 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc2c161560 0x7fbc2c1f8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:04.287 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.301+0000 7fbc2aa00640 1 -- 172.21.15.103:0/1677593553 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc2c15ca30 con 0x7fbc2c0b4d30 2024-04-25T01:16:04.287 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.301+0000 7fbc2b400640 1 --2- 172.21.15.103:0/1677593553 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc2c161940 0x7fbc2c15fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:16:04.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.302+0000 7fbc2aa00640 1 --2- 172.21.15.103:0/1677593553 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc2c0b4d30 0x7fbc2c152cf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbc1400e870 tx=0x7fbc1400ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:04.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.302+0000 7fbc23e00640 1 -- 172.21.15.103:0/1677593553 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc14017070 con 0x7fbc2c0b4d30 2024-04-25T01:16:04.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.302+0000 7fbc23e00640 1 -- 172.21.15.103:0/1677593553 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc1400c940 con 0x7fbc2c0b4d30 2024-04-25T01:16:04.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.302+0000 7fbc31e00640 1 -- 172.21.15.103:0/1677593553 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc2c1f87a0 con 0x7fbc2c0b4d30 2024-04-25T01:16:04.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.302+0000 7fbc23e00640 1 -- 172.21.15.103:0/1677593553 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1400cd30 con 0x7fbc2c0b4d30 2024-04-25T01:16:04.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.302+0000 7fbc31e00640 1 -- 172.21.15.103:0/1677593553 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc2c15bc70 con 0x7fbc2c0b4d30 2024-04-25T01:16:04.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.302+0000 7fbc23e00640 1 -- 172.21.15.103:0/1677593553 <== mon.2 v2:172.21.15.103:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fbc1401a070 con 0x7fbc2c0b4d30 2024-04-25T01:16:04.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.302+0000 7fbc23e00640 1 -- 172.21.15.103:0/1677593553 <== mon.2 v2:172.21.15.103:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7fbc14013470 con 0x7fbc2c0b4d30 2024-04-25T01:16:04.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.303+0000 7fbc31e00640 1 -- 172.21.15.103:0/1677593553 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc2c068fb0 con 0x7fbc2c0b4d30 2024-04-25T01:16:04.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.306+0000 7fbc23e00640 1 -- 172.21.15.103:0/1677593553 <== mon.2 v2:172.21.15.103:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fbc14006020 con 0x7fbc2c0b4d30 2024-04-25T01:16:04.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.385+0000 7fbc31e00640 1 -- 172.21.15.103:0/1677593553 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "osd new", "uuid": "eae07c14-363b-4f7f-8682-d1bd39111e41", "id": 1} v 0) -- 0x7fbc2c06b2d0 con 0x7fbc2c0b4d30 2024-04-25T01:16:04.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.393+0000 7fbc23e00640 1 -- 172.21.15.103:0/1677593553 <== mon.2 v2:172.21.15.103:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "eae07c14-363b-4f7f-8682-d1bd39111e41", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7fbc14005990 con 0x7fbc2c0b4d30 2024-04-25T01:16:04.378 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-04-25T01:16:04.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.395+0000 7fbc31e00640 1 -- 172.21.15.103:0/1677593553 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc2c0b4d30 msgr2=0x7fbc2c152cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:04.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.395+0000 7fbc31e00640 1 --2- 172.21.15.103:0/1677593553 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc2c0b4d30 0x7fbc2c152cf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbc1400e870 tx=0x7fbc1400ed30 comp rx=0 tx=0).stop 2024-04-25T01:16:04.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.395+0000 7fbc31e00640 1 -- 172.21.15.103:0/1677593553 shutdown_connections 2024-04-25T01:16:04.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.395+0000 7fbc31e00640 1 --2- 172.21.15.103:0/1677593553 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc2c161940 0x7fbc2c15fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:04.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.395+0000 7fbc31e00640 1 --2- 172.21.15.103:0/1677593553 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc2c161560 0x7fbc2c1f8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:04.381 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.395+0000 7fbc31e00640 1 --2- 172.21.15.103:0/1677593553 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc2c0b4d30 0x7fbc2c152cf0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:04.381 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.395+0000 7fbc31e00640 1 -- 172.21.15.103:0/1677593553 >> 172.21.15.103:0/1677593553 conn(0x7fbc2c0d3ee0 msgr2=0x7fbc2c0b5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:04.381 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.395+0000 7fbc31e00640 1 -- 172.21.15.103:0/1677593553 shutdown_connections 2024-04-25T01:16:04.381 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.395+0000 7fbc31e00640 1 -- 172.21.15.103:0/1677593553 wait complete. 2024-04-25T01:16:04.396 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 84b4e1c1-f54d-4e3a-aa55-b92747e5af15 2 2024-04-25T01:16:04.548 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-04-25T01:16:04.563+0000 7fa6dff67040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:16:04.552 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-25T01:16:04.568+0000 7f8b2b323040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:16:04.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.600+0000 7f85f0a00640 1 Processor -- start 2024-04-25T01:16:04.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.600+0000 7f85f0a00640 1 -- start start 2024-04-25T01:16:04.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.601+0000 7f85f0a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85ec0d49b0 0x7f85ec0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:04.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.601+0000 7f85f0a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f85ec0cf980 0x7f85ec0d27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:04.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.601+0000 7f85f0a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85ec15e4c0 con 0x7f85ec15e0e0 2024-04-25T01:16:04.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.601+0000 7f85f0a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f85ec0cf4f0 con 0x7f85ec0cf980 2024-04-25T01:16:04.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.601+0000 7f85f0a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f85ec0d1420 con 0x7f85ec0d49b0 2024-04-25T01:16:04.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.602+0000 7f85eb400640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f85ec15e0e0 0x7f85ec0d0d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55298/0 (socket says 172.21.15.103:55298) 2024-04-25T01:16:04.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.602+0000 7f85eb400640 1 -- 172.21.15.103:0/2432482763 learned_addr learned my addr 172.21.15.103:0/2432482763 (peer_addr_for_me v1:172.21.15.103:0/0) 2024-04-25T01:16:04.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.602+0000 7f85e3e00640 1 --2- 172.21.15.103:0/2432482763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85ec0d49b0 0x7f85ec0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:04.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.602+0000 7f85ebe00640 1 --2- 172.21.15.103:0/2432482763 >> v2:172.21.15.103:3300/0 conn(0x7f85ec0cf980 0x7f85ec0d27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:04.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.602+0000 7f85eaa00640 1 -- 172.21.15.103:0/2432482763 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2062203291 0 0) 0x7f85ec15e4c0 con 0x7f85ec15e0e0 2024-04-25T01:16:04.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.602+0000 7f85eaa00640 1 -- 172.21.15.103:0/2432482763 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85d0003520 con 0x7f85ec15e0e0 2024-04-25T01:16:04.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.602+0000 7f85e3e00640 1 -- 172.21.15.103:0/2432482763 >> v2:172.21.15.103:3300/0 conn(0x7f85ec0cf980 msgr2=0x7f85ec0d27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:04.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.602+0000 7f85e3e00640 1 --2- 172.21.15.103:0/2432482763 >> v2:172.21.15.103:3300/0 conn(0x7f85ec0cf980 0x7f85ec0d27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:04.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.602+0000 7f85e3e00640 1 -- 172.21.15.103:0/2432482763 >> v1:172.21.15.103:6789/0 conn(0x7f85ec15e0e0 legacy=0x7f85ec0d0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:04.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.602+0000 7f85e3e00640 1 -- 172.21.15.103:0/2432482763 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85ec0cebc0 con 0x7f85ec0d49b0 2024-04-25T01:16:04.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.603+0000 7f85e3e00640 1 --2- 172.21.15.103:0/2432482763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85ec0d49b0 0x7f85ec0d22b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f85ec0cf670 tx=0x7f85d402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b1abab30343ee32 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:04.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.604+0000 7f85eaa00640 1 -- 172.21.15.103:0/2432482763 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d4037070 con 0x7f85ec0d49b0 2024-04-25T01:16:04.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.604+0000 7f85eaa00640 1 -- 172.21.15.103:0/2432482763 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85d4032440 con 0x7f85ec0d49b0 2024-04-25T01:16:04.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.604+0000 7f85eaa00640 1 -- 172.21.15.103:0/2432482763 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d40327f0 con 0x7f85ec0d49b0 2024-04-25T01:16:04.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.604+0000 7f85f0a00640 1 -- 172.21.15.103:0/2432482763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85ec0d49b0 msgr2=0x7f85ec0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:04.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.604+0000 7f85f0a00640 1 --2- 172.21.15.103:0/2432482763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85ec0d49b0 0x7f85ec0d22b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f85ec0cf670 tx=0x7f85d402a180 comp rx=0 tx=0).stop 2024-04-25T01:16:04.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.605+0000 7f85f0a00640 1 -- 172.21.15.103:0/2432482763 shutdown_connections 2024-04-25T01:16:04.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.605+0000 7f85f0a00640 1 --2- 172.21.15.103:0/2432482763 >> v2:172.21.15.103:3300/0 conn(0x7f85ec0cf980 0x7f85ec0d27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:04.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.605+0000 7f85f0a00640 1 --2- 172.21.15.103:0/2432482763 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85ec0d49b0 0x7f85ec0d22b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:04.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.605+0000 7f85f0a00640 1 -- 172.21.15.103:0/2432482763 >> 172.21.15.103:0/2432482763 conn(0x7f85ec0d3cd0 msgr2=0x7f85ec0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:04.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.605+0000 7f85f0a00640 1 -- 172.21.15.103:0/2432482763 shutdown_connections 2024-04-25T01:16:04.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.605+0000 7f85f0a00640 1 -- 172.21.15.103:0/2432482763 wait complete. 2024-04-25T01:16:04.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.606+0000 7f85f0a00640 1 Processor -- start 2024-04-25T01:16:04.599 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.606+0000 7f85f0a00640 1 -- start start 2024-04-25T01:16:04.599 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.607+0000 7f85f0a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85ec0cf980 0x7f85ec14db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:04.599 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.607+0000 7f85f0a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85ec0d49b0 0x7f85ec14ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:04.599 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.607+0000 7f85f0a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85ec15e0e0 0x7f85ec14c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:04.599 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.607+0000 7f85f0a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f85ec0ce7e0 con 0x7f85ec0d49b0 2024-04-25T01:16:04.599 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.607+0000 7f85f0a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f85ec0cf180 con 0x7f85ec0cf980 2024-04-25T01:16:04.599 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.607+0000 7f85f0a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f85ec0ce960 con 0x7f85ec15e0e0 2024-04-25T01:16:04.599 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.608+0000 7f85eb400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85ec0cf980 0x7f85ec14db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:04.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.608+0000 7f85eb400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85ec0cf980 0x7f85ec14db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49882/0 (socket says 172.21.15.103:49882) 2024-04-25T01:16:04.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.608+0000 7f85eb400640 1 -- 172.21.15.103:0/2822709270 learned_addr learned my addr 172.21.15.103:0/2822709270 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-25T01:16:04.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.608+0000 7f85ebe00640 1 --2- 172.21.15.103:0/2822709270 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85ec15e0e0 0x7f85ec14c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:04.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.608+0000 7f85e3e00640 1 --2- 172.21.15.103:0/2822709270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85ec0d49b0 0x7f85ec14ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:04.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.608+0000 7f85eb400640 1 -- 172.21.15.103:0/2822709270 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85ec15e0e0 msgr2=0x7f85ec14c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:04.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.608+0000 7f85eb400640 1 --2- 172.21.15.103:0/2822709270 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85ec15e0e0 0x7f85ec14c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:04.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.608+0000 7f85eb400640 1 -- 172.21.15.103:0/2822709270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85ec0d49b0 msgr2=0x7f85ec14ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:04.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.608+0000 7f85eb400640 1 --2- 172.21.15.103:0/2822709270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85ec0d49b0 0x7f85ec14ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:04.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.608+0000 7f85eb400640 1 -- 172.21.15.103:0/2822709270 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85ec14caf0 con 0x7f85ec0cf980 2024-04-25T01:16:04.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.608+0000 7f85e3e00640 1 --2- 172.21.15.103:0/2822709270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85ec0d49b0 0x7f85ec14ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:04.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.608+0000 7f85eb400640 1 --2- 172.21.15.103:0/2822709270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85ec0cf980 0x7f85ec14db90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f85dc00c390 tx=0x7f85dc00c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:04.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.609+0000 7f85e3400640 1 -- 172.21.15.103:0/2822709270 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85dc017070 con 0x7f85ec0cf980 2024-04-25T01:16:04.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.609+0000 7f85e3400640 1 -- 172.21.15.103:0/2822709270 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85dc005ce0 con 0x7f85ec0cf980 2024-04-25T01:16:04.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.609+0000 7f85e3400640 1 -- 172.21.15.103:0/2822709270 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85dc0072f0 con 0x7f85ec0cf980 2024-04-25T01:16:04.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.609+0000 7f85f0a00640 1 -- 172.21.15.103:0/2822709270 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85ec14cdd0 con 0x7f85ec0cf980 2024-04-25T01:16:04.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.609+0000 7f85f0a00640 1 -- 172.21.15.103:0/2822709270 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85ec14d390 con 0x7f85ec0cf980 2024-04-25T01:16:04.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.609+0000 7f85e3400640 1 -- 172.21.15.103:0/2822709270 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f85dc01a070 con 0x7f85ec0cf980 2024-04-25T01:16:04.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.609+0000 7f85e3400640 1 -- 172.21.15.103:0/2822709270 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f85dc012820 con 0x7f85ec0cf980 2024-04-25T01:16:04.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.610+0000 7f85f0a00640 1 -- 172.21.15.103:0/2822709270 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85ec075350 con 0x7f85ec0cf980 2024-04-25T01:16:04.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.613+0000 7f85e3400640 1 -- 172.21.15.103:0/2822709270 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f85dc021070 con 0x7f85ec0cf980 2024-04-25T01:16:04.650 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-04-25T01:16:04.666+0000 7fa6dff67040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:16:04.696 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-25T01:16:04.712+0000 7f8b2b323040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:16:04.700 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-04-25T01:16:04.715+0000 7fa6dff67040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:16:04.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.735+0000 7f85f0a00640 1 -- 172.21.15.103:0/2822709270 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd new", "uuid": "84b4e1c1-f54d-4e3a-aa55-b92747e5af15", "id": 2} v 0) -- 0x7f85ec15d800 con 0x7f85ec0cf980 2024-04-25T01:16:04.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.743+0000 7f85e3400640 1 -- 172.21.15.103:0/2822709270 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "84b4e1c1-f54d-4e3a-aa55-b92747e5af15", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f85dc012550 con 0x7f85ec0cf980 2024-04-25T01:16:04.728 INFO:teuthology.orchestra.run.smithi103.stdout:2 2024-04-25T01:16:04.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.745+0000 7f85f0a00640 1 -- 172.21.15.103:0/2822709270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85ec0cf980 msgr2=0x7f85ec14db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:04.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.745+0000 7f85f0a00640 1 --2- 172.21.15.103:0/2822709270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85ec0cf980 0x7f85ec14db90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f85dc00c390 tx=0x7f85dc00c850 comp rx=0 tx=0).stop 2024-04-25T01:16:04.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.745+0000 7f85f0a00640 1 -- 172.21.15.103:0/2822709270 shutdown_connections 2024-04-25T01:16:04.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.745+0000 7f85f0a00640 1 --2- 172.21.15.103:0/2822709270 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f85ec15e0e0 0x7f85ec14c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:04.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.745+0000 7f85f0a00640 1 --2- 172.21.15.103:0/2822709270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85ec0d49b0 0x7f85ec14ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:04.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.745+0000 7f85f0a00640 1 --2- 172.21.15.103:0/2822709270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85ec0cf980 0x7f85ec14db90 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:04.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.745+0000 7f85f0a00640 1 -- 172.21.15.103:0/2822709270 >> 172.21.15.103:0/2822709270 conn(0x7f85ec0d3cd0 msgr2=0x7f85ec1606c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:04.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.745+0000 7f85f0a00640 1 -- 172.21.15.103:0/2822709270 shutdown_connections 2024-04-25T01:16:04.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.745+0000 7f85f0a00640 1 -- 172.21.15.103:0/2822709270 wait complete. 2024-04-25T01:16:04.746 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new ed9a6b3e-a131-4d9b-bb1a-704a83612ec1 3 2024-04-25T01:16:04.750 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-25T01:16:04.765+0000 7f8b2b323040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:16:04.901 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-04-25T01:16:04.917+0000 7fa6dff67040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:16:04.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.935+0000 7f326a200640 1 Processor -- start 2024-04-25T01:16:04.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.936+0000 7f326a200640 1 -- start start 2024-04-25T01:16:04.922 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.937+0000 7f326a200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32640cdc50 0x7f32640ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:04.922 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.937+0000 7f326a200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f32640700c0 0x7f32640cebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:04.922 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.937+0000 7f326a200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32640d00d0 con 0x7f326406f050 2024-04-25T01:16:04.922 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.937+0000 7f326a200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f32640cd350 con 0x7f32640700c0 2024-04-25T01:16:04.922 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.937+0000 7f326a200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f32640d03b0 con 0x7f32640cdc50 2024-04-25T01:16:04.922 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.937+0000 7f3263400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32640cdc50 0x7f32640ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:04.922 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.937+0000 7f3263400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32640cdc50 0x7f32640ce6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.103:46278/0 (socket says 172.21.15.103:46278) 2024-04-25T01:16:04.923 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.937+0000 7f3263400640 1 -- 172.21.15.103:0/1071674933 learned_addr learned my addr 172.21.15.103:0/1071674933 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-25T01:16:04.923 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.938+0000 7f3262a00640 1 --2- 172.21.15.103:0/1071674933 >> v2:172.21.15.90:3300/0 conn(0x7f32640700c0 0x7f32640cebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:04.923 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.938+0000 7f3263400640 1 -- 172.21.15.103:0/1071674933 >> v2:172.21.15.90:3300/0 conn(0x7f32640700c0 msgr2=0x7f32640cebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:04.923 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.938+0000 7f3263400640 1 --2- 172.21.15.103:0/1071674933 >> v2:172.21.15.90:3300/0 conn(0x7f32640700c0 0x7f32640cebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:04.923 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.938+0000 7f3263400640 1 -- 172.21.15.103:0/1071674933 >> v1:172.21.15.103:6789/0 conn(0x7f326406f050 legacy=0x7f3264161eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:04.923 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.938+0000 7f3262000640 1 -- 172.21.15.103:0/1071674933 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 927684351 0 0) 0x7f32640d00d0 con 0x7f326406f050 2024-04-25T01:16:04.923 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.938+0000 7f3263400640 1 -- 172.21.15.103:0/1071674933 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32640d0690 con 0x7f32640cdc50 2024-04-25T01:16:04.924 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.939+0000 7f3263400640 1 --2- 172.21.15.103:0/1071674933 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32640cdc50 0x7f32640ce6a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3254009870 tx=0x7f325402a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4954a1c60140670d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:04.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.939+0000 7f3262000640 1 -- 172.21.15.103:0/1071674933 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3254037070 con 0x7f32640cdc50 2024-04-25T01:16:04.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.939+0000 7f3262000640 1 -- 172.21.15.103:0/1071674933 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3254032440 con 0x7f32640cdc50 2024-04-25T01:16:04.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.939+0000 7f3262000640 1 -- 172.21.15.103:0/1071674933 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3254032810 con 0x7f32640cdc50 2024-04-25T01:16:04.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.940+0000 7f326a200640 1 -- 172.21.15.103:0/1071674933 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32640cdc50 msgr2=0x7f32640ce6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:04.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.940+0000 7f326a200640 1 --2- 172.21.15.103:0/1071674933 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32640cdc50 0x7f32640ce6a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3254009870 tx=0x7f325402a1c0 comp rx=0 tx=0).stop 2024-04-25T01:16:04.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.940+0000 7f326a200640 1 -- 172.21.15.103:0/1071674933 shutdown_connections 2024-04-25T01:16:04.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.940+0000 7f326a200640 1 --2- 172.21.15.103:0/1071674933 >> v2:172.21.15.90:3300/0 conn(0x7f32640700c0 0x7f32640cebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:04.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.940+0000 7f326a200640 1 --2- 172.21.15.103:0/1071674933 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32640cdc50 0x7f32640ce6a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:04.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.940+0000 7f326a200640 1 -- 172.21.15.103:0/1071674933 >> 172.21.15.103:0/1071674933 conn(0x7f32640d1cf0 msgr2=0x7f32640d2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:04.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.940+0000 7f326a200640 1 -- 172.21.15.103:0/1071674933 shutdown_connections 2024-04-25T01:16:04.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.940+0000 7f326a200640 1 -- 172.21.15.103:0/1071674933 wait complete. 2024-04-25T01:16:04.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.942+0000 7f326a200640 1 Processor -- start 2024-04-25T01:16:04.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.942+0000 7f326a200640 1 -- start start 2024-04-25T01:16:04.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.943+0000 7f326a200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f326406f050 0x7f326415a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:04.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.943+0000 7f326a200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32640700c0 0x7f32641f4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:04.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.943+0000 7f326a200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32640cdc50 0x7f32641582a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:04.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.943+0000 7f326a200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f32640d1170 con 0x7f32640cdc50 2024-04-25T01:16:04.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.943+0000 7f326a200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f32640d0c50 con 0x7f326406f050 2024-04-25T01:16:04.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.943+0000 7f326a200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f32640c5750 con 0x7f32640700c0 2024-04-25T01:16:04.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.943+0000 7f3263400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f326406f050 0x7f326415a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:04.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.943+0000 7f3263400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f326406f050 0x7f326415a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49886/0 (socket says 172.21.15.103:49886) 2024-04-25T01:16:04.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.943+0000 7f3263400640 1 -- 172.21.15.103:0/557871241 learned_addr learned my addr 172.21.15.103:0/557871241 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-25T01:16:04.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.943+0000 7f3262a00640 1 --2- 172.21.15.103:0/557871241 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32640700c0 0x7f32641f4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:04.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.943+0000 7f3263e00640 1 --2- 172.21.15.103:0/557871241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32640cdc50 0x7f32641582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:04.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.943+0000 7f3263400640 1 -- 172.21.15.103:0/557871241 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32640700c0 msgr2=0x7f32641f4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:04.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.943+0000 7f3263400640 1 --2- 172.21.15.103:0/557871241 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32640700c0 0x7f32641f4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:04.930 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.943+0000 7f3263400640 1 -- 172.21.15.103:0/557871241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32640cdc50 msgr2=0x7f32641582a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:04.930 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.943+0000 7f3263400640 1 --2- 172.21.15.103:0/557871241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32640cdc50 0x7f32641582a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:04.930 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.943+0000 7f3263400640 1 -- 172.21.15.103:0/557871241 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32640c5a30 con 0x7f326406f050 2024-04-25T01:16:04.930 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.943+0000 7f3263e00640 1 --2- 172.21.15.103:0/557871241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32640cdc50 0x7f32641582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:04.930 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.944+0000 7f3263400640 1 --2- 172.21.15.103:0/557871241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f326406f050 0x7f326415a5b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f325402a200 tx=0x7f32540046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:04.930 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.944+0000 7f325b400640 1 -- 172.21.15.103:0/557871241 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3254041020 con 0x7f326406f050 2024-04-25T01:16:04.930 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.944+0000 7f325b400640 1 -- 172.21.15.103:0/557871241 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32540306e0 con 0x7f326406f050 2024-04-25T01:16:04.930 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.944+0000 7f325b400640 1 -- 172.21.15.103:0/557871241 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3254037040 con 0x7f326406f050 2024-04-25T01:16:04.930 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.944+0000 7f326a200640 1 -- 172.21.15.103:0/557871241 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32640c5d10 con 0x7f326406f050 2024-04-25T01:16:04.930 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.944+0000 7f326a200640 1 -- 172.21.15.103:0/557871241 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32640c62d0 con 0x7f326406f050 2024-04-25T01:16:04.930 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.944+0000 7f325b400640 1 -- 172.21.15.103:0/557871241 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3254044020 con 0x7f326406f050 2024-04-25T01:16:04.931 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.944+0000 7f325b400640 1 -- 172.21.15.103:0/557871241 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f325403dc40 con 0x7f326406f050 2024-04-25T01:16:04.931 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.945+0000 7f326a200640 1 -- 172.21.15.103:0/557871241 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3264068fb0 con 0x7f326406f050 2024-04-25T01:16:04.934 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:04.950+0000 7f325b400640 1 -- 172.21.15.103:0/557871241 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f325404b030 con 0x7f326406f050 2024-04-25T01:16:04.953 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-25T01:16:04.969+0000 7f8b2b323040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:16:04.964 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-04-25T01:16:04.980+0000 7fa6dff67040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:16:05.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.028+0000 7f326a200640 1 -- 172.21.15.103:0/557871241 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ed9a6b3e-a131-4d9b-bb1a-704a83612ec1", "id": 3} v 0) -- 0x7f32640b7c70 con 0x7f326406f050 2024-04-25T01:16:05.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.035+0000 7f325b400640 1 -- 172.21.15.103:0/557871241 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ed9a6b3e-a131-4d9b-bb1a-704a83612ec1", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f325403d420 con 0x7f326406f050 2024-04-25T01:16:05.019 INFO:teuthology.orchestra.run.smithi103.stdout:3 2024-04-25T01:16:05.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.036+0000 7f326a200640 1 -- 172.21.15.103:0/557871241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f326406f050 msgr2=0x7f326415a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:05.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.036+0000 7f326a200640 1 --2- 172.21.15.103:0/557871241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f326406f050 0x7f326415a5b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f325402a200 tx=0x7f32540046c0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.021 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-25T01:16:05.036+0000 7f8b2b323040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:16:05.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.036+0000 7f326a200640 1 -- 172.21.15.103:0/557871241 shutdown_connections 2024-04-25T01:16:05.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.036+0000 7f326a200640 1 --2- 172.21.15.103:0/557871241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32640cdc50 0x7f32641582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.036+0000 7f326a200640 1 --2- 172.21.15.103:0/557871241 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32640700c0 0x7f32641f4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.036+0000 7f326a200640 1 --2- 172.21.15.103:0/557871241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f326406f050 0x7f326415a5b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.036+0000 7f326a200640 1 -- 172.21.15.103:0/557871241 >> 172.21.15.103:0/557871241 conn(0x7f32640d1cf0 msgr2=0x7f32641611f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:05.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.036+0000 7f326a200640 1 -- 172.21.15.103:0/557871241 shutdown_connections 2024-04-25T01:16:05.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.036+0000 7f326a200640 1 -- 172.21.15.103:0/557871241 wait complete. 2024-04-25T01:16:05.026 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-04-25T01:16:05.041+0000 7fa6dff67040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:16:05.043 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 6e51d4c4-c19d-443f-b453-0aa2487b2b07 4 2024-04-25T01:16:05.091 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-25T01:16:05.107+0000 7f8b2b323040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:16:05.152 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-04-25T01:16:05.167+0000 7fa6dff67040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:16:05.222 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.237+0000 7f4caee00640 1 Processor -- start 2024-04-25T01:16:05.222 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.237+0000 7f4caee00640 1 -- start start 2024-04-25T01:16:05.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.239+0000 7f4caee00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4ca815e0e0 0x7f4ca80d0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:05.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.239+0000 7f4caee00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ca80d49b0 0x7f4ca80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:05.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.239+0000 7f4caee00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4ca80cf980 0x7f4ca80d27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:05.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.239+0000 7f4caee00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f4ca80cf4f0 con 0x7f4ca80cf980 2024-04-25T01:16:05.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.239+0000 7f4caee00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4ca815e4c0 con 0x7f4ca815e0e0 2024-04-25T01:16:05.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.239+0000 7f4caee00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4ca80cebc0 con 0x7f4ca80d49b0 2024-04-25T01:16:05.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.239+0000 7f4cada00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4ca815e0e0 0x7f4ca80d0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:05.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.239+0000 7f4cad000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ca80d49b0 0x7f4ca80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:05.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.239+0000 7f4cada00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4ca815e0e0 0x7f4ca80d0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49892/0 (socket says 172.21.15.103:49892) 2024-04-25T01:16:05.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.239+0000 7f4cada00640 1 -- 172.21.15.103:0/1906123291 learned_addr learned my addr 172.21.15.103:0/1906123291 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-25T01:16:05.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.239+0000 7f4cad000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ca80d49b0 0x7f4ca80d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.103:46306/0 (socket says 172.21.15.103:46306) 2024-04-25T01:16:05.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.240+0000 7f4cae400640 1 --2- 172.21.15.103:0/1906123291 >> v2:172.21.15.90:3300/0 conn(0x7f4ca80cf980 0x7f4ca80d27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:05.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.240+0000 7f4cad000640 1 -- 172.21.15.103:0/1906123291 >> v2:172.21.15.103:3300/0 conn(0x7f4ca815e0e0 msgr2=0x7f4ca80d0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:05.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.240+0000 7f4cad000640 1 --2- 172.21.15.103:0/1906123291 >> v2:172.21.15.103:3300/0 conn(0x7f4ca815e0e0 0x7f4ca80d0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.240+0000 7f4cad000640 1 -- 172.21.15.103:0/1906123291 >> v2:172.21.15.90:3300/0 conn(0x7f4ca80cf980 msgr2=0x7f4ca80d27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:05.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.240+0000 7f4cad000640 1 --2- 172.21.15.103:0/1906123291 >> v2:172.21.15.90:3300/0 conn(0x7f4ca80cf980 0x7f4ca80d27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.240+0000 7f4cad000640 1 -- 172.21.15.103:0/1906123291 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca80ceea0 con 0x7f4ca80d49b0 2024-04-25T01:16:05.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.240+0000 7f4cae400640 1 --2- 172.21.15.103:0/1906123291 >> v2:172.21.15.90:3300/0 conn(0x7f4ca80cf980 0x7f4ca80d27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:05.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.241+0000 7f4cad000640 1 --2- 172.21.15.103:0/1906123291 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ca80d49b0 0x7f4ca80d22b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4ca808d620 tx=0x7f4c9402a110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a5173aae9bda5bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:05.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.241+0000 7f4ca7e00640 1 -- 172.21.15.103:0/1906123291 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c94037070 con 0x7f4ca80d49b0 2024-04-25T01:16:05.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.241+0000 7f4ca7e00640 1 -- 172.21.15.103:0/1906123291 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c94032440 con 0x7f4ca80d49b0 2024-04-25T01:16:05.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.241+0000 7f4ca7e00640 1 -- 172.21.15.103:0/1906123291 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9402ae60 con 0x7f4ca80d49b0 2024-04-25T01:16:05.227 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-25T01:16:05.241+0000 7f8b2b323040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:16:05.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.242+0000 7f4caee00640 1 -- 172.21.15.103:0/1906123291 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ca80d49b0 msgr2=0x7f4ca80d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:05.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.242+0000 7f4caee00640 1 --2- 172.21.15.103:0/1906123291 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ca80d49b0 0x7f4ca80d22b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4ca808d620 tx=0x7f4c9402a110 comp rx=0 tx=0).stop 2024-04-25T01:16:05.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.242+0000 7f4caee00640 1 -- 172.21.15.103:0/1906123291 shutdown_connections 2024-04-25T01:16:05.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.242+0000 7f4caee00640 1 --2- 172.21.15.103:0/1906123291 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ca80d49b0 0x7f4ca80d22b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.242+0000 7f4caee00640 1 --2- 172.21.15.103:0/1906123291 >> v2:172.21.15.90:3300/0 conn(0x7f4ca80cf980 0x7f4ca80d27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.242+0000 7f4caee00640 1 --2- 172.21.15.103:0/1906123291 >> v2:172.21.15.103:3300/0 conn(0x7f4ca815e0e0 0x7f4ca80d0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.242+0000 7f4caee00640 1 -- 172.21.15.103:0/1906123291 >> 172.21.15.103:0/1906123291 conn(0x7f4ca80d3cd0 msgr2=0x7f4ca80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:05.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.242+0000 7f4caee00640 1 -- 172.21.15.103:0/1906123291 shutdown_connections 2024-04-25T01:16:05.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.242+0000 7f4caee00640 1 -- 172.21.15.103:0/1906123291 wait complete. 2024-04-25T01:16:05.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.243+0000 7f4caee00640 1 Processor -- start 2024-04-25T01:16:05.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.244+0000 7f4caee00640 1 -- start start 2024-04-25T01:16:05.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.244+0000 7f4caee00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ca80cf980 0x7f4ca80b79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:05.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.244+0000 7f4caee00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ca80d49b0 0x7f4ca815b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:05.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.245+0000 7f4caee00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ca815e0e0 0x7f4ca80cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:05.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.245+0000 7f4caee00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca80ce960 con 0x7f4ca80d49b0 2024-04-25T01:16:05.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.245+0000 7f4caee00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca80ce7e0 con 0x7f4ca80cf980 2024-04-25T01:16:05.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.245+0000 7f4caee00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4ca80d1250 con 0x7f4ca815e0e0 2024-04-25T01:16:05.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.245+0000 7f4cada00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ca80cf980 0x7f4ca80b79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:05.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.245+0000 7f4cada00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ca80cf980 0x7f4ca80b79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49908/0 (socket says 172.21.15.103:49908) 2024-04-25T01:16:05.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.245+0000 7f4cada00640 1 -- 172.21.15.103:0/2518345731 learned_addr learned my addr 172.21.15.103:0/2518345731 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-25T01:16:05.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.245+0000 7f4cae400640 1 --2- 172.21.15.103:0/2518345731 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ca815e0e0 0x7f4ca80cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:05.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.245+0000 7f4cad000640 1 --2- 172.21.15.103:0/2518345731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ca80d49b0 0x7f4ca815b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:05.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.245+0000 7f4cada00640 1 -- 172.21.15.103:0/2518345731 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ca815e0e0 msgr2=0x7f4ca80cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:05.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.245+0000 7f4cada00640 1 --2- 172.21.15.103:0/2518345731 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ca815e0e0 0x7f4ca80cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.245+0000 7f4cada00640 1 -- 172.21.15.103:0/2518345731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ca80d49b0 msgr2=0x7f4ca815b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:05.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.245+0000 7f4cada00640 1 --2- 172.21.15.103:0/2518345731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ca80d49b0 0x7f4ca815b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.245+0000 7f4cada00640 1 -- 172.21.15.103:0/2518345731 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca80cdb90 con 0x7f4ca80cf980 2024-04-25T01:16:05.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.246+0000 7f4cada00640 1 --2- 172.21.15.103:0/2518345731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ca80cf980 0x7f4ca80b79b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c00e870 tx=0x7f4c9c00ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:05.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.246+0000 7f4ca5600640 1 -- 172.21.15.103:0/2518345731 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c017070 con 0x7f4ca80cf980 2024-04-25T01:16:05.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.246+0000 7f4ca5600640 1 -- 172.21.15.103:0/2518345731 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c9c00c8e0 con 0x7f4ca80cf980 2024-04-25T01:16:05.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.246+0000 7f4ca5600640 1 -- 172.21.15.103:0/2518345731 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c00ccd0 con 0x7f4ca80cf980 2024-04-25T01:16:05.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.246+0000 7f4caee00640 1 -- 172.21.15.103:0/2518345731 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ca81f6f20 con 0x7f4ca80cf980 2024-04-25T01:16:05.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.246+0000 7f4caee00640 1 -- 172.21.15.103:0/2518345731 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ca81f74e0 con 0x7f4ca80cf980 2024-04-25T01:16:05.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.246+0000 7f4ca5600640 1 -- 172.21.15.103:0/2518345731 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4c9c01a070 con 0x7f4ca80cf980 2024-04-25T01:16:05.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.246+0000 7f4ca5600640 1 -- 172.21.15.103:0/2518345731 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f4c9c006570 con 0x7f4ca80cf980 2024-04-25T01:16:05.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.247+0000 7f4caee00640 1 -- 172.21.15.103:0/2518345731 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ca8075350 con 0x7f4ca80cf980 2024-04-25T01:16:05.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.251+0000 7f4ca5600640 1 -- 172.21.15.103:0/2518345731 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4c9c006020 con 0x7f4ca80cf980 2024-04-25T01:16:05.313 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.328+0000 7f4caee00640 1 -- 172.21.15.103:0/2518345731 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6e51d4c4-c19d-443f-b453-0aa2487b2b07", "id": 4} v 0) -- 0x7f4ca80749e0 con 0x7f4ca80cf980 2024-04-25T01:16:05.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.335+0000 7f4ca5600640 1 -- 172.21.15.103:0/2518345731 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6e51d4c4-c19d-443f-b453-0aa2487b2b07", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f4c9c005990 con 0x7f4ca80cf980 2024-04-25T01:16:05.320 INFO:teuthology.orchestra.run.smithi103.stdout:4 2024-04-25T01:16:05.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.337+0000 7f4caee00640 1 -- 172.21.15.103:0/2518345731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ca80cf980 msgr2=0x7f4ca80b79b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:05.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.337+0000 7f4caee00640 1 --2- 172.21.15.103:0/2518345731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ca80cf980 0x7f4ca80b79b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c00e870 tx=0x7f4c9c00ed30 comp rx=0 tx=0).stop 2024-04-25T01:16:05.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.337+0000 7f4caee00640 1 -- 172.21.15.103:0/2518345731 shutdown_connections 2024-04-25T01:16:05.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.337+0000 7f4caee00640 1 --2- 172.21.15.103:0/2518345731 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ca815e0e0 0x7f4ca80cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.337+0000 7f4caee00640 1 --2- 172.21.15.103:0/2518345731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ca80d49b0 0x7f4ca815b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.337+0000 7f4caee00640 1 --2- 172.21.15.103:0/2518345731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ca80cf980 0x7f4ca80b79b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.337+0000 7f4caee00640 1 -- 172.21.15.103:0/2518345731 >> 172.21.15.103:0/2518345731 conn(0x7f4ca80d3cd0 msgr2=0x7f4ca815ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:05.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.337+0000 7f4caee00640 1 -- 172.21.15.103:0/2518345731 shutdown_connections 2024-04-25T01:16:05.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.337+0000 7f4caee00640 1 -- 172.21.15.103:0/2518345731 wait complete. 2024-04-25T01:16:05.337 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new a948695d-848b-413e-91d3-3f4efa56e573 5 2024-04-25T01:16:05.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.533+0000 7f1fe9c00640 1 Processor -- start 2024-04-25T01:16:05.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.533+0000 7f1fe9c00640 1 -- start start 2024-04-25T01:16:05.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.534+0000 7f1fe9c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1fe4161560 0x7f1fe40d0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:05.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.534+0000 7f1fe9c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1fe40d2ac0 0x7f1fe40d0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:05.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.534+0000 7f1fe9c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fe40d4c50 con 0x7f1fe40b4d30 2024-04-25T01:16:05.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.534+0000 7f1fe9c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f1fe40d1430 con 0x7f1fe40d2ac0 2024-04-25T01:16:05.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.534+0000 7f1fe9c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1fe40d4a00 con 0x7f1fe4161560 2024-04-25T01:16:05.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.535+0000 7f1fe2a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1fe4161560 0x7f1fe40d0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:05.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.535+0000 7f1fe2a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1fe4161560 0x7f1fe40d0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49910/0 (socket says 172.21.15.103:49910) 2024-04-25T01:16:05.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.535+0000 7f1fe2a00640 1 -- 172.21.15.103:0/1249800324 learned_addr learned my addr 172.21.15.103:0/1249800324 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-25T01:16:05.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.535+0000 7f1fdbe00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f1fe40b4d30 0x7f1fe4161e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55314/0 (socket says 172.21.15.103:55314) 2024-04-25T01:16:05.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.535+0000 7f1fe3400640 1 --2- 172.21.15.103:0/1249800324 >> v2:172.21.15.90:3300/0 conn(0x7f1fe40d2ac0 0x7f1fe40d0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:05.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.535+0000 7f1fe2a00640 1 -- 172.21.15.103:0/1249800324 >> v2:172.21.15.90:3300/0 conn(0x7f1fe40d2ac0 msgr2=0x7f1fe40d0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:05.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.535+0000 7f1fe2a00640 1 --2- 172.21.15.103:0/1249800324 >> v2:172.21.15.90:3300/0 conn(0x7f1fe40d2ac0 0x7f1fe40d0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.535+0000 7f1fe2a00640 1 -- 172.21.15.103:0/1249800324 >> v1:172.21.15.103:6789/0 conn(0x7f1fe40b4d30 legacy=0x7f1fe4161e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:05.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.535+0000 7f1fe2000640 1 -- 172.21.15.103:0/1249800324 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 556517145 0 0) 0x7f1fe40d4c50 con 0x7f1fe40b4d30 2024-04-25T01:16:05.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.535+0000 7f1fe2a00640 1 -- 172.21.15.103:0/1249800324 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe40cf730 con 0x7f1fe4161560 2024-04-25T01:16:05.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.536+0000 7f1fe2a00640 1 --2- 172.21.15.103:0/1249800324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fe4161560 0x7f1fe40d0700 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1fe40b6a60 tx=0x7f1fd402a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a548fc74204b0c43 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:05.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.537+0000 7f1fe2000640 1 -- 172.21.15.103:0/1249800324 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd4037070 con 0x7f1fe4161560 2024-04-25T01:16:05.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.537+0000 7f1fe2000640 1 -- 172.21.15.103:0/1249800324 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fd4032440 con 0x7f1fe4161560 2024-04-25T01:16:05.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.537+0000 7f1fe2000640 1 -- 172.21.15.103:0/1249800324 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd4032810 con 0x7f1fe4161560 2024-04-25T01:16:05.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.537+0000 7f1fe9c00640 1 -- 172.21.15.103:0/1249800324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fe4161560 msgr2=0x7f1fe40d0700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:05.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.537+0000 7f1fe9c00640 1 --2- 172.21.15.103:0/1249800324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fe4161560 0x7f1fe40d0700 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1fe40b6a60 tx=0x7f1fd402a210 comp rx=0 tx=0).stop 2024-04-25T01:16:05.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.537+0000 7f1fe9c00640 1 -- 172.21.15.103:0/1249800324 shutdown_connections 2024-04-25T01:16:05.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.537+0000 7f1fe9c00640 1 --2- 172.21.15.103:0/1249800324 >> v2:172.21.15.90:3300/0 conn(0x7f1fe40d2ac0 0x7f1fe40d0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.530 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.537+0000 7f1fe9c00640 1 --2- 172.21.15.103:0/1249800324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fe4161560 0x7f1fe40d0700 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.530 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.537+0000 7f1fe9c00640 1 -- 172.21.15.103:0/1249800324 >> 172.21.15.103:0/1249800324 conn(0x7f1fe40d3ee0 msgr2=0x7f1fe40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:05.530 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.537+0000 7f1fe9c00640 1 -- 172.21.15.103:0/1249800324 shutdown_connections 2024-04-25T01:16:05.530 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.537+0000 7f1fe9c00640 1 -- 172.21.15.103:0/1249800324 wait complete. 2024-04-25T01:16:05.530 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.539+0000 7f1fe9c00640 1 Processor -- start 2024-04-25T01:16:05.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.539+0000 7f1fe9c00640 1 -- start start 2024-04-25T01:16:05.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.540+0000 7f1fe9c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fe40b4d30 0x7f1fe41577d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:05.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.540+0000 7f1fe9c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fe40d2ac0 0x7f1fe4158e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:05.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.540+0000 7f1fe9c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1fe4161560 0x7f1fe4155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:05.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.540+0000 7f1fe9c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe40b52f0 con 0x7f1fe4161560 2024-04-25T01:16:05.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.540+0000 7f1fe9c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe40cf4e0 con 0x7f1fe40b4d30 2024-04-25T01:16:05.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.540+0000 7f1fe9c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1fe40b5470 con 0x7f1fe40d2ac0 2024-04-25T01:16:05.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.540+0000 7f1fe2a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fe40b4d30 0x7f1fe41577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:05.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.540+0000 7f1fdbe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fe40d2ac0 0x7f1fe4158e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:05.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.540+0000 7f1fe2a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fe40b4d30 0x7f1fe41577d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49914/0 (socket says 172.21.15.103:49914) 2024-04-25T01:16:05.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.540+0000 7f1fe2a00640 1 -- 172.21.15.103:0/2638529316 learned_addr learned my addr 172.21.15.103:0/2638529316 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-25T01:16:05.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.541+0000 7f1fe3400640 1 --2- 172.21.15.103:0/2638529316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1fe4161560 0x7f1fe4155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:05.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.541+0000 7f1fe2a00640 1 -- 172.21.15.103:0/2638529316 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fe40d2ac0 msgr2=0x7f1fe4158e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:05.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.541+0000 7f1fe2a00640 1 --2- 172.21.15.103:0/2638529316 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fe40d2ac0 0x7f1fe4158e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.541+0000 7f1fe2a00640 1 -- 172.21.15.103:0/2638529316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1fe4161560 msgr2=0x7f1fe4155330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:05.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.541+0000 7f1fe2a00640 1 --2- 172.21.15.103:0/2638529316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1fe4161560 0x7f1fe4155330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.541+0000 7f1fe2a00640 1 -- 172.21.15.103:0/2638529316 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe4159390 con 0x7f1fe40b4d30 2024-04-25T01:16:05.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.541+0000 7f1fe3400640 1 --2- 172.21.15.103:0/2638529316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1fe4161560 0x7f1fe4155330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:05.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.541+0000 7f1fe2a00640 1 --2- 172.21.15.103:0/2638529316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fe40b4d30 0x7f1fe41577d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4000c00 tx=0x7f1fd40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:05.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.541+0000 7f1fdaa00640 1 -- 172.21.15.103:0/2638529316 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd4041020 con 0x7f1fe40b4d30 2024-04-25T01:16:05.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.541+0000 7f1fdaa00640 1 -- 172.21.15.103:0/2638529316 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fd40306c0 con 0x7f1fe40b4d30 2024-04-25T01:16:05.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.541+0000 7f1fdaa00640 1 -- 172.21.15.103:0/2638529316 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd4037040 con 0x7f1fe40b4d30 2024-04-25T01:16:05.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.541+0000 7f1fe9c00640 1 -- 172.21.15.103:0/2638529316 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fe4158090 con 0x7f1fe40b4d30 2024-04-25T01:16:05.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.541+0000 7f1fe9c00640 1 -- 172.21.15.103:0/2638529316 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fe4158650 con 0x7f1fe40b4d30 2024-04-25T01:16:05.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.542+0000 7f1fdaa00640 1 -- 172.21.15.103:0/2638529316 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1fd4044020 con 0x7f1fe40b4d30 2024-04-25T01:16:05.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.542+0000 7f1fdaa00640 1 -- 172.21.15.103:0/2638529316 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f1fd403dbf0 con 0x7f1fe40b4d30 2024-04-25T01:16:05.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.542+0000 7f1fe9c00640 1 -- 172.21.15.103:0/2638529316 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fe4157e10 con 0x7f1fe40b4d30 2024-04-25T01:16:05.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.546+0000 7f1fdaa00640 1 -- 172.21.15.103:0/2638529316 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f1fe4157e10 con 0x7f1fe40b4d30 2024-04-25T01:16:05.602 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-04-25T01:16:05.617+0000 7fa6dff67040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:16:05.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.624+0000 7f1fe9c00640 1 -- 172.21.15.103:0/2638529316 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a948695d-848b-413e-91d3-3f4efa56e573", "id": 5} v 0) -- 0x7f1fe406b2d0 con 0x7f1fe40b4d30 2024-04-25T01:16:05.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.632+0000 7f1fdaa00640 1 -- 172.21.15.103:0/2638529316 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a948695d-848b-413e-91d3-3f4efa56e573", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f1fd404b030 con 0x7f1fe40b4d30 2024-04-25T01:16:05.617 INFO:teuthology.orchestra.run.smithi103.stdout:5 2024-04-25T01:16:05.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.633+0000 7f1fe9c00640 1 -- 172.21.15.103:0/2638529316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fe40b4d30 msgr2=0x7f1fe41577d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:05.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.633+0000 7f1fe9c00640 1 --2- 172.21.15.103:0/2638529316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fe40b4d30 0x7f1fe41577d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4000c00 tx=0x7f1fd40046c0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.633+0000 7f1fe9c00640 1 -- 172.21.15.103:0/2638529316 shutdown_connections 2024-04-25T01:16:05.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.633+0000 7f1fe9c00640 1 --2- 172.21.15.103:0/2638529316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1fe4161560 0x7f1fe4155330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.633+0000 7f1fe9c00640 1 --2- 172.21.15.103:0/2638529316 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1fe40d2ac0 0x7f1fe4158e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.633+0000 7f1fe9c00640 1 --2- 172.21.15.103:0/2638529316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fe40b4d30 0x7f1fe41577d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.633+0000 7f1fe9c00640 1 -- 172.21.15.103:0/2638529316 >> 172.21.15.103:0/2638529316 conn(0x7f1fe40d3ee0 msgr2=0x7f1fe406d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:05.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.634+0000 7f1fe9c00640 1 -- 172.21.15.103:0/2638529316 shutdown_connections 2024-04-25T01:16:05.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.634+0000 7f1fe9c00640 1 -- 172.21.15.103:0/2638529316 wait complete. 2024-04-25T01:16:05.634 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 19f8c647-a668-42a2-ac95-c4020fd212a3 6 2024-04-25T01:16:05.705 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-25T01:16:05.721+0000 7f8b2b323040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:16:05.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.826+0000 7fdf9a000640 1 Processor -- start 2024-04-25T01:16:05.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.826+0000 7fdf9a000640 1 -- start start 2024-04-25T01:16:05.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.828+0000 7fdf9a000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdf940d48a0 0x7fdf940d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:05.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.828+0000 7fdf9a000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdf940b39b0 0x7fdf940b3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:05.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.828+0000 7fdf9a000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf940d16b0 con 0x7fdf94161710 2024-04-25T01:16:05.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.828+0000 7fdf9a000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fdf940cfd30 con 0x7fdf940b39b0 2024-04-25T01:16:05.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.828+0000 7fdf9a000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fdf940cf490 con 0x7fdf940d48a0 2024-04-25T01:16:05.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.828+0000 7fdf93e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdf940d48a0 0x7fdf940d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:05.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.828+0000 7fdf93e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdf940d48a0 0x7fdf940d0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49926/0 (socket says 172.21.15.103:49926) 2024-04-25T01:16:05.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.828+0000 7fdf93e00640 1 -- 172.21.15.103:0/229540606 learned_addr learned my addr 172.21.15.103:0/229540606 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-25T01:16:05.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.828+0000 7fdf99600640 1 --2- 172.21.15.103:0/229540606 >> v2:172.21.15.90:3300/0 conn(0x7fdf940b39b0 0x7fdf940b3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:05.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.828+0000 7fdf93e00640 1 -- 172.21.15.103:0/229540606 >> v2:172.21.15.90:3300/0 conn(0x7fdf940b39b0 msgr2=0x7fdf940b3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:05.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.828+0000 7fdf93e00640 1 --2- 172.21.15.103:0/229540606 >> v2:172.21.15.90:3300/0 conn(0x7fdf940b39b0 0x7fdf940b3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.828+0000 7fdf93e00640 1 -- 172.21.15.103:0/229540606 >> v1:172.21.15.90:6789/0 conn(0x7fdf94161710 legacy=0x7fdf940d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:05.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.828+0000 7fdf93e00640 1 -- 172.21.15.103:0/229540606 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf940cf770 con 0x7fdf940d48a0 2024-04-25T01:16:05.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.829+0000 7fdf93e00640 1 --2- 172.21.15.103:0/229540606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf940d48a0 0x7fdf940d0db0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdf9408c350 tx=0x7fdf8002a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3bc340e630a776bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:05.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.830+0000 7fdf93400640 1 -- 172.21.15.103:0/229540606 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf80037070 con 0x7fdf940d48a0 2024-04-25T01:16:05.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.830+0000 7fdf93400640 1 -- 172.21.15.103:0/229540606 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf8002ae20 con 0x7fdf940d48a0 2024-04-25T01:16:05.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.830+0000 7fdf93400640 1 -- 172.21.15.103:0/229540606 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf80032660 con 0x7fdf940d48a0 2024-04-25T01:16:05.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.831+0000 7fdf9a000640 1 -- 172.21.15.103:0/229540606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf940d48a0 msgr2=0x7fdf940d0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:05.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.831+0000 7fdf9a000640 1 --2- 172.21.15.103:0/229540606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf940d48a0 0x7fdf940d0db0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdf9408c350 tx=0x7fdf8002a110 comp rx=0 tx=0).stop 2024-04-25T01:16:05.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.831+0000 7fdf9a000640 1 -- 172.21.15.103:0/229540606 shutdown_connections 2024-04-25T01:16:05.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.831+0000 7fdf9a000640 1 --2- 172.21.15.103:0/229540606 >> v2:172.21.15.90:3300/0 conn(0x7fdf940b39b0 0x7fdf940b3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.831+0000 7fdf9a000640 1 --2- 172.21.15.103:0/229540606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf940d48a0 0x7fdf940d0db0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.831+0000 7fdf9a000640 1 -- 172.21.15.103:0/229540606 >> 172.21.15.103:0/229540606 conn(0x7fdf940d3d80 msgr2=0x7fdf940d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:05.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.831+0000 7fdf9a000640 1 -- 172.21.15.103:0/229540606 shutdown_connections 2024-04-25T01:16:05.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.831+0000 7fdf9a000640 1 -- 172.21.15.103:0/229540606 wait complete. 2024-04-25T01:16:05.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.833+0000 7fdf9a000640 1 Processor -- start 2024-04-25T01:16:05.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.833+0000 7fdf9a000640 1 -- start start 2024-04-25T01:16:05.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.834+0000 7fdf9a000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf940b39b0 0x7fdf9415e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:05.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.834+0000 7fdf9a000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf940d48a0 0x7fdf941f3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:05.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.835+0000 7fdf9a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf94161710 0x7fdf940cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:05.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.835+0000 7fdf9a000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdf940d12f0 con 0x7fdf94161710 2024-04-25T01:16:05.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.835+0000 7fdf9a000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdf940b3330 con 0x7fdf940b39b0 2024-04-25T01:16:05.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.835+0000 7fdf9a000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdf940d1470 con 0x7fdf940d48a0 2024-04-25T01:16:05.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.835+0000 7fdf93e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf940d48a0 0x7fdf941f3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:05.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.835+0000 7fdf98c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf940b39b0 0x7fdf9415e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:05.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.835+0000 7fdf93e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf940d48a0 0x7fdf941f3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.103:46324/0 (socket says 172.21.15.103:46324) 2024-04-25T01:16:05.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.835+0000 7fdf93e00640 1 -- 172.21.15.103:0/4289426694 learned_addr learned my addr 172.21.15.103:0/4289426694 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-25T01:16:05.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.835+0000 7fdf99600640 1 --2- 172.21.15.103:0/4289426694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf94161710 0x7fdf940cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:05.821 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.835+0000 7fdf98c00640 1 -- 172.21.15.103:0/4289426694 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf940d48a0 msgr2=0x7fdf941f3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:05.821 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.835+0000 7fdf98c00640 1 --2- 172.21.15.103:0/4289426694 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf940d48a0 0x7fdf941f3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.821 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.835+0000 7fdf98c00640 1 -- 172.21.15.103:0/4289426694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf94161710 msgr2=0x7fdf940cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:05.821 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.835+0000 7fdf98c00640 1 --2- 172.21.15.103:0/4289426694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf94161710 0x7fdf940cd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.821 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.835+0000 7fdf98c00640 1 -- 172.21.15.103:0/4289426694 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf940c75b0 con 0x7fdf940b39b0 2024-04-25T01:16:05.821 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.835+0000 7fdf93e00640 1 --2- 172.21.15.103:0/4289426694 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf940d48a0 0x7fdf941f3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:16:05.821 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.836+0000 7fdf98c00640 1 --2- 172.21.15.103:0/4289426694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf940b39b0 0x7fdf9415e570 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fdf8800c410 tx=0x7fdf8800c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:05.822 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.836+0000 7fdf90c00640 1 -- 172.21.15.103:0/4289426694 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf88017070 con 0x7fdf940b39b0 2024-04-25T01:16:05.822 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.836+0000 7fdf90c00640 1 -- 172.21.15.103:0/4289426694 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf88005230 con 0x7fdf940b39b0 2024-04-25T01:16:05.822 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.836+0000 7fdf90c00640 1 -- 172.21.15.103:0/4289426694 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf880045e0 con 0x7fdf940b39b0 2024-04-25T01:16:05.822 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.836+0000 7fdf9a000640 1 -- 172.21.15.103:0/4289426694 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf940c7890 con 0x7fdf940b39b0 2024-04-25T01:16:05.822 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.836+0000 7fdf9a000640 1 -- 172.21.15.103:0/4289426694 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf940c7e50 con 0x7fdf940b39b0 2024-04-25T01:16:05.822 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.836+0000 7fdf90c00640 1 -- 172.21.15.103:0/4289426694 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdf8801a070 con 0x7fdf940b39b0 2024-04-25T01:16:05.822 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.836+0000 7fdf90c00640 1 -- 172.21.15.103:0/4289426694 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7fdf88007610 con 0x7fdf940b39b0 2024-04-25T01:16:05.822 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.838+0000 7fdf9a000640 1 -- 172.21.15.103:0/4289426694 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf94067dd0 con 0x7fdf940b39b0 2024-04-25T01:16:05.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.840+0000 7fdf90c00640 1 -- 172.21.15.103:0/4289426694 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fdf880067d0 con 0x7fdf940b39b0 2024-04-25T01:16:05.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.918+0000 7fdf9a000640 1 -- 172.21.15.103:0/4289426694 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd new", "uuid": "19f8c647-a668-42a2-ac95-c4020fd212a3", "id": 6} v 0) -- 0x7fdf940b45f0 con 0x7fdf940b39b0 2024-04-25T01:16:05.906 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-04-25T01:16:05.921+0000 7fa6dff67040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:16:05.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.927+0000 7fdf90c00640 1 -- 172.21.15.103:0/4289426694 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "19f8c647-a668-42a2-ac95-c4020fd212a3", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7fdf88006e10 con 0x7fdf940b39b0 2024-04-25T01:16:05.912 INFO:teuthology.orchestra.run.smithi103.stdout:6 2024-04-25T01:16:05.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.928+0000 7fdf9a000640 1 -- 172.21.15.103:0/4289426694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf940b39b0 msgr2=0x7fdf9415e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:05.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.928+0000 7fdf9a000640 1 --2- 172.21.15.103:0/4289426694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf940b39b0 0x7fdf9415e570 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fdf8800c410 tx=0x7fdf8800c8d0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.929+0000 7fdf9a000640 1 -- 172.21.15.103:0/4289426694 shutdown_connections 2024-04-25T01:16:05.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.929+0000 7fdf9a000640 1 --2- 172.21.15.103:0/4289426694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf94161710 0x7fdf940cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.929+0000 7fdf9a000640 1 --2- 172.21.15.103:0/4289426694 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf940d48a0 0x7fdf941f3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.929+0000 7fdf9a000640 1 --2- 172.21.15.103:0/4289426694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf940b39b0 0x7fdf9415e570 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:05.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.929+0000 7fdf9a000640 1 -- 172.21.15.103:0/4289426694 >> 172.21.15.103:0/4289426694 conn(0x7fdf940d3d80 msgr2=0x7fdf9406b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:05.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.929+0000 7fdf9a000640 1 -- 172.21.15.103:0/4289426694 shutdown_connections 2024-04-25T01:16:05.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:05.929+0000 7fdf9a000640 1 -- 172.21.15.103:0/4289426694 wait complete. 2024-04-25T01:16:05.936 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 847a35bd-7317-46c1-a9df-2a0832bd1056 7 2024-04-25T01:16:05.967 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-04-25T01:16:05.983+0000 7fa6dff67040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:16:06.025 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-25T01:16:06.040+0000 7f8b2b323040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:16:06.087 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-25T01:16:06.103+0000 7f8b2b323040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:16:06.098 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-04-25T01:16:06.114+0000 7fa6dff67040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:16:06.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.133+0000 7f4deb200640 1 Processor -- start 2024-04-25T01:16:06.118 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.133+0000 7f4deb200640 1 -- start start 2024-04-25T01:16:06.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.135+0000 7f4deb200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4dec0d29c0 0x7f4dec168bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:06.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.135+0000 7f4deb200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4dec15e4c0 con 0x7f4dec0d49b0 2024-04-25T01:16:06.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.135+0000 7f4deb200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4dec0cebc0 con 0x7f4dec15e0e0 2024-04-25T01:16:06.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.135+0000 7f4deb200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f4dec0cf4f0 con 0x7f4dec0d29c0 2024-04-25T01:16:06.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.135+0000 7f4de9e00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f4dec15e0e0 0x7f4dec0d0d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:55318/0 (socket says 172.21.15.103:55318) 2024-04-25T01:16:06.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.135+0000 7f4de9e00640 1 -- 172.21.15.103:0/2626533975 learned_addr learned my addr 172.21.15.103:0/2626533975 (peer_addr_for_me v1:172.21.15.103:0/0) 2024-04-25T01:16:06.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.135+0000 7f4de8a00640 1 -- 172.21.15.103:0/2626533975 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 495048956 0 0) 0x7f4dec0cebc0 con 0x7f4dec15e0e0 2024-04-25T01:16:06.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.135+0000 7f4de8a00640 1 -- 172.21.15.103:0/2626533975 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4dd0003520 con 0x7f4dec15e0e0 2024-04-25T01:16:06.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.136+0000 7f4dea800640 1 --2- 172.21.15.103:0/2626533975 >> v2:172.21.15.90:3300/0 conn(0x7f4dec0d29c0 0x7f4dec168bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:06.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.136+0000 7f4de8a00640 1 -- 172.21.15.103:0/2626533975 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3652933542 0 0) 0x7f4dd0003520 con 0x7f4dec15e0e0 2024-04-25T01:16:06.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.136+0000 7f4de8a00640 1 -- 172.21.15.103:0/2626533975 >> v2:172.21.15.90:3300/0 conn(0x7f4dec0d29c0 msgr2=0x7f4dec168bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:06.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.136+0000 7f4de8a00640 1 --2- 172.21.15.103:0/2626533975 >> v2:172.21.15.90:3300/0 conn(0x7f4dec0d29c0 0x7f4dec168bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:06.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.136+0000 7f4de8a00640 1 -- 172.21.15.103:0/2626533975 >> v1:172.21.15.90:6789/0 conn(0x7f4dec0d49b0 legacy=0x7f4dec0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:06.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.136+0000 7f4de8a00640 1 -- 172.21.15.103:0/2626533975 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dec0ceea0 con 0x7f4dec15e0e0 2024-04-25T01:16:06.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.136+0000 7f4de8a00640 1 -- 172.21.15.103:0/2626533975 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f4ddc001670 con 0x7f4dec15e0e0 2024-04-25T01:16:06.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.136+0000 7f4dea800640 1 --2- 172.21.15.103:0/2626533975 >> v2:172.21.15.90:3300/0 conn(0x7f4dec0d29c0 0x7f4dec168bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:06.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.136+0000 7f4de8a00640 1 -- 172.21.15.103:0/2626533975 >> v1:172.21.15.103:6789/0 conn(0x7f4dec15e0e0 legacy=0x7f4dec0d0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:06.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.136+0000 7f4de8a00640 1 --2- 172.21.15.103:0/2626533975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4dd00039b0 0x7f4dd0003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:06.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.136+0000 7f4de8a00640 1 --2- 172.21.15.103:0/2626533975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4dd0004800 0x7f4dd0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:06.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.136+0000 7f4de8a00640 1 --2- 172.21.15.103:0/2626533975 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dd00052e0 0x7f4dd000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:06.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.136+0000 7f4de8a00640 1 -- 172.21.15.103:0/2626533975 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4dec0cf4f0 con 0x7f4dd00052e0 2024-04-25T01:16:06.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.136+0000 7f4de8a00640 1 -- 172.21.15.103:0/2626533975 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd000c1e0 con 0x7f4dd00039b0 2024-04-25T01:16:06.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.136+0000 7f4de8a00640 1 -- 172.21.15.103:0/2626533975 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4dd000c3e0 con 0x7f4dd0004800 2024-04-25T01:16:06.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.136+0000 7f4de9e00640 1 --2- 172.21.15.103:0/2626533975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4dd00039b0 0x7f4dd0003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:06.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.136+0000 7f4de9400640 1 --2- 172.21.15.103:0/2626533975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4dd0004800 0x7f4dd0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:06.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.136+0000 7f4dea800640 1 --2- 172.21.15.103:0/2626533975 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dd00052e0 0x7f4dd000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:06.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.137+0000 7f4de9e00640 1 -- 172.21.15.103:0/2626533975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4dd0004800 msgr2=0x7f4dd0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:06.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.137+0000 7f4de9e00640 1 --2- 172.21.15.103:0/2626533975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4dd0004800 0x7f4dd0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:06.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.137+0000 7f4de9e00640 1 -- 172.21.15.103:0/2626533975 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dd00052e0 msgr2=0x7f4dd000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:06.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.137+0000 7f4de9e00640 1 --2- 172.21.15.103:0/2626533975 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dd00052e0 0x7f4dd000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:06.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.137+0000 7f4de9e00640 1 -- 172.21.15.103:0/2626533975 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dd0003520 con 0x7f4dd00039b0 2024-04-25T01:16:06.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.137+0000 7f4dea800640 1 --2- 172.21.15.103:0/2626533975 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dd00052e0 0x7f4dd000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:06.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.137+0000 7f4dea800640 1 -- 172.21.15.103:0/2626533975 reap_dead start 2024-04-25T01:16:06.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.138+0000 7f4de9e00640 1 --2- 172.21.15.103:0/2626533975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4dd00039b0 0x7f4dd0003db0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc00c210 tx=0x7f4dd0005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51eba42fc3236081 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:06.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.138+0000 7f4de8a00640 1 -- 172.21.15.103:0/2626533975 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc007d10 con 0x7f4dd00039b0 2024-04-25T01:16:06.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.138+0000 7f4de8a00640 1 -- 172.21.15.103:0/2626533975 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ddc0058d0 con 0x7f4dd00039b0 2024-04-25T01:16:06.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.138+0000 7f4de8a00640 1 -- 172.21.15.103:0/2626533975 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc005d20 con 0x7f4dd00039b0 2024-04-25T01:16:06.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.138+0000 7f4deb200640 1 -- 172.21.15.103:0/2626533975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4dd00039b0 msgr2=0x7f4dd0003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:06.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.138+0000 7f4deb200640 1 --2- 172.21.15.103:0/2626533975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4dd00039b0 0x7f4dd0003db0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc00c210 tx=0x7f4dd0005830 comp rx=0 tx=0).stop 2024-04-25T01:16:06.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.139+0000 7f4deb200640 1 -- 172.21.15.103:0/2626533975 shutdown_connections 2024-04-25T01:16:06.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.139+0000 7f4deb200640 1 --2- 172.21.15.103:0/2626533975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4dd00039b0 0x7f4dd0003db0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:06.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.139+0000 7f4deb200640 1 -- 172.21.15.103:0/2626533975 >> 172.21.15.103:0/2626533975 conn(0x7f4dec0d3cd0 msgr2=0x7f4dec0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:06.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.139+0000 7f4deb200640 1 -- 172.21.15.103:0/2626533975 shutdown_connections 2024-04-25T01:16:06.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.139+0000 7f4deb200640 1 -- 172.21.15.103:0/2626533975 wait complete. 2024-04-25T01:16:06.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.141+0000 7f4deb200640 1 Processor -- start 2024-04-25T01:16:06.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.142+0000 7f4deb200640 1 -- start start 2024-04-25T01:16:06.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.143+0000 7f4deb200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4dd00039b0 0x7f4dec15c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:06.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.143+0000 7f4deb200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4dec0cd430 0x7f4dec0cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:06.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.143+0000 7f4deb200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dec0b7400 0x7f4dec0b7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:06.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.143+0000 7f4deb200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4dec0ce960 con 0x7f4dec0b7400 2024-04-25T01:16:06.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.143+0000 7f4deb200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4dec0ce7e0 con 0x7f4dd00039b0 2024-04-25T01:16:06.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.143+0000 7f4deb200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4dec0cf980 con 0x7f4dec0cd430 2024-04-25T01:16:06.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.143+0000 7f4de9e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4dd00039b0 0x7f4dec15c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:06.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.143+0000 7f4de9400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4dec0cd430 0x7f4dec0cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:06.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.143+0000 7f4de9400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4dec0cd430 0x7f4dec0cd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.103:46338/0 (socket says 172.21.15.103:46338) 2024-04-25T01:16:06.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.143+0000 7f4de9e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4dd00039b0 0x7f4dec15c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49954/0 (socket says 172.21.15.103:49954) 2024-04-25T01:16:06.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.143+0000 7f4de9400640 1 -- 172.21.15.103:0/2462818207 learned_addr learned my addr 172.21.15.103:0/2462818207 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-25T01:16:06.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.144+0000 7f4dea800640 1 --2- 172.21.15.103:0/2462818207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dec0b7400 0x7f4dec0b7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:06.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.144+0000 7f4de9e00640 1 -- 172.21.15.103:0/2462818207 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4dec0cd430 msgr2=0x7f4dec0cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:06.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.144+0000 7f4de9e00640 1 --2- 172.21.15.103:0/2462818207 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4dec0cd430 0x7f4dec0cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:06.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.144+0000 7f4de9e00640 1 -- 172.21.15.103:0/2462818207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dec0b7400 msgr2=0x7f4dec0b7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:06.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.144+0000 7f4de9e00640 1 --2- 172.21.15.103:0/2462818207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dec0b7400 0x7f4dec0b7860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:06.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.144+0000 7f4de9e00640 1 -- 172.21.15.103:0/2462818207 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dec0caf40 con 0x7f4dd00039b0 2024-04-25T01:16:06.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.144+0000 7f4de9e00640 1 --2- 172.21.15.103:0/2462818207 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4dd00039b0 0x7f4dec15c310 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc030000 tx=0x7f4dec0ced90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:06.131 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.144+0000 7f4de2000640 1 -- 172.21.15.103:0/2462818207 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc001c50 con 0x7f4dd00039b0 2024-04-25T01:16:06.131 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.144+0000 7f4de2000640 1 -- 172.21.15.103:0/2462818207 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ddc0049f0 con 0x7f4dd00039b0 2024-04-25T01:16:06.131 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.144+0000 7f4de2000640 1 -- 172.21.15.103:0/2462818207 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc004ba0 con 0x7f4dd00039b0 2024-04-25T01:16:06.131 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.144+0000 7f4deb200640 1 -- 172.21.15.103:0/2462818207 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dec15c850 con 0x7f4dd00039b0 2024-04-25T01:16:06.131 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.145+0000 7f4deb200640 1 -- 172.21.15.103:0/2462818207 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dec0cb4c0 con 0x7f4dd00039b0 2024-04-25T01:16:06.131 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.145+0000 7f4de2000640 1 -- 172.21.15.103:0/2462818207 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4ddc03b070 con 0x7f4dd00039b0 2024-04-25T01:16:06.131 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.145+0000 7f4de2000640 1 -- 172.21.15.103:0/2462818207 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f4ddc0056a0 con 0x7f4dd00039b0 2024-04-25T01:16:06.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.146+0000 7f4deb200640 1 -- 172.21.15.103:0/2462818207 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4dec075350 con 0x7f4dd00039b0 2024-04-25T01:16:06.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.148+0000 7f4de2000640 1 -- 172.21.15.103:0/2462818207 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4ddc007d50 con 0x7f4dd00039b0 2024-04-25T01:16:06.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.248+0000 7f4deb200640 1 -- 172.21.15.103:0/2462818207 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd new", "uuid": "847a35bd-7317-46c1-a9df-2a0832bd1056", "id": 7} v 0) -- 0x7f4dec15d7a0 con 0x7f4dd00039b0 2024-04-25T01:16:06.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.258+0000 7f4de2000640 1 -- 172.21.15.103:0/2462818207 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "847a35bd-7317-46c1-a9df-2a0832bd1056", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f4ddc0058a0 con 0x7f4dd00039b0 2024-04-25T01:16:06.243 INFO:teuthology.orchestra.run.smithi103.stdout:7 2024-04-25T01:16:06.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.260+0000 7f4deb200640 1 -- 172.21.15.103:0/2462818207 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4dd00039b0 msgr2=0x7f4dec15c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:06.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.260+0000 7f4deb200640 1 --2- 172.21.15.103:0/2462818207 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4dd00039b0 0x7f4dec15c310 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc030000 tx=0x7f4dec0ced90 comp rx=0 tx=0).stop 2024-04-25T01:16:06.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.260+0000 7f4deb200640 1 -- 172.21.15.103:0/2462818207 shutdown_connections 2024-04-25T01:16:06.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.260+0000 7f4deb200640 1 --2- 172.21.15.103:0/2462818207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dec0b7400 0x7f4dec0b7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:06.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.260+0000 7f4deb200640 1 --2- 172.21.15.103:0/2462818207 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4dec0cd430 0x7f4dec0cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:06.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.260+0000 7f4deb200640 1 --2- 172.21.15.103:0/2462818207 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4dd00039b0 0x7f4dec15c310 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:06.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.260+0000 7f4deb200640 1 -- 172.21.15.103:0/2462818207 >> 172.21.15.103:0/2462818207 conn(0x7f4dec0d3cd0 msgr2=0x7f4dec06cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:06.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.261+0000 7f4deb200640 1 -- 172.21.15.103:0/2462818207 shutdown_connections 2024-04-25T01:16:06.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:16:06.261+0000 7f4deb200640 1 -- 172.21.15.103:0/2462818207 wait complete. 2024-04-25T01:16:06.246 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-25T01:16:06.262+0000 7f8b2b323040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:16:06.262 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-25T01:16:06.262 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-25T01:16:06.270 INFO:tasks.ceph.osd.0:Started 2024-04-25T01:16:06.270 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-25T01:16:06.270 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-25T01:16:06.278 INFO:tasks.ceph.osd.1:Started 2024-04-25T01:16:06.278 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-25T01:16:06.278 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-25T01:16:06.282 INFO:tasks.ceph.osd.2:Started 2024-04-25T01:16:06.283 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-25T01:16:06.283 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-25T01:16:06.288 INFO:tasks.ceph.osd.3:Started 2024-04-25T01:16:06.288 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-25T01:16:06.288 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-25T01:16:06.294 INFO:tasks.ceph.osd.4:Started 2024-04-25T01:16:06.294 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-25T01:16:06.294 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-25T01:16:06.300 INFO:tasks.ceph.osd.5:Started 2024-04-25T01:16:06.300 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-25T01:16:06.300 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-25T01:16:06.305 INFO:tasks.ceph.osd.6:Started 2024-04-25T01:16:06.305 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-25T01:16:06.306 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-25T01:16:06.311 INFO:tasks.ceph.osd.7:Started 2024-04-25T01:16:06.312 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:16:06.490 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-04-25T01:16:06.505+0000 7fa6dff67040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:16:06.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.536+0000 7f5713200640 1 Processor -- start 2024-04-25T01:16:06.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.536+0000 7f5713200640 1 -- start start 2024-04-25T01:16:06.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.537+0000 7f5713200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f571406b830 0x7f571415e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:06.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.537+0000 7f5713200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57140ae9d0 0x7f571415ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:06.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.537+0000 7f5713200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57140adc10 con 0x7f57140b41f0 2024-04-25T01:16:06.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.537+0000 7f5713200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f57140adef0 con 0x7f571406b830 2024-04-25T01:16:06.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.537+0000 7f5713200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f57140ae1d0 con 0x7f57140ae9d0 2024-04-25T01:16:06.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.538+0000 7f5711400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f571406b830 0x7f571415e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:06.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.538+0000 7f5711400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f571406b830 0x7f571415e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33644/0 (socket says 172.21.15.90:33644) 2024-04-25T01:16:06.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.538+0000 7f5711400640 1 -- 172.21.15.90:0/4292321316 learned_addr learned my addr 172.21.15.90:0/4292321316 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:06.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.538+0000 7f5711e00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f57140b41f0 0x7f571406b120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.90:34114/0 (socket says 172.21.15.90:34114) 2024-04-25T01:16:06.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.538+0000 7f5712800640 1 --2- 172.21.15.90:0/4292321316 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57140ae9d0 0x7f571415ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:06.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.538+0000 7f5711400640 1 -- 172.21.15.90:0/4292321316 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57140ae9d0 msgr2=0x7f571415ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:06.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.538+0000 7f5711400640 1 --2- 172.21.15.90:0/4292321316 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57140ae9d0 0x7f571415ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:06.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.538+0000 7f5711400640 1 -- 172.21.15.90:0/4292321316 >> v1:172.21.15.103:6789/0 conn(0x7f57140b41f0 legacy=0x7f571406b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:06.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.538+0000 7f5711400640 1 -- 172.21.15.90:0/4292321316 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57140ae4b0 con 0x7f571406b830 2024-04-25T01:16:06.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.539+0000 7f5711400640 1 --2- 172.21.15.90:0/4292321316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f571406b830 0x7f571415e6f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5700009870 tx=0x7f570002a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bf3b67acad51579 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:06.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.539+0000 7f5710a00640 1 -- 172.21.15.90:0/4292321316 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5700037070 con 0x7f571406b830 2024-04-25T01:16:06.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.539+0000 7f5710a00640 1 -- 172.21.15.90:0/4292321316 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5700032440 con 0x7f571406b830 2024-04-25T01:16:06.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.539+0000 7f5710a00640 1 -- 172.21.15.90:0/4292321316 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5700032810 con 0x7f571406b830 2024-04-25T01:16:06.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.540+0000 7f5713200640 1 -- 172.21.15.90:0/4292321316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f571406b830 msgr2=0x7f571415e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:06.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.540+0000 7f5713200640 1 --2- 172.21.15.90:0/4292321316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f571406b830 0x7f571415e6f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5700009870 tx=0x7f570002a250 comp rx=0 tx=0).stop 2024-04-25T01:16:06.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.540+0000 7f5713200640 1 -- 172.21.15.90:0/4292321316 shutdown_connections 2024-04-25T01:16:06.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.540+0000 7f5713200640 1 --2- 172.21.15.90:0/4292321316 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57140ae9d0 0x7f571415ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:06.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.540+0000 7f5713200640 1 --2- 172.21.15.90:0/4292321316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f571406b830 0x7f571415e6f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:06.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.540+0000 7f5713200640 1 -- 172.21.15.90:0/4292321316 >> 172.21.15.90:0/4292321316 conn(0x7f57140b1660 msgr2=0x7f57140b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:06.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.540+0000 7f5713200640 1 -- 172.21.15.90:0/4292321316 shutdown_connections 2024-04-25T01:16:06.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.540+0000 7f5713200640 1 -- 172.21.15.90:0/4292321316 wait complete. 2024-04-25T01:16:06.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.542+0000 7f5713200640 1 Processor -- start 2024-04-25T01:16:06.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.542+0000 7f5713200640 1 -- start start 2024-04-25T01:16:06.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.543+0000 7f5713200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f571406b830 0x7f57141443e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:06.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.543+0000 7f5713200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57140ae9d0 0x7f5714160fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:06.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.543+0000 7f5713200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57140b41f0 0x7f57140afbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:06.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.543+0000 7f5713200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5714157db0 con 0x7f571406b830 2024-04-25T01:16:06.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.543+0000 7f5713200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5714157c30 con 0x7f57140b41f0 2024-04-25T01:16:06.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.543+0000 7f5713200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5714157f30 con 0x7f57140ae9d0 2024-04-25T01:16:06.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.543+0000 7f5711e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f571406b830 0x7f57141443e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:06.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.543+0000 7f5711e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f571406b830 0x7f57141443e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33650/0 (socket says 172.21.15.90:33650) 2024-04-25T01:16:06.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.543+0000 7f5711e00640 1 -- 172.21.15.90:0/3975215316 learned_addr learned my addr 172.21.15.90:0/3975215316 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:06.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.543+0000 7f5711400640 1 --2- 172.21.15.90:0/3975215316 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57140ae9d0 0x7f5714160fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:06.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.543+0000 7f5712800640 1 --2- 172.21.15.90:0/3975215316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57140b41f0 0x7f57140afbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:06.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.544+0000 7f5712800640 1 -- 172.21.15.90:0/3975215316 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57140ae9d0 msgr2=0x7f5714160fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:06.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.544+0000 7f5712800640 1 --2- 172.21.15.90:0/3975215316 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57140ae9d0 0x7f5714160fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:06.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.544+0000 7f5712800640 1 -- 172.21.15.90:0/3975215316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f571406b830 msgr2=0x7f57141443e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:06.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.544+0000 7f5712800640 1 --2- 172.21.15.90:0/3975215316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f571406b830 0x7f57141443e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:06.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.544+0000 7f5712800640 1 -- 172.21.15.90:0/3975215316 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5714144920 con 0x7f57140b41f0 2024-04-25T01:16:06.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.544+0000 7f5711e00640 1 --2- 172.21.15.90:0/3975215316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f571406b830 0x7f57141443e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:16:06.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.544+0000 7f5712800640 1 --2- 172.21.15.90:0/3975215316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57140b41f0 0x7f57140afbb0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f570c00efc0 tx=0x7f570c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:06.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.544+0000 7f570a000640 1 -- 172.21.15.90:0/3975215316 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570c018020 con 0x7f57140b41f0 2024-04-25T01:16:06.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.544+0000 7f570a000640 1 -- 172.21.15.90:0/3975215316 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f570c014480 con 0x7f57140b41f0 2024-04-25T01:16:06.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.544+0000 7f570a000640 1 -- 172.21.15.90:0/3975215316 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570c014870 con 0x7f57140b41f0 2024-04-25T01:16:06.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.544+0000 7f5713200640 1 -- 172.21.15.90:0/3975215316 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57141e6b60 con 0x7f57140b41f0 2024-04-25T01:16:06.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.545+0000 7f5713200640 1 -- 172.21.15.90:0/3975215316 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57141e7120 con 0x7f57140b41f0 2024-04-25T01:16:06.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.545+0000 7f570a000640 1 -- 172.21.15.90:0/3975215316 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f570c005d10 con 0x7f57140b41f0 2024-04-25T01:16:06.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.545+0000 7f570a000640 1 -- 172.21.15.90:0/3975215316 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f570c014da0 con 0x7f57140b41f0 2024-04-25T01:16:06.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.545+0000 7f5713200640 1 -- 172.21.15.90:0/3975215316 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5714075350 con 0x7f57140b41f0 2024-04-25T01:16:06.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.548+0000 7f570a000640 1 -- 172.21.15.90:0/3975215316 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f570c022030 con 0x7f57140b41f0 2024-04-25T01:16:06.549 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-04-25T01:16:06.564+0000 7fa6dff67040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:16:06.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.612+0000 7f5713200640 1 -- 172.21.15.90:0/3975215316 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f57140b2a30 con 0x7f57140b41f0 2024-04-25T01:16:06.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.614+0000 7f570a000640 1 -- 172.21.15.90:0/3975215316 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f5714144920 con 0x7f57140b41f0 2024-04-25T01:16:06.599 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:16:06.600 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":10,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:16:06.250204+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:16:06.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.616+0000 7f5713200640 1 -- 172.21.15.90:0/3975215316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57140b41f0 msgr2=0x7f57140afbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:06.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.616+0000 7f5713200640 1 --2- 172.21.15.90:0/3975215316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57140b41f0 0x7f57140afbb0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f570c00efc0 tx=0x7f570c00c490 comp rx=0 tx=0).stop 2024-04-25T01:16:06.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.616+0000 7f5713200640 1 -- 172.21.15.90:0/3975215316 shutdown_connections 2024-04-25T01:16:06.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.616+0000 7f5713200640 1 --2- 172.21.15.90:0/3975215316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57140b41f0 0x7f57140afbb0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:06.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.616+0000 7f5713200640 1 --2- 172.21.15.90:0/3975215316 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57140ae9d0 0x7f5714160fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:06.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.616+0000 7f5713200640 1 --2- 172.21.15.90:0/3975215316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f571406b830 0x7f57141443e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:06.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.616+0000 7f5713200640 1 -- 172.21.15.90:0/3975215316 >> 172.21.15.90:0/3975215316 conn(0x7f57140b1660 msgr2=0x7f57140b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:06.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.616+0000 7f5713200640 1 -- 172.21.15.90:0/3975215316 shutdown_connections 2024-04-25T01:16:06.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:06.616+0000 7f5713200640 1 -- 172.21.15.90:0/3975215316 wait complete. 2024-04-25T01:16:06.617 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-25T01:16:06.617 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-25T01:16:06.625 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-25T01:16:06.641+0000 7f8b2b323040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:16:06.680 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-25T01:16:06.696+0000 7f8b2b323040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:16:06.734 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-04-25T01:16:06.749+0000 7fa6dff67040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:16:06.864 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-25T01:16:06.879+0000 7f8b2b323040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:16:06.902 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-04-25T01:16:06.917+0000 7fa6dff67040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:16:06.960 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-04-25T01:16:06.975+0000 7fa6dff67040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:16:06.976 INFO:tasks.ceph.osd.7.smithi103.stderr:2024-04-25T01:16:06.991+0000 7fc3b0b68640 -1 Falling back to public interface 2024-04-25T01:16:06.976 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-25T01:16:06.991+0000 7f1523f57640 -1 Falling back to public interface 2024-04-25T01:16:06.976 INFO:tasks.ceph.osd.7.smithi103.stderr:2024-04-25T01:16:06.992+0000 7fc3af600640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6801/0: (98) Address already in use 2024-04-25T01:16:06.977 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-25T01:16:06.993+0000 7f5a06ba7640 -1 Falling back to public interface 2024-04-25T01:16:06.990 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-25T01:16:07.006+0000 7fdf6562c640 -1 Falling back to public interface 2024-04-25T01:16:06.993 INFO:tasks.ceph.osd.3.smithi090.stderr:2024-04-25T01:16:07.008+0000 7fbab0021640 -1 Falling back to public interface 2024-04-25T01:16:06.995 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-25T01:16:07.011+0000 7ff9f33d8640 -1 Falling back to public interface 2024-04-25T01:16:06.997 INFO:tasks.ceph.osd.6.smithi103.stderr:2024-04-25T01:16:07.013+0000 7f4e8f0e5640 -1 Falling back to public interface 2024-04-25T01:16:07.009 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-25T01:16:07.025+0000 7fe57ba18640 -1 Falling back to public interface 2024-04-25T01:16:07.037 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-25T01:16:07.053+0000 7f8b2b323040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:16:07.049 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-04-25T01:16:07.064+0000 7fa6dff67040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:16:07.094 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-25T01:16:07.110+0000 7f8b2b323040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:16:07.101 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-04-25T01:16:07.117+0000 7fa6dff67040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:16:07.173 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-25T01:16:07.189+0000 7f8b2b323040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:16:07.225 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-25T01:16:07.241+0000 7f8b2b323040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:16:07.423 DEBUG:teuthology.orchestra.run.smithi090:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T01:16:07.582 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.597+0000 7f544a400640 1 Processor -- start 2024-04-25T01:16:07.582 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.598+0000 7f544a400640 1 -- start start 2024-04-25T01:16:07.583 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.599+0000 7f544a400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f54440d46e0 0x7f5444168c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:07.584 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.599+0000 7f544a400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54440b3610 con 0x7f544415e1e0 2024-04-25T01:16:07.584 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.599+0000 7f544a400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54440d1bb0 con 0x7f54440d2b90 2024-04-25T01:16:07.584 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.599+0000 7f544a400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f54440b1940 con 0x7f54440d46e0 2024-04-25T01:16:07.584 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.599+0000 7f5443400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f544415e1e0 0x7f54440d2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50434/0 (socket says 172.21.15.90:50434) 2024-04-25T01:16:07.584 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.599+0000 7f5443400640 1 -- 172.21.15.90:0/3989276357 learned_addr learned my addr 172.21.15.90:0/3989276357 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:07.584 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.599+0000 7f5443e00640 1 --2- 172.21.15.90:0/3989276357 >> v2:172.21.15.90:3300/0 conn(0x7f54440d46e0 0x7f5444168c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:07.584 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.600+0000 7f5442a00640 1 -- 172.21.15.90:0/3989276357 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1109348196 0 0) 0x7f54440b3610 con 0x7f544415e1e0 2024-04-25T01:16:07.585 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.600+0000 7f5442a00640 1 -- 172.21.15.90:0/3989276357 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5428003520 con 0x7f544415e1e0 2024-04-25T01:16:07.585 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.600+0000 7f5443e00640 1 -- 172.21.15.90:0/3989276357 >> v1:172.21.15.103:6789/0 conn(0x7f54440d2b90 legacy=0x7f54440b1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:07.585 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.600+0000 7f5443e00640 1 -- 172.21.15.90:0/3989276357 >> v1:172.21.15.90:6789/0 conn(0x7f544415e1e0 legacy=0x7f54440d2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:07.585 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.600+0000 7f5443e00640 1 -- 172.21.15.90:0/3989276357 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54440d1e90 con 0x7f54440d46e0 2024-04-25T01:16:07.585 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.601+0000 7f5443e00640 1 --2- 172.21.15.90:0/3989276357 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54440d46e0 0x7f5444168c70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f543c009d90 tx=0x7f543c02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4300a4f4b29a1780 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:07.586 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.601+0000 7f5442a00640 1 -- 172.21.15.90:0/3989276357 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f543c007740 con 0x7f54440d46e0 2024-04-25T01:16:07.586 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.601+0000 7f5442a00640 1 -- 172.21.15.90:0/3989276357 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f543c0078e0 con 0x7f54440d46e0 2024-04-25T01:16:07.586 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.601+0000 7f5442a00640 1 -- 172.21.15.90:0/3989276357 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f543c007cb0 con 0x7f54440d46e0 2024-04-25T01:16:07.587 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.602+0000 7f544a400640 1 -- 172.21.15.90:0/3989276357 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54440d46e0 msgr2=0x7f5444168c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:07.587 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.602+0000 7f544a400640 1 --2- 172.21.15.90:0/3989276357 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54440d46e0 0x7f5444168c70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f543c009d90 tx=0x7f543c02a6f0 comp rx=0 tx=0).stop 2024-04-25T01:16:07.587 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.602+0000 7f544a400640 1 -- 172.21.15.90:0/3989276357 shutdown_connections 2024-04-25T01:16:07.587 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.602+0000 7f544a400640 1 --2- 172.21.15.90:0/3989276357 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54440d46e0 0x7f5444168c70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:07.587 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.602+0000 7f544a400640 1 -- 172.21.15.90:0/3989276357 >> 172.21.15.90:0/3989276357 conn(0x7f544409d360 msgr2=0x7f54440d3fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:07.587 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.602+0000 7f544a400640 1 -- 172.21.15.90:0/3989276357 shutdown_connections 2024-04-25T01:16:07.587 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.602+0000 7f544a400640 1 -- 172.21.15.90:0/3989276357 wait complete. 2024-04-25T01:16:07.588 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.604+0000 7f544a400640 1 Processor -- start 2024-04-25T01:16:07.588 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.604+0000 7f544a400640 1 -- start start 2024-04-25T01:16:07.591 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.605+0000 7f544a400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54440d2b90 0x7f544414c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:07.591 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.605+0000 7f544a400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54440d46e0 0x7f54441605d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:07.591 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.605+0000 7f544a400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f544415e1e0 0x7f5444160b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:07.591 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.605+0000 7f544a400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f54440cfcb0 con 0x7f54440d2b90 2024-04-25T01:16:07.591 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.605+0000 7f544a400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f54440b15d0 con 0x7f54440d46e0 2024-04-25T01:16:07.591 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.605+0000 7f544a400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f54440cfe30 con 0x7f544415e1e0 2024-04-25T01:16:07.591 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.605+0000 7f5443400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54440d2b90 0x7f544414c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:07.591 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.605+0000 7f5443400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54440d2b90 0x7f544414c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33670/0 (socket says 172.21.15.90:33670) 2024-04-25T01:16:07.591 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.605+0000 7f5443400640 1 -- 172.21.15.90:0/4270807108 learned_addr learned my addr 172.21.15.90:0/4270807108 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:07.591 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.605+0000 7f543be00640 1 --2- 172.21.15.90:0/4270807108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54440d46e0 0x7f54441605d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:07.592 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.605+0000 7f5443e00640 1 --2- 172.21.15.90:0/4270807108 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f544415e1e0 0x7f5444160b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:07.592 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.606+0000 7f5443400640 1 -- 172.21.15.90:0/4270807108 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f544415e1e0 msgr2=0x7f5444160b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:07.592 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.606+0000 7f5443400640 1 --2- 172.21.15.90:0/4270807108 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f544415e1e0 0x7f5444160b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:07.592 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.606+0000 7f5443400640 1 -- 172.21.15.90:0/4270807108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54440d46e0 msgr2=0x7f54441605d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:07.592 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.606+0000 7f5443400640 1 --2- 172.21.15.90:0/4270807108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54440d46e0 0x7f54441605d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:07.592 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.606+0000 7f5443400640 1 -- 172.21.15.90:0/4270807108 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f544414c550 con 0x7f54440d2b90 2024-04-25T01:16:07.592 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.606+0000 7f5443400640 1 --2- 172.21.15.90:0/4270807108 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54440d2b90 0x7f544414c010 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f543400c410 tx=0x7f543400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:07.592 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.606+0000 7f543b400640 1 -- 172.21.15.90:0/4270807108 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5434017070 con 0x7f54440d2b90 2024-04-25T01:16:07.592 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.606+0000 7f543b400640 1 -- 172.21.15.90:0/4270807108 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5434005ce0 con 0x7f54440d2b90 2024-04-25T01:16:07.592 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.606+0000 7f544a400640 1 -- 172.21.15.90:0/4270807108 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54440b9210 con 0x7f54440d2b90 2024-04-25T01:16:07.592 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.606+0000 7f543b400640 1 -- 172.21.15.90:0/4270807108 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54340072f0 con 0x7f54440d2b90 2024-04-25T01:16:07.592 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.606+0000 7f544a400640 1 -- 172.21.15.90:0/4270807108 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54440b97d0 con 0x7f54440d2b90 2024-04-25T01:16:07.592 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.606+0000 7f543b400640 1 -- 172.21.15.90:0/4270807108 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f543401a070 con 0x7f54440d2b90 2024-04-25T01:16:07.592 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.607+0000 7f543b400640 1 -- 172.21.15.90:0/4270807108 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f5434007490 con 0x7f54440d2b90 2024-04-25T01:16:07.593 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.607+0000 7f544a400640 1 -- 172.21.15.90:0/4270807108 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f544404f500 con 0x7f54440d2b90 2024-04-25T01:16:07.594 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.610+0000 7f543b400640 1 -- 172.21.15.90:0/4270807108 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f5434021020 con 0x7f54440d2b90 2024-04-25T01:16:07.648 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-04-25T01:16:07.664+0000 7fa6dff67040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:16:07.661 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.676+0000 7f544a400640 1 -- 172.21.15.90:0/4270807108 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f54440d42b0 con 0x7f54440d2b90 2024-04-25T01:16:07.662 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.677+0000 7f543b400640 1 -- 172.21.15.90:0/4270807108 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f543401c440 con 0x7f54440d2b90 2024-04-25T01:16:07.662 INFO:teuthology.misc.health.smithi090.stdout: 2024-04-25T01:16:07.662 INFO:teuthology.misc.health.smithi090.stdout:{"epoch":10,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:16:06.250204+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:16:07.663 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.678+0000 7f544a400640 1 -- 172.21.15.90:0/4270807108 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54440d2b90 msgr2=0x7f544414c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:07.663 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.678+0000 7f544a400640 1 --2- 172.21.15.90:0/4270807108 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54440d2b90 0x7f544414c010 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f543400c410 tx=0x7f543400c8d0 comp rx=0 tx=0).stop 2024-04-25T01:16:07.663 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.678+0000 7f544a400640 1 -- 172.21.15.90:0/4270807108 shutdown_connections 2024-04-25T01:16:07.663 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.678+0000 7f544a400640 1 --2- 172.21.15.90:0/4270807108 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f544415e1e0 0x7f5444160b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:07.663 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.678+0000 7f544a400640 1 --2- 172.21.15.90:0/4270807108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54440d46e0 0x7f54441605d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:07.663 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.678+0000 7f544a400640 1 --2- 172.21.15.90:0/4270807108 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54440d2b90 0x7f544414c010 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:07.663 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.678+0000 7f544a400640 1 -- 172.21.15.90:0/4270807108 >> 172.21.15.90:0/4270807108 conn(0x7f544409d360 msgr2=0x7f54440cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:07.664 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.679+0000 7f544a400640 1 -- 172.21.15.90:0/4270807108 shutdown_connections 2024-04-25T01:16:07.664 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:07.679+0000 7f544a400640 1 -- 172.21.15.90:0/4270807108 wait complete. 2024-04-25T01:16:07.675 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-25T01:16:07.761 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-25T01:16:07.777+0000 7f8b2b323040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:16:07.780 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-04-25T01:16:07.796+0000 7fa6dff67040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:16:07.903 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-25T01:16:07.918+0000 7f8b2b323040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:16:08.835 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-04-25T01:16:08.850+0000 7fa681000640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-25T01:16:08.836 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-04-25T01:16:08.851+0000 7fa681000640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-25T01:16:10.544 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-25T01:16:10.559+0000 7f5a06ba7640 -1 osd.1 0 log_to_monitors true 2024-04-25T01:16:10.595 INFO:tasks.ceph.osd.7.smithi103.stderr:2024-04-25T01:16:10.610+0000 7fc3b0b68640 -1 osd.7 0 log_to_monitors true 2024-04-25T01:16:10.643 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-25T01:16:10.658+0000 7ff9f33d8640 -1 osd.0 0 log_to_monitors true 2024-04-25T01:16:10.644 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-25T01:16:10.660+0000 7fdf6562c640 -1 osd.2 0 log_to_monitors true 2024-04-25T01:16:10.656 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-25T01:16:10.672+0000 7fe57ba18640 -1 osd.4 0 log_to_monitors true 2024-04-25T01:16:10.657 INFO:tasks.ceph.osd.3.smithi090.stderr:2024-04-25T01:16:10.673+0000 7fbab0021640 -1 osd.3 0 log_to_monitors true 2024-04-25T01:16:10.852 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-25T01:16:10.867+0000 7f1523f57640 -1 osd.5 0 log_to_monitors true 2024-04-25T01:16:10.910 INFO:tasks.ceph.osd.6.smithi103.stderr:2024-04-25T01:16:10.926+0000 7f4e8f0e5640 -1 osd.6 0 log_to_monitors true 2024-04-25T01:16:13.409 INFO:tasks.ceph.osd.7.smithi103.stderr:2024-04-25T01:16:13.424+0000 7fc3aba00640 -1 osd.7 0 waiting for initial osdmap 2024-04-25T01:16:13.427 INFO:tasks.ceph.osd.7.smithi103.stderr:2024-04-25T01:16:13.443+0000 7fc3a1a00640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:16:13.440 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-25T01:16:13.455+0000 7fe576200640 -1 osd.4 0 waiting for initial osdmap 2024-04-25T01:16:13.458 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-25T01:16:13.474+0000 7fe56c200640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:16:14.139 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-25T01:16:14.154+0000 7fdf60400640 -1 osd.2 0 waiting for initial osdmap 2024-04-25T01:16:14.152 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-25T01:16:14.168+0000 7fdf56400640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:16:14.165 INFO:tasks.ceph.osd.3.smithi090.stderr:2024-04-25T01:16:14.181+0000 7fbaaae00640 -1 osd.3 0 waiting for initial osdmap 2024-04-25T01:16:14.174 INFO:tasks.ceph.osd.3.smithi090.stderr:2024-04-25T01:16:14.189+0000 7fbaa0e00640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:16:14.268 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-25T01:16:14.284+0000 7ff9ee200640 -1 osd.0 0 waiting for initial osdmap 2024-04-25T01:16:14.270 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-25T01:16:14.286+0000 7f5a01a00640 -1 osd.1 0 waiting for initial osdmap 2024-04-25T01:16:14.282 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-25T01:16:14.298+0000 7ff9e4200640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:16:14.291 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-25T01:16:14.307+0000 7f59f7a00640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:16:14.358 INFO:tasks.ceph.osd.6.smithi103.stderr:2024-04-25T01:16:14.374+0000 7f4e8a000640 -1 osd.6 0 waiting for initial osdmap 2024-04-25T01:16:14.364 INFO:tasks.ceph.osd.6.smithi103.stderr:2024-04-25T01:16:14.380+0000 7f4e80000640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:16:14.387 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-25T01:16:14.403+0000 7f151ee00640 -1 osd.5 0 waiting for initial osdmap 2024-04-25T01:16:14.398 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-25T01:16:14.413+0000 7f1514e00640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:16:14.482 DEBUG:teuthology.orchestra.run.smithi090:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T01:16:14.636 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.651+0000 7fc21ea00640 1 Processor -- start 2024-04-25T01:16:14.636 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.651+0000 7fc21ea00640 1 -- start start 2024-04-25T01:16:14.638 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.653+0000 7fc21ea00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc21806dae0 0x7fc21815c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:14.638 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.653+0000 7fc21ea00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc2180715b0 0x7fc2180cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:14.638 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.653+0000 7fc21ea00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2180719e0 con 0x7fc21815cfb0 2024-04-25T01:16:14.638 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.653+0000 7fc21ea00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fc21815d390 con 0x7fc2180715b0 2024-04-25T01:16:14.638 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.653+0000 7fc21ea00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc218072880 con 0x7fc21806dae0 2024-04-25T01:16:14.638 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.653+0000 7fc21d600640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc21815cfb0 0x7fc21806d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50448/0 (socket says 172.21.15.90:50448) 2024-04-25T01:16:14.638 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.653+0000 7fc21d600640 1 -- 172.21.15.90:0/1648252879 learned_addr learned my addr 172.21.15.90:0/1648252879 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:14.639 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.653+0000 7fc21e000640 1 --2- 172.21.15.90:0/1648252879 >> v2:172.21.15.90:3300/0 conn(0x7fc2180715b0 0x7fc2180cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:14.639 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.654+0000 7fc21cc00640 1 --2- 172.21.15.90:0/1648252879 >> v2:172.21.15.103:3300/0 conn(0x7fc21806dae0 0x7fc21815c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:14.639 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.654+0000 7fc21e000640 1 -- 172.21.15.90:0/1648252879 >> v2:172.21.15.103:3300/0 conn(0x7fc21806dae0 msgr2=0x7fc21815c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:14.639 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.654+0000 7fc21e000640 1 --2- 172.21.15.90:0/1648252879 >> v2:172.21.15.103:3300/0 conn(0x7fc21806dae0 0x7fc21815c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:14.639 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.654+0000 7fc21e000640 1 -- 172.21.15.90:0/1648252879 >> v1:172.21.15.90:6789/0 conn(0x7fc21815cfb0 legacy=0x7fc21806d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:14.639 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.654+0000 7fc217e00640 1 -- 172.21.15.90:0/1648252879 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4076153282 0 0) 0x7fc2180719e0 con 0x7fc21815cfb0 2024-04-25T01:16:14.639 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.654+0000 7fc21e000640 1 -- 172.21.15.90:0/1648252879 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc218071c70 con 0x7fc2180715b0 2024-04-25T01:16:14.640 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.655+0000 7fc21e000640 1 --2- 172.21.15.90:0/1648252879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2180715b0 0x7fc2180cd9e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc2180b4e00 tx=0x7fc21002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad84f3cdb1336114 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:14.640 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.655+0000 7fc217e00640 1 -- 172.21.15.90:0/1648252879 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc210007740 con 0x7fc2180715b0 2024-04-25T01:16:14.640 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.655+0000 7fc217e00640 1 -- 172.21.15.90:0/1648252879 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2100078e0 con 0x7fc2180715b0 2024-04-25T01:16:14.640 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.655+0000 7fc217e00640 1 -- 172.21.15.90:0/1648252879 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc210007cb0 con 0x7fc2180715b0 2024-04-25T01:16:14.641 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.655+0000 7fc21ea00640 1 -- 172.21.15.90:0/1648252879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2180715b0 msgr2=0x7fc2180cd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:14.641 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.655+0000 7fc21ea00640 1 --2- 172.21.15.90:0/1648252879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2180715b0 0x7fc2180cd9e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc2180b4e00 tx=0x7fc21002a680 comp rx=0 tx=0).stop 2024-04-25T01:16:14.641 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.655+0000 7fc21ea00640 1 -- 172.21.15.90:0/1648252879 shutdown_connections 2024-04-25T01:16:14.641 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.655+0000 7fc21ea00640 1 --2- 172.21.15.90:0/1648252879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2180715b0 0x7fc2180cd9e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:14.641 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.655+0000 7fc21ea00640 1 --2- 172.21.15.90:0/1648252879 >> v2:172.21.15.103:3300/0 conn(0x7fc21806dae0 0x7fc21815c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:14.641 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.655+0000 7fc21ea00640 1 -- 172.21.15.90:0/1648252879 >> 172.21.15.90:0/1648252879 conn(0x7fc2180b1660 msgr2=0x7fc2180b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:14.641 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.656+0000 7fc21ea00640 1 -- 172.21.15.90:0/1648252879 shutdown_connections 2024-04-25T01:16:14.641 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.656+0000 7fc21ea00640 1 -- 172.21.15.90:0/1648252879 wait complete. 2024-04-25T01:16:14.641 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.657+0000 7fc21ea00640 1 Processor -- start 2024-04-25T01:16:14.642 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.657+0000 7fc21ea00640 1 -- start start 2024-04-25T01:16:14.644 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.658+0000 7fc21ea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc21806dae0 0x7fc2181417b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:14.644 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.658+0000 7fc21ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2180715b0 0x7fc218141d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:14.644 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.658+0000 7fc21ea00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc21815cfb0 0x7fc2181422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:14.644 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.658+0000 7fc21ea00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc218072510 con 0x7fc2180715b0 2024-04-25T01:16:14.644 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.658+0000 7fc21ea00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc218072230 con 0x7fc21806dae0 2024-04-25T01:16:14.644 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.658+0000 7fc21ea00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc21806cba0 con 0x7fc21815cfb0 2024-04-25T01:16:14.644 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.658+0000 7fc21cc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2180715b0 0x7fc218141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:14.644 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.658+0000 7fc21cc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2180715b0 0x7fc218141d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33780/0 (socket says 172.21.15.90:33780) 2024-04-25T01:16:14.644 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.658+0000 7fc21cc00640 1 -- 172.21.15.90:0/1922509437 learned_addr learned my addr 172.21.15.90:0/1922509437 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:14.644 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.659+0000 7fc21d600640 1 --2- 172.21.15.90:0/1922509437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc21806dae0 0x7fc2181417b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:14.644 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.659+0000 7fc21cc00640 1 -- 172.21.15.90:0/1922509437 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc21815cfb0 msgr2=0x7fc2181422a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:14.644 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.659+0000 7fc21cc00640 1 --2- 172.21.15.90:0/1922509437 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc21815cfb0 0x7fc2181422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:14.645 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.659+0000 7fc21cc00640 1 -- 172.21.15.90:0/1922509437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc21806dae0 msgr2=0x7fc2181417b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:14.645 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.659+0000 7fc21cc00640 1 --2- 172.21.15.90:0/1922509437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc21806dae0 0x7fc2181417b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:14.645 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.659+0000 7fc21cc00640 1 -- 172.21.15.90:0/1922509437 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2180cd1b0 con 0x7fc2180715b0 2024-04-25T01:16:14.645 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.659+0000 7fc21cc00640 1 --2- 172.21.15.90:0/1922509437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2180715b0 0x7fc218141d60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc20800c7f0 tx=0x7fc20800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:14.645 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.659+0000 7fc215600640 1 -- 172.21.15.90:0/1922509437 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc208015070 con 0x7fc2180715b0 2024-04-25T01:16:14.645 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.659+0000 7fc215600640 1 -- 172.21.15.90:0/1922509437 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc208004240 con 0x7fc2180715b0 2024-04-25T01:16:14.645 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.659+0000 7fc215600640 1 -- 172.21.15.90:0/1922509437 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc208007620 con 0x7fc2180715b0 2024-04-25T01:16:14.645 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.659+0000 7fc21ea00640 1 -- 172.21.15.90:0/1922509437 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc218159500 con 0x7fc2180715b0 2024-04-25T01:16:14.645 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.659+0000 7fc21ea00640 1 -- 172.21.15.90:0/1922509437 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc218158d10 con 0x7fc2180715b0 2024-04-25T01:16:14.646 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.660+0000 7fc21ea00640 1 -- 172.21.15.90:0/1922509437 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc21804fd50 con 0x7fc2180715b0 2024-04-25T01:16:14.646 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.661+0000 7fc215600640 1 -- 172.21.15.90:0/1922509437 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 4) ==== 90442+0+0 (secure 0 0 0) 0x7fc2080079d0 con 0x7fc2180715b0 2024-04-25T01:16:14.647 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.662+0000 7fc215600640 1 --2- 172.21.15.90:0/1922509437 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc1fc06bda0 0x7fc1fc06e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:14.647 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.663+0000 7fc215600640 1 -- 172.21.15.90:0/1922509437 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 4471+0+0 (secure 0 0 0) 0x7fc20808ad10 con 0x7fc2180715b0 2024-04-25T01:16:14.647 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.663+0000 7fc21d600640 1 --2- 172.21.15.90:0/1922509437 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc1fc06bda0 0x7fc1fc06e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:14.647 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.663+0000 7fc21d600640 1 --2- 172.21.15.90:0/1922509437 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc1fc06bda0 0x7fc1fc06e260 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc20c000f60 tx=0x7fc20c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:14.652 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.668+0000 7fc215600640 1 -- 172.21.15.90:0/1922509437 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc208091080 con 0x7fc2180715b0 2024-04-25T01:16:14.837 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.852+0000 7fc21ea00640 1 -- 172.21.15.90:0/1922509437 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc2180008d0 con 0x7fc2180715b0 2024-04-25T01:16:14.838 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.853+0000 7fc215600640 1 -- 172.21.15.90:0/1922509437 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+10873 (secure 0 0 0) 0x7fc2180cd1b0 con 0x7fc2180715b0 2024-04-25T01:16:14.838 INFO:teuthology.misc.health.smithi090.stdout: 2024-04-25T01:16:14.838 INFO:teuthology.misc.health.smithi090.stdout:{"epoch":14,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:16:14.433294+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:16:14.840 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.856+0000 7fc21ea00640 1 -- 172.21.15.90:0/1922509437 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc1fc06bda0 msgr2=0x7fc1fc06e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:14.841 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.856+0000 7fc21ea00640 1 --2- 172.21.15.90:0/1922509437 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc1fc06bda0 0x7fc1fc06e260 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc20c000f60 tx=0x7fc20c009040 comp rx=0 tx=0).stop 2024-04-25T01:16:14.841 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.856+0000 7fc21ea00640 1 -- 172.21.15.90:0/1922509437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2180715b0 msgr2=0x7fc218141d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:14.841 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.856+0000 7fc21ea00640 1 --2- 172.21.15.90:0/1922509437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2180715b0 0x7fc218141d60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc20800c7f0 tx=0x7fc20800ccb0 comp rx=0 tx=0).stop 2024-04-25T01:16:14.841 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.856+0000 7fc21ea00640 1 -- 172.21.15.90:0/1922509437 shutdown_connections 2024-04-25T01:16:14.841 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.856+0000 7fc21ea00640 1 --2- 172.21.15.90:0/1922509437 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc1fc06bda0 0x7fc1fc06e260 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:14.841 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.856+0000 7fc21ea00640 1 --2- 172.21.15.90:0/1922509437 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc21815cfb0 0x7fc2181422a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:14.841 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.856+0000 7fc21ea00640 1 --2- 172.21.15.90:0/1922509437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2180715b0 0x7fc218141d60 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:14.841 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.856+0000 7fc21ea00640 1 --2- 172.21.15.90:0/1922509437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc21806dae0 0x7fc2181417b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:14.841 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.856+0000 7fc21ea00640 1 -- 172.21.15.90:0/1922509437 >> 172.21.15.90:0/1922509437 conn(0x7fc2180b1660 msgr2=0x7fc2180af5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:14.841 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.856+0000 7fc21ea00640 1 -- 172.21.15.90:0/1922509437 shutdown_connections 2024-04-25T01:16:14.841 INFO:teuthology.misc.health.smithi090.stderr:2024-04-25T01:16:14.856+0000 7fc21ea00640 1 -- 172.21.15.90:0/1922509437 wait complete. 2024-04-25T01:16:14.854 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-25T01:16:14.854 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-25T01:16:14.854 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T01:16:14.854 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T01:16:14.859 INFO:tasks.ceph.mds.a:Started 2024-04-25T01:16:14.859 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T01:16:14.859 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T01:16:14.863 INFO:tasks.ceph.mds.c:Started 2024-04-25T01:16:14.863 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T01:16:14.863 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T01:16:14.871 INFO:tasks.ceph.mds.b:Started 2024-04-25T01:16:14.871 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T01:16:14.871 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T01:16:14.878 INFO:tasks.ceph.mds.d:Started 2024-04-25T01:16:14.878 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-04-25T01:16:14.879 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:16:14.998 INFO:tasks.ceph.mds.c.smithi090.stdout:starting mds.c at 2024-04-25T01:16:14.998 INFO:tasks.ceph.mds.a.smithi090.stdout:starting mds.a at 2024-04-25T01:16:15.024 INFO:tasks.ceph.mds.b.smithi103.stdout:starting mds.b at 2024-04-25T01:16:15.025 INFO:tasks.ceph.mds.d.smithi103.stdout:starting mds.d at 2024-04-25T01:16:15.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.060+0000 7fa92f800640 1 Processor -- start 2024-04-25T01:16:15.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.060+0000 7fa92f800640 1 -- start start 2024-04-25T01:16:15.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.062+0000 7fa92f800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa92806b120 0x7fa9280afdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:15.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.062+0000 7fa92f800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa9280d1760 0x7fa92806b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:15.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.062+0000 7fa92f800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa92806ba40 con 0x7fa9280cea60 2024-04-25T01:16:15.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.062+0000 7fa92f800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa928071d80 con 0x7fa9280d1760 2024-04-25T01:16:15.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.062+0000 7fa92f800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa9280b0300 con 0x7fa92806b120 2024-04-25T01:16:15.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.062+0000 7fa92d800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa9280cea60 0x7fa928072210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41608/0 (socket says 172.21.15.90:41608) 2024-04-25T01:16:15.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.062+0000 7fa92d800640 1 -- 172.21.15.90:0/1568611732 learned_addr learned my addr 172.21.15.90:0/1568611732 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:15.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.062+0000 7fa92ce00640 1 --2- 172.21.15.90:0/1568611732 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa92806b120 0x7fa9280afdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:15.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.062+0000 7fa927400640 1 -- 172.21.15.90:0/1568611732 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3183531892 0 0) 0x7fa92806ba40 con 0x7fa9280cea60 2024-04-25T01:16:15.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.062+0000 7fa927e00640 1 --2- 172.21.15.90:0/1568611732 >> v2:172.21.15.103:3300/0 conn(0x7fa9280d1760 0x7fa92806b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:15.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.062+0000 7fa927400640 1 -- 172.21.15.90:0/1568611732 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa910003520 con 0x7fa9280cea60 2024-04-25T01:16:15.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.063+0000 7fa92ce00640 1 -- 172.21.15.90:0/1568611732 >> v2:172.21.15.103:3300/0 conn(0x7fa9280d1760 msgr2=0x7fa92806b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:15.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.063+0000 7fa92ce00640 1 --2- 172.21.15.90:0/1568611732 >> v2:172.21.15.103:3300/0 conn(0x7fa9280d1760 0x7fa92806b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:15.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.063+0000 7fa927400640 1 -- 172.21.15.90:0/1568611732 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2925926301 0 0) 0x7fa910003520 con 0x7fa9280cea60 2024-04-25T01:16:15.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.063+0000 7fa92ce00640 1 -- 172.21.15.90:0/1568611732 >> v1:172.21.15.90:6789/0 conn(0x7fa9280cea60 legacy=0x7fa928072210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:15.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.063+0000 7fa92ce00640 1 -- 172.21.15.90:0/1568611732 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9280b5430 con 0x7fa92806b120 2024-04-25T01:16:15.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.064+0000 7fa92ce00640 1 --2- 172.21.15.90:0/1568611732 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa92806b120 0x7fa9280afdc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa91c0097a0 tx=0x7fa91c02a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59e2e1fa9539e4b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:15.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.064+0000 7fa927400640 1 -- 172.21.15.90:0/1568611732 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa91c037070 con 0x7fa92806b120 2024-04-25T01:16:15.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.064+0000 7fa927400640 1 -- 172.21.15.90:0/1568611732 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa91c02ab10 con 0x7fa92806b120 2024-04-25T01:16:15.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.064+0000 7fa927400640 1 -- 172.21.15.90:0/1568611732 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9200019f0 con 0x7fa92806b120 2024-04-25T01:16:15.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.065+0000 7fa92f800640 1 -- 172.21.15.90:0/1568611732 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa92806b120 msgr2=0x7fa9280afdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:15.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.065+0000 7fa92f800640 1 --2- 172.21.15.90:0/1568611732 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa92806b120 0x7fa9280afdc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa91c0097a0 tx=0x7fa91c02a080 comp rx=0 tx=0).stop 2024-04-25T01:16:15.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.065+0000 7fa92f800640 1 -- 172.21.15.90:0/1568611732 shutdown_connections 2024-04-25T01:16:15.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.065+0000 7fa92f800640 1 --2- 172.21.15.90:0/1568611732 >> v2:172.21.15.103:3300/0 conn(0x7fa9280d1760 0x7fa92806b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:15.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.065+0000 7fa92f800640 1 --2- 172.21.15.90:0/1568611732 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa92806b120 0x7fa9280afdc0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:15.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.065+0000 7fa92f800640 1 -- 172.21.15.90:0/1568611732 >> 172.21.15.90:0/1568611732 conn(0x7fa9280d0b60 msgr2=0x7fa9280d0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:15.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.065+0000 7fa92f800640 1 -- 172.21.15.90:0/1568611732 shutdown_connections 2024-04-25T01:16:15.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.065+0000 7fa92f800640 1 -- 172.21.15.90:0/1568611732 wait complete. 2024-04-25T01:16:15.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.066+0000 7fa92f800640 1 Processor -- start 2024-04-25T01:16:15.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.066+0000 7fa92f800640 1 -- start start 2024-04-25T01:16:15.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.067+0000 7fa92f800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa92806b120 0x7fa928153580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:15.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.067+0000 7fa92f800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9280cea60 0x7fa9280ae130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:15.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.067+0000 7fa92f800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9280d1760 0x7fa928155bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:15.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.067+0000 7fa92f800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa9280b5cd0 con 0x7fa9280d1760 2024-04-25T01:16:15.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.067+0000 7fa92f800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa9280b59f0 con 0x7fa9280cea60 2024-04-25T01:16:15.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.067+0000 7fa92f800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa92815e530 con 0x7fa92806b120 2024-04-25T01:16:15.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.068+0000 7fa92d800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9280d1760 0x7fa928155bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:15.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.068+0000 7fa92ce00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa92806b120 0x7fa928153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:15.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.068+0000 7fa92d800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9280d1760 0x7fa928155bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38160/0 (socket says 172.21.15.90:38160) 2024-04-25T01:16:15.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.068+0000 7fa92d800640 1 -- 172.21.15.90:0/3437511213 learned_addr learned my addr 172.21.15.90:0/3437511213 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:15.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.068+0000 7fa927e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9280cea60 0x7fa9280ae130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:15.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.068+0000 7fa927e00640 1 -- 172.21.15.90:0/3437511213 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa92806b120 msgr2=0x7fa928153580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:15.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.068+0000 7fa927e00640 1 --2- 172.21.15.90:0/3437511213 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa92806b120 0x7fa928153580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:15.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.068+0000 7fa927e00640 1 -- 172.21.15.90:0/3437511213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9280d1760 msgr2=0x7fa928155bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:15.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.068+0000 7fa927e00640 1 --2- 172.21.15.90:0/3437511213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9280d1760 0x7fa928155bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:15.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.068+0000 7fa927e00640 1 -- 172.21.15.90:0/3437511213 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9280ae670 con 0x7fa9280cea60 2024-04-25T01:16:15.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.068+0000 7fa92d800640 1 --2- 172.21.15.90:0/3437511213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9280d1760 0x7fa928155bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:16:15.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.068+0000 7fa92ce00640 1 --2- 172.21.15.90:0/3437511213 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa92806b120 0x7fa928153580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:16:15.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.068+0000 7fa927e00640 1 --2- 172.21.15.90:0/3437511213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9280cea60 0x7fa9280ae130 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa91400e8a0 tx=0x7fa91400ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:15.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.068+0000 7fa924c00640 1 -- 172.21.15.90:0/3437511213 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa914017070 con 0x7fa9280cea60 2024-04-25T01:16:15.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.069+0000 7fa924c00640 1 -- 172.21.15.90:0/3437511213 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa91400c940 con 0x7fa9280cea60 2024-04-25T01:16:15.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.069+0000 7fa924c00640 1 -- 172.21.15.90:0/3437511213 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa914009e60 con 0x7fa9280cea60 2024-04-25T01:16:15.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.069+0000 7fa92f800640 1 -- 172.21.15.90:0/3437511213 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa92815abb0 con 0x7fa9280cea60 2024-04-25T01:16:15.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.069+0000 7fa92f800640 1 -- 172.21.15.90:0/3437511213 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9281553b0 con 0x7fa9280cea60 2024-04-25T01:16:15.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.070+0000 7fa92f800640 1 -- 172.21.15.90:0/3437511213 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9280b5080 con 0x7fa9280cea60 2024-04-25T01:16:15.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.071+0000 7fa924c00640 1 -- 172.21.15.90:0/3437511213 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 4) ==== 90442+0+0 (secure 0 0 0) 0x7fa91401a070 con 0x7fa9280cea60 2024-04-25T01:16:15.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.071+0000 7fa924c00640 1 --2- 172.21.15.90:0/3437511213 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa90806bda0 0x7fa90806e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:15.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.072+0000 7fa924c00640 1 -- 172.21.15.90:0/3437511213 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 4471+0+0 (secure 0 0 0) 0x7fa91408a5e0 con 0x7fa9280cea60 2024-04-25T01:16:15.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.072+0000 7fa92ce00640 1 --2- 172.21.15.90:0/3437511213 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa90806bda0 0x7fa90806e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:15.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.072+0000 7fa92ce00640 1 --2- 172.21.15.90:0/3437511213 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa90806bda0 0x7fa90806e260 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa91c034000 tx=0x7fa91c02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:15.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.077+0000 7fa924c00640 1 -- 172.21.15.90:0/3437511213 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa9280b5080 con 0x7fa9280cea60 2024-04-25T01:16:15.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.263+0000 7fa92f800640 1 -- 172.21.15.90:0/3437511213 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa9280d03b0 con 0x7fa9280cea60 2024-04-25T01:16:15.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.266+0000 7fa924c00640 1 -- 172.21.15.90:0/3437511213 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+10873 (secure 0 0 0) 0x7fa9280d03b0 con 0x7fa9280cea60 2024-04-25T01:16:15.251 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:16:15.251 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":14,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:16:14.433294+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:16:15.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.268+0000 7fa92f800640 1 -- 172.21.15.90:0/3437511213 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa90806bda0 msgr2=0x7fa90806e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:15.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.268+0000 7fa92f800640 1 --2- 172.21.15.90:0/3437511213 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa90806bda0 0x7fa90806e260 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa91c034000 tx=0x7fa91c02e000 comp rx=0 tx=0).stop 2024-04-25T01:16:15.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.268+0000 7fa92f800640 1 -- 172.21.15.90:0/3437511213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9280cea60 msgr2=0x7fa9280ae130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:15.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.268+0000 7fa92f800640 1 --2- 172.21.15.90:0/3437511213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9280cea60 0x7fa9280ae130 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa91400e8a0 tx=0x7fa91400ed60 comp rx=0 tx=0).stop 2024-04-25T01:16:15.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.268+0000 7fa92f800640 1 -- 172.21.15.90:0/3437511213 shutdown_connections 2024-04-25T01:16:15.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.268+0000 7fa92f800640 1 --2- 172.21.15.90:0/3437511213 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa90806bda0 0x7fa90806e260 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:15.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.268+0000 7fa92f800640 1 --2- 172.21.15.90:0/3437511213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9280d1760 0x7fa928155bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:15.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.268+0000 7fa92f800640 1 --2- 172.21.15.90:0/3437511213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9280cea60 0x7fa9280ae130 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:15.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.268+0000 7fa92f800640 1 --2- 172.21.15.90:0/3437511213 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa92806b120 0x7fa928153580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:15.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.268+0000 7fa92f800640 1 -- 172.21.15.90:0/3437511213 >> 172.21.15.90:0/3437511213 conn(0x7fa9280d0b60 msgr2=0x7fa928072110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:15.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.268+0000 7fa92f800640 1 -- 172.21.15.90:0/3437511213 shutdown_connections 2024-04-25T01:16:15.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.268+0000 7fa92f800640 1 -- 172.21.15.90:0/3437511213 wait complete. 2024-04-25T01:16:15.270 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2024-04-25T01:16:15.270 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:16:15.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.481+0000 7fc999c00640 1 Processor -- start 2024-04-25T01:16:15.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.481+0000 7fc999c00640 1 -- start start 2024-04-25T01:16:15.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.483+0000 7fc999c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc9940b41f0 0x7fc99406b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:15.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.483+0000 7fc999c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9940adc10 con 0x7fc99415ee00 2024-04-25T01:16:15.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.483+0000 7fc999c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9940adef0 con 0x7fc9940ae9d0 2024-04-25T01:16:15.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.483+0000 7fc999c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fc99415f240 con 0x7fc9940b41f0 2024-04-25T01:16:15.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.483+0000 7fc993e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc9940b41f0 0x7fc99406b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:15.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.483+0000 7fc993e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc9940b41f0 0x7fc99406b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38168/0 (socket says 172.21.15.90:38168) 2024-04-25T01:16:15.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.483+0000 7fc993e00640 1 -- 172.21.15.90:0/2755831332 learned_addr learned my addr 172.21.15.90:0/2755831332 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:15.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.484+0000 7fc993e00640 1 -- 172.21.15.90:0/2755831332 >> v1:172.21.15.103:6789/0 conn(0x7fc9940ae9d0 legacy=0x7fc99415e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:15.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.484+0000 7fc993e00640 1 -- 172.21.15.90:0/2755831332 >> v1:172.21.15.90:6789/0 conn(0x7fc99415ee00 legacy=0x7fc994157c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:15.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.484+0000 7fc993e00640 1 -- 172.21.15.90:0/2755831332 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9940ae1d0 con 0x7fc9940b41f0 2024-04-25T01:16:15.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.484+0000 7fc993400640 1 -- 172.21.15.90:0/2755831332 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1274890815 0 0) 0x7fc9940adc10 con 0x7fc99415ee00 2024-04-25T01:16:15.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.485+0000 7fc993e00640 1 --2- 172.21.15.90:0/2755831332 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9940b41f0 0x7fc99406b120 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc984009870 tx=0x7fc98402a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1435c6a35c5b7a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:15.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.485+0000 7fc993400640 1 -- 172.21.15.90:0/2755831332 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc984037070 con 0x7fc9940b41f0 2024-04-25T01:16:15.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.485+0000 7fc993400640 1 -- 172.21.15.90:0/2755831332 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc984032440 con 0x7fc9940b41f0 2024-04-25T01:16:15.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.485+0000 7fc993400640 1 -- 172.21.15.90:0/2755831332 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc984032810 con 0x7fc9940b41f0 2024-04-25T01:16:15.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.486+0000 7fc999c00640 1 -- 172.21.15.90:0/2755831332 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9940b41f0 msgr2=0x7fc99406b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:15.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.486+0000 7fc999c00640 1 --2- 172.21.15.90:0/2755831332 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9940b41f0 0x7fc99406b120 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc984009870 tx=0x7fc98402a2a0 comp rx=0 tx=0).stop 2024-04-25T01:16:15.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.486+0000 7fc999c00640 1 -- 172.21.15.90:0/2755831332 shutdown_connections 2024-04-25T01:16:15.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.486+0000 7fc999c00640 1 --2- 172.21.15.90:0/2755831332 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9940b41f0 0x7fc99406b120 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:15.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.486+0000 7fc999c00640 1 -- 172.21.15.90:0/2755831332 >> 172.21.15.90:0/2755831332 conn(0x7fc9940b1660 msgr2=0x7fc9940b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:15.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.486+0000 7fc999c00640 1 -- 172.21.15.90:0/2755831332 shutdown_connections 2024-04-25T01:16:15.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.486+0000 7fc999c00640 1 -- 172.21.15.90:0/2755831332 wait complete. 2024-04-25T01:16:15.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.488+0000 7fc999c00640 1 Processor -- start 2024-04-25T01:16:15.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.488+0000 7fc999c00640 1 -- start start 2024-04-25T01:16:15.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.489+0000 7fc999c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9940ae9d0 0x7fc994149430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:15.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.489+0000 7fc999c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9940b41f0 0x7fc9941601d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:15.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.489+0000 7fc999c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc99415ee00 0x7fc994160710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:15.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.489+0000 7fc999c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc99406b7e0 con 0x7fc9940b41f0 2024-04-25T01:16:15.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.489+0000 7fc999c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc99406b660 con 0x7fc9940ae9d0 2024-04-25T01:16:15.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.489+0000 7fc999c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc99406b960 con 0x7fc99415ee00 2024-04-25T01:16:15.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.489+0000 7fc98b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9940b41f0 0x7fc9941601d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:15.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.489+0000 7fc98b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9940b41f0 0x7fc9941601d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38180/0 (socket says 172.21.15.90:38180) 2024-04-25T01:16:15.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.489+0000 7fc98b400640 1 -- 172.21.15.90:0/4193075802 learned_addr learned my addr 172.21.15.90:0/4193075802 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:15.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.489+0000 7fc993e00640 1 --2- 172.21.15.90:0/4193075802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9940ae9d0 0x7fc994149430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:15.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.489+0000 7fc999200640 1 --2- 172.21.15.90:0/4193075802 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc99415ee00 0x7fc994160710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:15.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.489+0000 7fc98b400640 1 -- 172.21.15.90:0/4193075802 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc99415ee00 msgr2=0x7fc994160710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:15.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.489+0000 7fc98b400640 1 --2- 172.21.15.90:0/4193075802 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc99415ee00 0x7fc994160710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:15.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.489+0000 7fc98b400640 1 -- 172.21.15.90:0/4193075802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9940ae9d0 msgr2=0x7fc994149430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:15.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.489+0000 7fc98b400640 1 --2- 172.21.15.90:0/4193075802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9940ae9d0 0x7fc994149430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:15.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.489+0000 7fc98b400640 1 -- 172.21.15.90:0/4193075802 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc99415c250 con 0x7fc9940b41f0 2024-04-25T01:16:15.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.490+0000 7fc98b400640 1 --2- 172.21.15.90:0/4193075802 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9940b41f0 0x7fc9941601d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc97c00c430 tx=0x7fc97c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:15.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.490+0000 7fc990c00640 1 -- 172.21.15.90:0/4193075802 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97c011790 con 0x7fc9940b41f0 2024-04-25T01:16:15.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.490+0000 7fc990c00640 1 -- 172.21.15.90:0/4193075802 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc97c017070 con 0x7fc9940b41f0 2024-04-25T01:16:15.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.490+0000 7fc990c00640 1 -- 172.21.15.90:0/4193075802 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97c011bb0 con 0x7fc9940b41f0 2024-04-25T01:16:15.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.490+0000 7fc999c00640 1 -- 172.21.15.90:0/4193075802 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc99415c530 con 0x7fc9940b41f0 2024-04-25T01:16:15.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.490+0000 7fc999c00640 1 -- 172.21.15.90:0/4193075802 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc99415caf0 con 0x7fc9940b41f0 2024-04-25T01:16:15.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.491+0000 7fc999c00640 1 -- 172.21.15.90:0/4193075802 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc99406b120 con 0x7fc9940b41f0 2024-04-25T01:16:15.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.492+0000 7fc990c00640 1 -- 172.21.15.90:0/4193075802 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 4) ==== 90442+0+0 (secure 0 0 0) 0x7fc97c01c460 con 0x7fc9940b41f0 2024-04-25T01:16:15.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.493+0000 7fc990c00640 1 --2- 172.21.15.90:0/4193075802 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc97006bda0 0x7fc97006e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:15.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.493+0000 7fc990c00640 1 -- 172.21.15.90:0/4193075802 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4471+0+0 (secure 0 0 0) 0x7fc97c08a750 con 0x7fc9940b41f0 2024-04-25T01:16:15.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.493+0000 7fc993e00640 1 --2- 172.21.15.90:0/4193075802 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc97006bda0 0x7fc97006e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:15.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.494+0000 7fc993e00640 1 --2- 172.21.15.90:0/4193075802 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc97006bda0 0x7fc97006e260 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc984034000 tx=0x7fc98402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:15.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.498+0000 7fc990c00640 1 -- 172.21.15.90:0/4193075802 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc99406b120 con 0x7fc9940b41f0 2024-04-25T01:16:15.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.717+0000 7fc999c00640 1 -- 172.21.15.90:0/4193075802 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc994075350 con 0x7fc9940b41f0 2024-04-25T01:16:15.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.718+0000 7fc990c00640 1 -- 172.21.15.90:0/4193075802 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1389 (secure 0 0 0) 0x7fc97c0596a0 con 0x7fc9940b41f0 2024-04-25T01:16:15.703 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:16:15.703 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6836/704639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":704639426},{"type":"v1","addr":"172.21.15.90:6836","nonce":704639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T01:16:15.703 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 2 2024-04-25T01:16:15.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.720+0000 7fc999c00640 1 -- 172.21.15.90:0/4193075802 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc97006bda0 msgr2=0x7fc97006e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:15.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.720+0000 7fc999c00640 1 --2- 172.21.15.90:0/4193075802 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc97006bda0 0x7fc97006e260 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc984034000 tx=0x7fc98402e000 comp rx=0 tx=0).stop 2024-04-25T01:16:15.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.721+0000 7fc999c00640 1 -- 172.21.15.90:0/4193075802 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9940b41f0 msgr2=0x7fc9941601d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:15.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.721+0000 7fc999c00640 1 --2- 172.21.15.90:0/4193075802 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9940b41f0 0x7fc9941601d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc97c00c430 tx=0x7fc97c00c8f0 comp rx=0 tx=0).stop 2024-04-25T01:16:15.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.721+0000 7fc999c00640 1 -- 172.21.15.90:0/4193075802 shutdown_connections 2024-04-25T01:16:15.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.721+0000 7fc999c00640 1 --2- 172.21.15.90:0/4193075802 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc97006bda0 0x7fc97006e260 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:15.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.721+0000 7fc999c00640 1 --2- 172.21.15.90:0/4193075802 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc99415ee00 0x7fc994160710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:15.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.721+0000 7fc999c00640 1 --2- 172.21.15.90:0/4193075802 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9940b41f0 0x7fc9941601d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:15.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.721+0000 7fc999c00640 1 --2- 172.21.15.90:0/4193075802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9940ae9d0 0x7fc994149430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:15.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.721+0000 7fc999c00640 1 -- 172.21.15.90:0/4193075802 >> 172.21.15.90:0/4193075802 conn(0x7fc9940b1660 msgr2=0x7fc9940af660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:15.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.721+0000 7fc999c00640 1 -- 172.21.15.90:0/4193075802 shutdown_connections 2024-04-25T01:16:15.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:15.721+0000 7fc999c00640 1 -- 172.21.15.90:0/4193075802 wait complete. 2024-04-25T01:16:17.724 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:16:17.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.916+0000 7fa4cca00640 1 Processor -- start 2024-04-25T01:16:17.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.916+0000 7fa4cca00640 1 -- start start 2024-04-25T01:16:17.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.917+0000 7fa4cca00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa4c80cff30 0x7fa4c80d0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:17.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.917+0000 7fa4cca00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa4c806d240 0x7fa4c80aedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:17.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.917+0000 7fa4cca00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4c80d0850 con 0x7fa4c815e420 2024-04-25T01:16:17.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.917+0000 7fa4cca00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fa4c806d740 con 0x7fa4c806d240 2024-04-25T01:16:17.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.917+0000 7fa4cca00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa4c806da20 con 0x7fa4c80cff30 2024-04-25T01:16:17.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.917+0000 7fa4c6800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa4c806d240 0x7fa4c80aedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:17.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.917+0000 7fa4c6800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa4c806d240 0x7fa4c80aedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38208/0 (socket says 172.21.15.90:38208) 2024-04-25T01:16:17.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.917+0000 7fa4c6800640 1 -- 172.21.15.90:0/4180635657 learned_addr learned my addr 172.21.15.90:0/4180635657 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:17.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.918+0000 7fa4c5e00640 1 --2- 172.21.15.90:0/4180635657 >> v2:172.21.15.103:3300/0 conn(0x7fa4c80cff30 0x7fa4c80d0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:17.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.918+0000 7fa4c6800640 1 -- 172.21.15.90:0/4180635657 >> v2:172.21.15.103:3300/0 conn(0x7fa4c80cff30 msgr2=0x7fa4c80d0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:17.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.918+0000 7fa4c6800640 1 --2- 172.21.15.90:0/4180635657 >> v2:172.21.15.103:3300/0 conn(0x7fa4c80cff30 0x7fa4c80d0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:17.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.918+0000 7fa4c6800640 1 -- 172.21.15.90:0/4180635657 >> v1:172.21.15.90:6789/0 conn(0x7fa4c815e420 legacy=0x7fa4c80d0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:17.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.918+0000 7fa4c6800640 1 -- 172.21.15.90:0/4180635657 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c806dd00 con 0x7fa4c806d240 2024-04-25T01:16:17.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.919+0000 7fa4c6800640 1 --2- 172.21.15.90:0/4180635657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa4c806d240 0x7fa4c80aedd0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa4c80710f0 tx=0x7fa4c002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44af565a13e05111 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:17.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.919+0000 7fa4c5400640 1 -- 172.21.15.90:0/4180635657 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c0036070 con 0x7fa4c806d240 2024-04-25T01:16:17.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.919+0000 7fa4c5400640 1 -- 172.21.15.90:0/4180635657 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4c0031600 con 0x7fa4c806d240 2024-04-25T01:16:17.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.919+0000 7fa4c5400640 1 -- 172.21.15.90:0/4180635657 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c00319d0 con 0x7fa4c806d240 2024-04-25T01:16:17.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.920+0000 7fa4cca00640 1 -- 172.21.15.90:0/4180635657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa4c806d240 msgr2=0x7fa4c80aedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:17.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.920+0000 7fa4cca00640 1 --2- 172.21.15.90:0/4180635657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa4c806d240 0x7fa4c80aedd0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa4c80710f0 tx=0x7fa4c002a730 comp rx=0 tx=0).stop 2024-04-25T01:16:17.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.920+0000 7fa4cca00640 1 -- 172.21.15.90:0/4180635657 shutdown_connections 2024-04-25T01:16:17.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.920+0000 7fa4cca00640 1 --2- 172.21.15.90:0/4180635657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa4c806d240 0x7fa4c80aedd0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:17.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.920+0000 7fa4cca00640 1 --2- 172.21.15.90:0/4180635657 >> v2:172.21.15.103:3300/0 conn(0x7fa4c80cff30 0x7fa4c80d0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:17.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.920+0000 7fa4cca00640 1 -- 172.21.15.90:0/4180635657 >> 172.21.15.90:0/4180635657 conn(0x7fa4c80d1ca0 msgr2=0x7fa4c80d20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:17.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.920+0000 7fa4cca00640 1 -- 172.21.15.90:0/4180635657 shutdown_connections 2024-04-25T01:16:17.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.920+0000 7fa4cca00640 1 -- 172.21.15.90:0/4180635657 wait complete. 2024-04-25T01:16:17.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.922+0000 7fa4cca00640 1 Processor -- start 2024-04-25T01:16:17.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.922+0000 7fa4cca00640 1 -- start start 2024-04-25T01:16:17.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.923+0000 7fa4cca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4c806d240 0x7fa4c809d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:17.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.923+0000 7fa4cca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4c80cff30 0x7fa4c80b2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:17.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.923+0000 7fa4cca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa4c815e420 0x7fa4c80b0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:17.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.923+0000 7fa4cca00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c80af490 con 0x7fa4c815e420 2024-04-25T01:16:17.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.923+0000 7fa4cca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c80af310 con 0x7fa4c806d240 2024-04-25T01:16:17.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.923+0000 7fa4cca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa4c80af610 con 0x7fa4c80cff30 2024-04-25T01:16:17.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.923+0000 7fa4c6800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa4c815e420 0x7fa4c80b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:17.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.923+0000 7fa4c6800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa4c815e420 0x7fa4c80b0280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38210/0 (socket says 172.21.15.90:38210) 2024-04-25T01:16:17.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.923+0000 7fa4c6800640 1 -- 172.21.15.90:0/1069112415 learned_addr learned my addr 172.21.15.90:0/1069112415 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:17.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.924+0000 7fa4c5e00640 1 --2- 172.21.15.90:0/1069112415 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4c806d240 0x7fa4c809d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:17.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.924+0000 7fa4bfe00640 1 --2- 172.21.15.90:0/1069112415 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4c80cff30 0x7fa4c80b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:17.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.924+0000 7fa4c6800640 1 -- 172.21.15.90:0/1069112415 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4c80cff30 msgr2=0x7fa4c80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:17.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.924+0000 7fa4c6800640 1 --2- 172.21.15.90:0/1069112415 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4c80cff30 0x7fa4c80b2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:17.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.924+0000 7fa4c6800640 1 -- 172.21.15.90:0/1069112415 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4c806d240 msgr2=0x7fa4c809d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:17.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.924+0000 7fa4c6800640 1 --2- 172.21.15.90:0/1069112415 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4c806d240 0x7fa4c809d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:17.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.924+0000 7fa4c6800640 1 -- 172.21.15.90:0/1069112415 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c81f82c0 con 0x7fa4c815e420 2024-04-25T01:16:17.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.924+0000 7fa4c6800640 1 --2- 172.21.15.90:0/1069112415 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa4c815e420 0x7fa4c80b0280 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0009f40 tx=0x7fa4c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:17.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.924+0000 7fa4be000640 1 -- 172.21.15.90:0/1069112415 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c0036050 con 0x7fa4c815e420 2024-04-25T01:16:17.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.925+0000 7fa4be000640 1 -- 172.21.15.90:0/1069112415 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4c0042070 con 0x7fa4c815e420 2024-04-25T01:16:17.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.925+0000 7fa4be000640 1 -- 172.21.15.90:0/1069112415 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c0009510 con 0x7fa4c815e420 2024-04-25T01:16:17.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.925+0000 7fa4cca00640 1 -- 172.21.15.90:0/1069112415 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4c80b2100 con 0x7fa4c815e420 2024-04-25T01:16:17.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.925+0000 7fa4cca00640 1 -- 172.21.15.90:0/1069112415 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4c81f7800 con 0x7fa4c815e420 2024-04-25T01:16:17.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.926+0000 7fa4cca00640 1 -- 172.21.15.90:0/1069112415 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4c8068fb0 con 0x7fa4c815e420 2024-04-25T01:16:17.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.927+0000 7fa4be000640 1 -- 172.21.15.90:0/1069112415 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 4) ==== 90442+0+0 (secure 0 0 0) 0x7fa4c0007bf0 con 0x7fa4c815e420 2024-04-25T01:16:17.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.928+0000 7fa4be000640 1 --2- 172.21.15.90:0/1069112415 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa4a406bda0 0x7fa4a406e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:17.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.928+0000 7fa4be000640 1 -- 172.21.15.90:0/1069112415 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fa4c00ab830 con 0x7fa4c815e420 2024-04-25T01:16:17.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.928+0000 7fa4c5e00640 1 --2- 172.21.15.90:0/1069112415 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa4a406bda0 0x7fa4a406e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:17.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.929+0000 7fa4c5e00640 1 --2- 172.21.15.90:0/1069112415 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa4a406bda0 0x7fa4a406e260 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8006fd0 tx=0x7fa4b8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:17.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:17.933+0000 7fa4be000640 1 -- 172.21.15.90:0/1069112415 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa4c0045360 con 0x7fa4c815e420 2024-04-25T01:16:18.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.182+0000 7fa4cca00640 1 -- 172.21.15.90:0/1069112415 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4c80b2370 con 0x7fa4c815e420 2024-04-25T01:16:18.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.183+0000 7fa4be000640 1 -- 172.21.15.90:0/1069112415 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3867 (secure 0 0 0) 0x7fa4c007a570 con 0x7fa4c815e420 2024-04-25T01:16:18.168 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:16:18.168 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6834/2605945289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":2605945289},{"type":"v1","addr":"172.21.15.103:6834","nonce":2605945289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6836/704639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":704639426},{"type":"v1","addr":"172.21.15.90:6836","nonce":704639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4245,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3477867755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6835","nonce":3477867755},{"type":"v1","addr":"172.21.15.90:6837","nonce":3477867755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4254,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/4291578198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6833","nonce":4291578198},{"type":"v1","addr":"172.21.15.103:6835","nonce":4291578198}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T01:16:18.168 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 3 2024-04-25T01:16:18.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.186+0000 7fa4cca00640 1 -- 172.21.15.90:0/1069112415 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa4a406bda0 msgr2=0x7fa4a406e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:18.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.186+0000 7fa4cca00640 1 --2- 172.21.15.90:0/1069112415 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa4a406bda0 0x7fa4a406e260 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8006fd0 tx=0x7fa4b8008040 comp rx=0 tx=0).stop 2024-04-25T01:16:18.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.186+0000 7fa4cca00640 1 -- 172.21.15.90:0/1069112415 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa4c815e420 msgr2=0x7fa4c80b0280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:18.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.186+0000 7fa4cca00640 1 --2- 172.21.15.90:0/1069112415 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa4c815e420 0x7fa4c80b0280 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0009f40 tx=0x7fa4c0004a40 comp rx=0 tx=0).stop 2024-04-25T01:16:18.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.186+0000 7fa4cca00640 1 -- 172.21.15.90:0/1069112415 shutdown_connections 2024-04-25T01:16:18.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.186+0000 7fa4cca00640 1 --2- 172.21.15.90:0/1069112415 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa4a406bda0 0x7fa4a406e260 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:18.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.186+0000 7fa4cca00640 1 --2- 172.21.15.90:0/1069112415 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa4c815e420 0x7fa4c80b0280 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:18.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.186+0000 7fa4cca00640 1 --2- 172.21.15.90:0/1069112415 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4c80cff30 0x7fa4c80b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:18.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.186+0000 7fa4cca00640 1 --2- 172.21.15.90:0/1069112415 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4c806d240 0x7fa4c809d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:18.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.186+0000 7fa4cca00640 1 -- 172.21.15.90:0/1069112415 >> 172.21.15.90:0/1069112415 conn(0x7fa4c80d1ca0 msgr2=0x7fa4c80af9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:18.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.186+0000 7fa4cca00640 1 -- 172.21.15.90:0/1069112415 shutdown_connections 2024-04-25T01:16:18.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.186+0000 7fa4cca00640 1 -- 172.21.15.90:0/1069112415 wait complete. 2024-04-25T01:16:18.188 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:16:18.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.387+0000 7f2913400640 1 Processor -- start 2024-04-25T01:16:18.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.387+0000 7f2913400640 1 -- start start 2024-04-25T01:16:18.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.389+0000 7f2913400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f290c15f9b0 0x7f290c15fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:18.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.389+0000 7f2913400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f290c06d270 0x7f290c06d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:18.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.389+0000 7f2913400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f290c0b4240 con 0x7f290c145590 2024-04-25T01:16:18.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.389+0000 7f2913400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f290c160830 con 0x7f290c06d270 2024-04-25T01:16:18.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.389+0000 7f2913400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f290c160350 con 0x7f290c15f9b0 2024-04-25T01:16:18.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.389+0000 7f2911400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f290c06d270 0x7f290c06d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:18.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.389+0000 7f2911400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f290c06d270 0x7f290c06d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38222/0 (socket says 172.21.15.90:38222) 2024-04-25T01:16:18.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.389+0000 7f2911400640 1 -- 172.21.15.90:0/1104686453 learned_addr learned my addr 172.21.15.90:0/1104686453 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:18.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.390+0000 7f290be00640 1 --2- 172.21.15.90:0/1104686453 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f290c15f9b0 0x7f290c15fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:18.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.390+0000 7f2911400640 1 -- 172.21.15.90:0/1104686453 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f290c15f9b0 msgr2=0x7f290c15fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:18.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.390+0000 7f2911400640 1 --2- 172.21.15.90:0/1104686453 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f290c15f9b0 0x7f290c15fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:18.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.390+0000 7f2911400640 1 -- 172.21.15.90:0/1104686453 >> v1:172.21.15.103:6789/0 conn(0x7f290c145590 legacy=0x7f290c0b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:18.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.390+0000 7f2911400640 1 -- 172.21.15.90:0/1104686453 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f290c0b39a0 con 0x7f290c06d270 2024-04-25T01:16:18.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.390+0000 7f290be00640 1 --2- 172.21.15.90:0/1104686453 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f290c15f9b0 0x7f290c15fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:18.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.392+0000 7f2911400640 1 --2- 172.21.15.90:0/1104686453 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f290c06d270 0x7f290c06d700 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2904009e50 tx=0x7f290402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4d607cb2bdede8b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:18.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.392+0000 7f290b400640 1 -- 172.21.15.90:0/1104686453 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2904036070 con 0x7f290c06d270 2024-04-25T01:16:18.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.392+0000 7f290b400640 1 -- 172.21.15.90:0/1104686453 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2904031600 con 0x7f290c06d270 2024-04-25T01:16:18.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.392+0000 7f290b400640 1 -- 172.21.15.90:0/1104686453 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29040319d0 con 0x7f290c06d270 2024-04-25T01:16:18.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.393+0000 7f2913400640 1 -- 172.21.15.90:0/1104686453 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f290c06d270 msgr2=0x7f290c06d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:18.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.393+0000 7f2913400640 1 --2- 172.21.15.90:0/1104686453 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f290c06d270 0x7f290c06d700 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2904009e50 tx=0x7f290402a710 comp rx=0 tx=0).stop 2024-04-25T01:16:18.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.393+0000 7f2913400640 1 -- 172.21.15.90:0/1104686453 shutdown_connections 2024-04-25T01:16:18.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.393+0000 7f2913400640 1 --2- 172.21.15.90:0/1104686453 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f290c06d270 0x7f290c06d700 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:18.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.393+0000 7f2913400640 1 --2- 172.21.15.90:0/1104686453 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f290c15f9b0 0x7f290c15fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:18.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.393+0000 7f2913400640 1 -- 172.21.15.90:0/1104686453 >> 172.21.15.90:0/1104686453 conn(0x7f290c0b1820 msgr2=0x7f290c0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:18.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.393+0000 7f2913400640 1 -- 172.21.15.90:0/1104686453 shutdown_connections 2024-04-25T01:16:18.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.393+0000 7f2913400640 1 -- 172.21.15.90:0/1104686453 wait complete. 2024-04-25T01:16:18.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.395+0000 7f2913400640 1 Processor -- start 2024-04-25T01:16:18.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.395+0000 7f2913400640 1 -- start start 2024-04-25T01:16:18.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.396+0000 7f2913400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f290c06d270 0x7f290c15a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:18.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.396+0000 7f2913400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f290c145590 0x7f290c1f3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:18.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.396+0000 7f2913400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f290c15f9b0 0x7f290c071ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:18.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.396+0000 7f2913400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f290c0b3f60 con 0x7f290c06d270 2024-04-25T01:16:18.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.396+0000 7f2913400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f290c06cde0 con 0x7f290c145590 2024-04-25T01:16:18.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.396+0000 7f2913400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f290c145970 con 0x7f290c15f9b0 2024-04-25T01:16:18.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.396+0000 7f2910a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f290c06d270 0x7f290c15a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:18.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.396+0000 7f2910a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f290c06d270 0x7f290c15a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38232/0 (socket says 172.21.15.90:38232) 2024-04-25T01:16:18.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.396+0000 7f2910a00640 1 -- 172.21.15.90:0/4244615202 learned_addr learned my addr 172.21.15.90:0/4244615202 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:18.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.396+0000 7f2910a00640 1 -- 172.21.15.90:0/4244615202 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f290c15f9b0 msgr2=0x7f290c071ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:18.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.396+0000 7f2910a00640 1 --2- 172.21.15.90:0/4244615202 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f290c15f9b0 0x7f290c071ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:18.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.396+0000 7f2910a00640 1 -- 172.21.15.90:0/4244615202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f290c145590 msgr2=0x7f290c1f3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:18.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.396+0000 7f2910a00640 1 --2- 172.21.15.90:0/4244615202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f290c145590 0x7f290c1f3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:18.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.396+0000 7f2910a00640 1 -- 172.21.15.90:0/4244615202 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f290c159260 con 0x7f290c06d270 2024-04-25T01:16:18.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.397+0000 7f2910a00640 1 --2- 172.21.15.90:0/4244615202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f290c06d270 0x7f290c15a020 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f290000c450 tx=0x7f290000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:18.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.397+0000 7f2908c00640 1 -- 172.21.15.90:0/4244615202 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29000052d0 con 0x7f290c06d270 2024-04-25T01:16:18.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.397+0000 7f2908c00640 1 -- 172.21.15.90:0/4244615202 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2900005470 con 0x7f290c06d270 2024-04-25T01:16:18.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.397+0000 7f2913400640 1 -- 172.21.15.90:0/4244615202 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f290c159540 con 0x7f290c06d270 2024-04-25T01:16:18.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.397+0000 7f2908c00640 1 -- 172.21.15.90:0/4244615202 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2900012870 con 0x7f290c06d270 2024-04-25T01:16:18.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.397+0000 7f2913400640 1 -- 172.21.15.90:0/4244615202 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f290c15a560 con 0x7f290c06d270 2024-04-25T01:16:18.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.398+0000 7f2913400640 1 -- 172.21.15.90:0/4244615202 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f290c068fb0 con 0x7f290c06d270 2024-04-25T01:16:18.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.400+0000 7f2908c00640 1 -- 172.21.15.90:0/4244615202 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 4) ==== 90442+0+0 (secure 0 0 0) 0x7f2900012a10 con 0x7f290c06d270 2024-04-25T01:16:18.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.401+0000 7f2908c00640 1 --2- 172.21.15.90:0/4244615202 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f28ec06bda0 0x7f28ec06e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:18.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.401+0000 7f2908c00640 1 -- 172.21.15.90:0/4244615202 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f290008a610 con 0x7f290c06d270 2024-04-25T01:16:18.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.401+0000 7f290be00640 1 --2- 172.21.15.90:0/4244615202 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f28ec06bda0 0x7f28ec06e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:18.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.402+0000 7f290be00640 1 --2- 172.21.15.90:0/4244615202 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f28ec06bda0 0x7f28ec06e260 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f28f8006fd0 tx=0x7f28f8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:18.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.405+0000 7f2908c00640 1 -- 172.21.15.90:0/4244615202 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f29000593e0 con 0x7f290c06d270 2024-04-25T01:16:18.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.484+0000 7f2908c00640 1 -- 172.21.15.90:0/4244615202 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2900012d00 con 0x7f290c06d270 2024-04-25T01:16:18.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.611+0000 7f2913400640 1 -- 172.21.15.90:0/4244615202 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f290c06ea50 con 0x7f290c06d270 2024-04-25T01:16:18.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.613+0000 7f2908c00640 1 -- 172.21.15.90:0/4244615202 <== mon.0 v2:172.21.15.90:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12848 (secure 0 0 0) 0x7f2900077230 con 0x7f290c06d270 2024-04-25T01:16:18.599 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:16:18.599 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":18,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:16:18.457731+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:16:18.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.616+0000 7f2913400640 1 -- 172.21.15.90:0/4244615202 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f28ec06bda0 msgr2=0x7f28ec06e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:18.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.616+0000 7f2913400640 1 --2- 172.21.15.90:0/4244615202 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f28ec06bda0 0x7f28ec06e260 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f28f8006fd0 tx=0x7f28f8008040 comp rx=0 tx=0).stop 2024-04-25T01:16:18.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.616+0000 7f2913400640 1 -- 172.21.15.90:0/4244615202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f290c06d270 msgr2=0x7f290c15a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:18.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.616+0000 7f2913400640 1 --2- 172.21.15.90:0/4244615202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f290c06d270 0x7f290c15a020 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f290000c450 tx=0x7f290000c910 comp rx=0 tx=0).stop 2024-04-25T01:16:18.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.616+0000 7f2913400640 1 -- 172.21.15.90:0/4244615202 shutdown_connections 2024-04-25T01:16:18.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.616+0000 7f2913400640 1 --2- 172.21.15.90:0/4244615202 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f28ec06bda0 0x7f28ec06e260 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:18.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.616+0000 7f2913400640 1 --2- 172.21.15.90:0/4244615202 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f290c15f9b0 0x7f290c071ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:18.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.616+0000 7f2913400640 1 --2- 172.21.15.90:0/4244615202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f290c145590 0x7f290c1f3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:18.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.616+0000 7f2913400640 1 --2- 172.21.15.90:0/4244615202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f290c06d270 0x7f290c15a020 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:18.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.616+0000 7f2913400640 1 -- 172.21.15.90:0/4244615202 >> 172.21.15.90:0/4244615202 conn(0x7f290c0b1820 msgr2=0x7f290c0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:18.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.616+0000 7f2913400640 1 -- 172.21.15.90:0/4244615202 shutdown_connections 2024-04-25T01:16:18.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.616+0000 7f2913400640 1 -- 172.21.15.90:0/4244615202 wait complete. 2024-04-25T01:16:18.620 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:16:15.894126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:16:18.620 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:16:18.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.825+0000 7f882a200640 1 Processor -- start 2024-04-25T01:16:18.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.825+0000 7f882a200640 1 -- start start 2024-04-25T01:16:18.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.826+0000 7f882a200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f882415fc40 0x7f88241600a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:18.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.826+0000 7f882a200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f88240cedc0 0x7f88240cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:18.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.826+0000 7f882a200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f882406d110 con 0x7f88240722e0 2024-04-25T01:16:18.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.826+0000 7f882a200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f8824160aa0 con 0x7f882415fc40 2024-04-25T01:16:18.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.826+0000 7f882a200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f882406da40 con 0x7f88240cedc0 2024-04-25T01:16:18.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.827+0000 7f881be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f882415fc40 0x7f88241600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:18.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.827+0000 7f881be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f882415fc40 0x7f88241600a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38236/0 (socket says 172.21.15.90:38236) 2024-04-25T01:16:18.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.827+0000 7f881be00640 1 -- 172.21.15.90:0/616116925 learned_addr learned my addr 172.21.15.90:0/616116925 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:18.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.827+0000 7f8823e00640 1 --2- 172.21.15.90:0/616116925 >> v2:172.21.15.103:3300/0 conn(0x7f88240cedc0 0x7f88240cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:18.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.827+0000 7f881be00640 1 -- 172.21.15.90:0/616116925 >> v2:172.21.15.103:3300/0 conn(0x7f88240cedc0 msgr2=0x7f88240cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:18.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.827+0000 7f881be00640 1 --2- 172.21.15.90:0/616116925 >> v2:172.21.15.103:3300/0 conn(0x7f88240cedc0 0x7f88240cf1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:18.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.827+0000 7f881be00640 1 -- 172.21.15.90:0/616116925 >> v1:172.21.15.103:6789/0 conn(0x7f88240722e0 legacy=0x7f882406ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:18.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.827+0000 7f881be00640 1 -- 172.21.15.90:0/616116925 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f882406d3f0 con 0x7f882415fc40 2024-04-25T01:16:18.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.828+0000 7f881be00640 1 --2- 172.21.15.90:0/616116925 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f882415fc40 0x7f88241600a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8824072920 tx=0x7f880c02a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d32e91bfe0322e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:18.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.829+0000 7f8822a00640 1 -- 172.21.15.90:0/616116925 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c037070 con 0x7f882415fc40 2024-04-25T01:16:18.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.829+0000 7f8822a00640 1 -- 172.21.15.90:0/616116925 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f880c032440 con 0x7f882415fc40 2024-04-25T01:16:18.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.829+0000 7f8822a00640 1 -- 172.21.15.90:0/616116925 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c032810 con 0x7f882415fc40 2024-04-25T01:16:18.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.829+0000 7f882a200640 1 -- 172.21.15.90:0/616116925 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f882415fc40 msgr2=0x7f88241600a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:18.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.829+0000 7f882a200640 1 --2- 172.21.15.90:0/616116925 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f882415fc40 0x7f88241600a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8824072920 tx=0x7f880c02a200 comp rx=0 tx=0).stop 2024-04-25T01:16:18.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.829+0000 7f882a200640 1 -- 172.21.15.90:0/616116925 shutdown_connections 2024-04-25T01:16:18.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.829+0000 7f882a200640 1 --2- 172.21.15.90:0/616116925 >> v2:172.21.15.103:3300/0 conn(0x7f88240cedc0 0x7f88240cf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:18.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.829+0000 7f882a200640 1 --2- 172.21.15.90:0/616116925 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f882415fc40 0x7f88241600a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:18.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.829+0000 7f882a200640 1 -- 172.21.15.90:0/616116925 >> 172.21.15.90:0/616116925 conn(0x7f882406b720 msgr2=0x7f88240b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:18.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.829+0000 7f882a200640 1 -- 172.21.15.90:0/616116925 shutdown_connections 2024-04-25T01:16:18.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.830+0000 7f882a200640 1 -- 172.21.15.90:0/616116925 wait complete. 2024-04-25T01:16:18.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.831+0000 7f882a200640 1 Processor -- start 2024-04-25T01:16:18.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.831+0000 7f882a200640 1 -- start start 2024-04-25T01:16:18.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.832+0000 7f882a200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88240722e0 0x7f882414a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:18.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.832+0000 7f882a200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88240cedc0 0x7f88240a9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:18.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.832+0000 7f882a200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f882415fc40 0x7f8824158490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:18.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.832+0000 7f882a200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f882406cec0 con 0x7f88240722e0 2024-04-25T01:16:18.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.832+0000 7f882a200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f88240cf730 con 0x7f88240cedc0 2024-04-25T01:16:18.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.832+0000 7f882a200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8824070b50 con 0x7f882415fc40 2024-04-25T01:16:18.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.832+0000 7f8823400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88240722e0 0x7f882414a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:18.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.833+0000 7f8823400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88240722e0 0x7f882414a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38248/0 (socket says 172.21.15.90:38248) 2024-04-25T01:16:18.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.833+0000 7f8823400640 1 -- 172.21.15.90:0/146855456 learned_addr learned my addr 172.21.15.90:0/146855456 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:18.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.833+0000 7f881be00640 1 --2- 172.21.15.90:0/146855456 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88240cedc0 0x7f88240a9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:18.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.833+0000 7f8823400640 1 -- 172.21.15.90:0/146855456 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f882415fc40 msgr2=0x7f8824158490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:18.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.833+0000 7f8823400640 1 --2- 172.21.15.90:0/146855456 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f882415fc40 0x7f8824158490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:18.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.833+0000 7f8823400640 1 -- 172.21.15.90:0/146855456 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88240cedc0 msgr2=0x7f88240a9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:18.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.833+0000 7f8823400640 1 --2- 172.21.15.90:0/146855456 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88240cedc0 0x7f88240a9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:18.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.833+0000 7f8823400640 1 -- 172.21.15.90:0/146855456 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88240aa710 con 0x7f88240722e0 2024-04-25T01:16:18.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.833+0000 7f8823400640 1 --2- 172.21.15.90:0/146855456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88240722e0 0x7f882414a930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f881400c400 tx=0x7f881400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:18.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.833+0000 7f881b400640 1 -- 172.21.15.90:0/146855456 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88140052d0 con 0x7f88240722e0 2024-04-25T01:16:18.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.834+0000 7f881b400640 1 -- 172.21.15.90:0/146855456 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8814005490 con 0x7f88240722e0 2024-04-25T01:16:18.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.834+0000 7f881b400640 1 -- 172.21.15.90:0/146855456 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8814007260 con 0x7f88240722e0 2024-04-25T01:16:18.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.834+0000 7f882a200640 1 -- 172.21.15.90:0/146855456 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88240aa9f0 con 0x7f88240722e0 2024-04-25T01:16:18.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.834+0000 7f882a200640 1 -- 172.21.15.90:0/146855456 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88240aa420 con 0x7f88240722e0 2024-04-25T01:16:18.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.835+0000 7f882a200640 1 -- 172.21.15.90:0/146855456 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8824068fb0 con 0x7f88240722e0 2024-04-25T01:16:18.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.836+0000 7f881b400640 1 -- 172.21.15.90:0/146855456 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8814007400 con 0x7f88240722e0 2024-04-25T01:16:18.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.837+0000 7f881b400640 1 --2- 172.21.15.90:0/146855456 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f880006be80 0x7f880006e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:18.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.837+0000 7f881b400640 1 -- 172.21.15.90:0/146855456 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f8814089f30 con 0x7f88240722e0 2024-04-25T01:16:18.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.837+0000 7f881be00640 1 --2- 172.21.15.90:0/146855456 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f880006be80 0x7f880006e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:18.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.838+0000 7f881be00640 1 --2- 172.21.15.90:0/146855456 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f880006be80 0x7f880006e340 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f880c034000 tx=0x7f880c02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:18.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:18.841+0000 7f881b400640 1 -- 172.21.15.90:0/146855456 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8814058da0 con 0x7f88240722e0 2024-04-25T01:16:19.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.038+0000 7f882a200640 1 -- 172.21.15.90:0/146855456 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f88241602f0 con 0x7f88240722e0 2024-04-25T01:16:19.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.039+0000 7f881b400640 1 -- 172.21.15.90:0/146855456 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) ==== 93+0+10 (secure 0 0 0) 0x7f88140586a0 con 0x7f88240722e0 2024-04-25T01:16:19.025 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T01:16:19.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.042+0000 7f882a200640 1 -- 172.21.15.90:0/146855456 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f880006be80 msgr2=0x7f880006e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:19.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.042+0000 7f882a200640 1 --2- 172.21.15.90:0/146855456 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f880006be80 0x7f880006e340 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f880c034000 tx=0x7f880c02e000 comp rx=0 tx=0).stop 2024-04-25T01:16:19.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.042+0000 7f882a200640 1 -- 172.21.15.90:0/146855456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88240722e0 msgr2=0x7f882414a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:19.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.042+0000 7f882a200640 1 --2- 172.21.15.90:0/146855456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88240722e0 0x7f882414a930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f881400c400 tx=0x7f881400c8c0 comp rx=0 tx=0).stop 2024-04-25T01:16:19.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.042+0000 7f882a200640 1 -- 172.21.15.90:0/146855456 shutdown_connections 2024-04-25T01:16:19.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.042+0000 7f882a200640 1 --2- 172.21.15.90:0/146855456 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f880006be80 0x7f880006e340 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:19.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.042+0000 7f882a200640 1 --2- 172.21.15.90:0/146855456 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f882415fc40 0x7f8824158490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:19.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.042+0000 7f882a200640 1 --2- 172.21.15.90:0/146855456 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88240cedc0 0x7f88240a9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:19.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.042+0000 7f882a200640 1 --2- 172.21.15.90:0/146855456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88240722e0 0x7f882414a930 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:19.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.042+0000 7f882a200640 1 -- 172.21.15.90:0/146855456 >> 172.21.15.90:0/146855456 conn(0x7f882406b720 msgr2=0x7f882415c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:19.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.042+0000 7f882a200640 1 -- 172.21.15.90:0/146855456 shutdown_connections 2024-04-25T01:16:19.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.042+0000 7f882a200640 1 -- 172.21.15.90:0/146855456 wait complete. 2024-04-25T01:16:19.046 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T01:16:19.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.258+0000 7fe891a00640 1 Processor -- start 2024-04-25T01:16:19.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.258+0000 7fe891a00640 1 -- start start 2024-04-25T01:16:19.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.260+0000 7fe891a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe88c15fe10 0x7fe88c06d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:19.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.260+0000 7fe891a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe88c0b4240 con 0x7fe88c15f9b0 2024-04-25T01:16:19.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.260+0000 7fe891a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe88c0b39a0 con 0x7fe88c145590 2024-04-25T01:16:19.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.260+0000 7fe891a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe88c160830 con 0x7fe88c15fe10 2024-04-25T01:16:19.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.260+0000 7fe882000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe88c15f9b0 0x7fe88c06d270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41640/0 (socket says 172.21.15.90:41640) 2024-04-25T01:16:19.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.260+0000 7fe882000640 1 -- 172.21.15.90:0/1650154513 learned_addr learned my addr 172.21.15.90:0/1650154513 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:19.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.260+0000 7fe88b400640 1 --2- 172.21.15.90:0/1650154513 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe88c15fe10 0x7fe88c06d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:19.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.260+0000 7fe88a000640 1 -- 172.21.15.90:0/1650154513 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1567990843 0 0) 0x7fe88c0b4240 con 0x7fe88c15f9b0 2024-04-25T01:16:19.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.260+0000 7fe88a000640 1 -- 172.21.15.90:0/1650154513 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe870003520 con 0x7fe88c15f9b0 2024-04-25T01:16:19.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.261+0000 7fe88a000640 1 -- 172.21.15.90:0/1650154513 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2109011313 0 0) 0x7fe88c0b39a0 con 0x7fe88c145590 2024-04-25T01:16:19.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.261+0000 7fe88a000640 1 -- 172.21.15.90:0/1650154513 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe88c0b4240 con 0x7fe88c145590 2024-04-25T01:16:19.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.261+0000 7fe88a000640 1 -- 172.21.15.90:0/1650154513 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4226951646 0 0) 0x7fe870003520 con 0x7fe88c15f9b0 2024-04-25T01:16:19.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.261+0000 7fe88a000640 1 -- 172.21.15.90:0/1650154513 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe88c15fe10 msgr2=0x7fe88c06d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:19.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.261+0000 7fe88a000640 1 --2- 172.21.15.90:0/1650154513 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe88c15fe10 0x7fe88c06d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:19.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.261+0000 7fe88a000640 1 -- 172.21.15.90:0/1650154513 >> v1:172.21.15.103:6789/0 conn(0x7fe88c145590 legacy=0x7fe88c0b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:19.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.261+0000 7fe88a000640 1 -- 172.21.15.90:0/1650154513 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe88c0b3c80 con 0x7fe88c15f9b0 2024-04-25T01:16:19.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.261+0000 7fe88a000640 1 -- 172.21.15.90:0/1650154513 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7fe874001670 con 0x7fe88c15f9b0 2024-04-25T01:16:19.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.261+0000 7fe88a000640 1 -- 172.21.15.90:0/1650154513 >> v1:172.21.15.90:6789/0 conn(0x7fe88c15f9b0 legacy=0x7fe88c06d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:19.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.261+0000 7fe88a000640 1 --2- 172.21.15.90:0/1650154513 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe870003b70 0x7fe870003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:19.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.261+0000 7fe88a000640 1 --2- 172.21.15.90:0/1650154513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe870004800 0x7fe870004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:19.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.261+0000 7fe88a000640 1 --2- 172.21.15.90:0/1650154513 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe88c15fe10 0x7fe8700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:19.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.261+0000 7fe88a000640 1 -- 172.21.15.90:0/1650154513 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe87000ba70 con 0x7fe870003b70 2024-04-25T01:16:19.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.261+0000 7fe88a000640 1 -- 172.21.15.90:0/1650154513 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe87000bbf0 con 0x7fe870004800 2024-04-25T01:16:19.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.261+0000 7fe88a000640 1 -- 172.21.15.90:0/1650154513 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe87000bdf0 con 0x7fe88c15fe10 2024-04-25T01:16:19.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.261+0000 7fe88aa00640 1 --2- 172.21.15.90:0/1650154513 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe870003b70 0x7fe870003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:19.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.261+0000 7fe882000640 1 --2- 172.21.15.90:0/1650154513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe870004800 0x7fe870004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:19.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.262+0000 7fe88b400640 1 --2- 172.21.15.90:0/1650154513 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe88c15fe10 0x7fe8700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:19.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.262+0000 7fe88aa00640 1 -- 172.21.15.90:0/1650154513 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe88c15fe10 msgr2=0x7fe8700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:19.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.262+0000 7fe88aa00640 1 --2- 172.21.15.90:0/1650154513 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe88c15fe10 0x7fe8700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:19.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.262+0000 7fe88aa00640 1 -- 172.21.15.90:0/1650154513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe870004800 msgr2=0x7fe870004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:19.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.262+0000 7fe88aa00640 1 --2- 172.21.15.90:0/1650154513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe870004800 0x7fe870004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:19.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.262+0000 7fe88aa00640 1 -- 172.21.15.90:0/1650154513 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe870003520 con 0x7fe870003b70 2024-04-25T01:16:19.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.262+0000 7fe882000640 1 --2- 172.21.15.90:0/1650154513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe870004800 0x7fe870004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:16:19.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.262+0000 7fe88b400640 1 --2- 172.21.15.90:0/1650154513 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe88c15fe10 0x7fe8700051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:19.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.263+0000 7fe88aa00640 1 --2- 172.21.15.90:0/1650154513 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe870003b70 0x7fe870003f70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe87c0018d0 tx=0x7fe870005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb2b6996b1984c5d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:19.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.263+0000 7fe88a000640 1 -- 172.21.15.90:0/1650154513 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87c0014f0 con 0x7fe870003b70 2024-04-25T01:16:19.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.263+0000 7fe88a000640 1 -- 172.21.15.90:0/1650154513 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe87c0052d0 con 0x7fe870003b70 2024-04-25T01:16:19.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.263+0000 7fe88a000640 1 -- 172.21.15.90:0/1650154513 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87c02aca0 con 0x7fe870003b70 2024-04-25T01:16:19.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.264+0000 7fe891a00640 1 -- 172.21.15.90:0/1650154513 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe870003b70 msgr2=0x7fe870003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:19.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.264+0000 7fe891a00640 1 --2- 172.21.15.90:0/1650154513 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe870003b70 0x7fe870003f70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe87c0018d0 tx=0x7fe870005830 comp rx=0 tx=0).stop 2024-04-25T01:16:19.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.264+0000 7fe88b400640 1 -- 172.21.15.90:0/1650154513 reap_dead start 2024-04-25T01:16:19.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.264+0000 7fe891a00640 1 -- 172.21.15.90:0/1650154513 shutdown_connections 2024-04-25T01:16:19.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.264+0000 7fe891a00640 1 -- 172.21.15.90:0/1650154513 >> 172.21.15.90:0/1650154513 conn(0x7fe88c0b1820 msgr2=0x7fe88c0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:19.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.264+0000 7fe891a00640 1 -- 172.21.15.90:0/1650154513 shutdown_connections 2024-04-25T01:16:19.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.264+0000 7fe891a00640 1 -- 172.21.15.90:0/1650154513 wait complete. 2024-04-25T01:16:19.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.265+0000 7fe891a00640 1 Processor -- start 2024-04-25T01:16:19.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.266+0000 7fe891a00640 1 -- start start 2024-04-25T01:16:19.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.267+0000 7fe891a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe870003b70 0x7fe88c15b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:19.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.267+0000 7fe891a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe88c0ae240 0x7fe88c0ae6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:19.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.267+0000 7fe891a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe88c070da0 0x7fe88c071220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:19.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.267+0000 7fe891a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe88c1602a0 con 0x7fe88c0ae240 2024-04-25T01:16:19.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.267+0000 7fe891a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe88c06cde0 con 0x7fe870003b70 2024-04-25T01:16:19.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.267+0000 7fe891a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe88c145970 con 0x7fe88c070da0 2024-04-25T01:16:19.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.267+0000 7fe882000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe88c0ae240 0x7fe88c0ae6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:19.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.267+0000 7fe882000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe88c0ae240 0x7fe88c0ae6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38278/0 (socket says 172.21.15.90:38278) 2024-04-25T01:16:19.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.267+0000 7fe882000640 1 -- 172.21.15.90:0/2432826928 learned_addr learned my addr 172.21.15.90:0/2432826928 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:19.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.267+0000 7fe88aa00640 1 --2- 172.21.15.90:0/2432826928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe870003b70 0x7fe88c15b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:19.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.267+0000 7fe88b400640 1 --2- 172.21.15.90:0/2432826928 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe88c070da0 0x7fe88c071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:19.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.267+0000 7fe882000640 1 -- 172.21.15.90:0/2432826928 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe88c070da0 msgr2=0x7fe88c071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:19.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.267+0000 7fe882000640 1 --2- 172.21.15.90:0/2432826928 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe88c070da0 0x7fe88c071220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:19.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.267+0000 7fe882000640 1 -- 172.21.15.90:0/2432826928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe870003b70 msgr2=0x7fe88c15b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:19.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.267+0000 7fe882000640 1 --2- 172.21.15.90:0/2432826928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe870003b70 0x7fe88c15b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:19.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.267+0000 7fe882000640 1 -- 172.21.15.90:0/2432826928 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe88c0720b0 con 0x7fe88c0ae240 2024-04-25T01:16:19.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.268+0000 7fe882000640 1 --2- 172.21.15.90:0/2432826928 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe88c0ae240 0x7fe88c0ae6c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe87400e730 tx=0x7fe87400ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:19.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.268+0000 7fe883400640 1 -- 172.21.15.90:0/2432826928 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87400c6c0 con 0x7fe88c0ae240 2024-04-25T01:16:19.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.268+0000 7fe883400640 1 -- 172.21.15.90:0/2432826928 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe874018070 con 0x7fe88c0ae240 2024-04-25T01:16:19.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.268+0000 7fe891a00640 1 -- 172.21.15.90:0/2432826928 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe88c071af0 con 0x7fe88c0ae240 2024-04-25T01:16:19.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.268+0000 7fe883400640 1 -- 172.21.15.90:0/2432826928 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87400ca60 con 0x7fe88c0ae240 2024-04-25T01:16:19.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.268+0000 7fe891a00640 1 -- 172.21.15.90:0/2432826928 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe88c15dab0 con 0x7fe88c0ae240 2024-04-25T01:16:19.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.269+0000 7fe891a00640 1 -- 172.21.15.90:0/2432826928 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe88c068fb0 con 0x7fe88c0ae240 2024-04-25T01:16:19.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.270+0000 7fe883400640 1 -- 172.21.15.90:0/2432826928 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe87400cc00 con 0x7fe88c0ae240 2024-04-25T01:16:19.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.271+0000 7fe883400640 1 --2- 172.21.15.90:0/2432826928 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe86806be30 0x7fe86806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:19.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.271+0000 7fe883400640 1 -- 172.21.15.90:0/2432826928 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7fe87408a890 con 0x7fe88c0ae240 2024-04-25T01:16:19.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.271+0000 7fe88aa00640 1 --2- 172.21.15.90:0/2432826928 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe86806be30 0x7fe86806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:19.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.272+0000 7fe88aa00640 1 --2- 172.21.15.90:0/2432826928 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe86806be30 0x7fe86806e2f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe87c032000 tx=0x7fe87c009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:19.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.276+0000 7fe883400640 1 -- 172.21.15.90:0/2432826928 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe874059590 con 0x7fe88c0ae240 2024-04-25T01:16:19.411 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:16:19.417 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:16:19.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.480+0000 7fe891a00640 1 -- 172.21.15.90:0/2432826928 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fe88c06ea50 con 0x7fe88c0ae240 2024-04-25T01:16:19.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.542+0000 7fe883400640 1 -- 172.21.15.90:0/2432826928 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) ==== 84+0+65 (secure 0 0 0) 0x7fe874058e90 con 0x7fe88c0ae240 2024-04-25T01:16:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:16:19.528 INFO:teuthology.orchestra.run.smithi090.stdout:[ 2024-04-25T01:16:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T01:16:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: "poolnum": 1, 2024-04-25T01:16:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: "poolname": ".mgr" 2024-04-25T01:16:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:16:19.529 INFO:teuthology.orchestra.run.smithi090.stdout:] 2024-04-25T01:16:19.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.545+0000 7fe891a00640 1 -- 172.21.15.90:0/2432826928 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe86806be30 msgr2=0x7fe86806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:19.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.545+0000 7fe891a00640 1 --2- 172.21.15.90:0/2432826928 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe86806be30 0x7fe86806e2f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe87c032000 tx=0x7fe87c009030 comp rx=0 tx=0).stop 2024-04-25T01:16:19.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.545+0000 7fe891a00640 1 -- 172.21.15.90:0/2432826928 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe88c0ae240 msgr2=0x7fe88c0ae6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:19.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.545+0000 7fe891a00640 1 --2- 172.21.15.90:0/2432826928 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe88c0ae240 0x7fe88c0ae6c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe87400e730 tx=0x7fe87400ebf0 comp rx=0 tx=0).stop 2024-04-25T01:16:19.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.545+0000 7fe891a00640 1 -- 172.21.15.90:0/2432826928 shutdown_connections 2024-04-25T01:16:19.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.545+0000 7fe891a00640 1 --2- 172.21.15.90:0/2432826928 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe86806be30 0x7fe86806e2f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:19.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.545+0000 7fe891a00640 1 --2- 172.21.15.90:0/2432826928 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe88c070da0 0x7fe88c071220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:19.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.545+0000 7fe891a00640 1 --2- 172.21.15.90:0/2432826928 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe88c0ae240 0x7fe88c0ae6c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:19.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.545+0000 7fe891a00640 1 --2- 172.21.15.90:0/2432826928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe870003b70 0x7fe88c15b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:19.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.545+0000 7fe891a00640 1 -- 172.21.15.90:0/2432826928 >> 172.21.15.90:0/2432826928 conn(0x7fe88c0b1820 msgr2=0x7fe88c0b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:19.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.545+0000 7fe891a00640 1 -- 172.21.15.90:0/2432826928 shutdown_connections 2024-04-25T01:16:19.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.545+0000 7fe891a00640 1 -- 172.21.15.90:0/2432826928 wait complete. 2024-04-25T01:16:19.550 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T01:16:19.550 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T01:16:19.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.815+0000 7f8d7ae00640 1 Processor -- start 2024-04-25T01:16:19.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.816+0000 7f8d7ae00640 1 -- start start 2024-04-25T01:16:19.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.817+0000 7f8d7ae00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8d740d2920 0x7f8d740d4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:19.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.817+0000 7f8d7ae00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8d740b2e30 0x7f8d740b0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:19.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.817+0000 7f8d7ae00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d740d1b50 con 0x7f8d7415e270 2024-04-25T01:16:19.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.817+0000 7f8d7ae00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f8d740cf5a0 con 0x7f8d740d2920 2024-04-25T01:16:19.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.817+0000 7f8d7ae00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8d740d1e30 con 0x7f8d740b2e30 2024-04-25T01:16:19.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.817+0000 7f8d6b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8d740d2920 0x7f8d740d4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:19.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.818+0000 7f8d6b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8d740d2920 0x7f8d740d4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38290/0 (socket says 172.21.15.90:38290) 2024-04-25T01:16:19.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.818+0000 7f8d6b400640 1 -- 172.21.15.90:0/1613157913 learned_addr learned my addr 172.21.15.90:0/1613157913 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:19.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.818+0000 7f8d78e00640 1 --2- 172.21.15.90:0/1613157913 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8d740b2e30 0x7f8d740b0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:19.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.818+0000 7f8d6b400640 1 -- 172.21.15.90:0/1613157913 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8d740b2e30 msgr2=0x7f8d740b0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:19.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.818+0000 7f8d6b400640 1 --2- 172.21.15.90:0/1613157913 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8d740b2e30 0x7f8d740b0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:19.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.818+0000 7f8d6b400640 1 -- 172.21.15.90:0/1613157913 >> v1:172.21.15.103:6789/0 conn(0x7f8d7415e270 legacy=0x7f8d740d2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:19.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.818+0000 7f8d6b400640 1 -- 172.21.15.90:0/1613157913 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d740cf7f0 con 0x7f8d740d2920 2024-04-25T01:16:19.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.819+0000 7f8d6b400640 1 --2- 172.21.15.90:0/1613157913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d740d2920 0x7f8d740d4a70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c009870 tx=0x7f8d5c02a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b764c85795a97919 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:19.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.819+0000 7f8d73400640 1 -- 172.21.15.90:0/1613157913 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c037070 con 0x7f8d740d2920 2024-04-25T01:16:19.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.819+0000 7f8d73400640 1 -- 172.21.15.90:0/1613157913 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d5c032440 con 0x7f8d740d2920 2024-04-25T01:16:19.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.819+0000 7f8d73400640 1 -- 172.21.15.90:0/1613157913 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c032810 con 0x7f8d740d2920 2024-04-25T01:16:19.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.820+0000 7f8d7ae00640 1 -- 172.21.15.90:0/1613157913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d740d2920 msgr2=0x7f8d740d4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:19.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.820+0000 7f8d7ae00640 1 --2- 172.21.15.90:0/1613157913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d740d2920 0x7f8d740d4a70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c009870 tx=0x7f8d5c02a130 comp rx=0 tx=0).stop 2024-04-25T01:16:19.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.820+0000 7f8d7ae00640 1 -- 172.21.15.90:0/1613157913 shutdown_connections 2024-04-25T01:16:19.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.820+0000 7f8d7ae00640 1 --2- 172.21.15.90:0/1613157913 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8d740b2e30 0x7f8d740b0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:19.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.820+0000 7f8d7ae00640 1 --2- 172.21.15.90:0/1613157913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d740d2920 0x7f8d740d4a70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:19.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.820+0000 7f8d7ae00640 1 -- 172.21.15.90:0/1613157913 >> 172.21.15.90:0/1613157913 conn(0x7f8d740d3f80 msgr2=0x7f8d740d4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:19.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.820+0000 7f8d7ae00640 1 -- 172.21.15.90:0/1613157913 shutdown_connections 2024-04-25T01:16:19.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.820+0000 7f8d7ae00640 1 -- 172.21.15.90:0/1613157913 wait complete. 2024-04-25T01:16:19.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.822+0000 7f8d7ae00640 1 Processor -- start 2024-04-25T01:16:19.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.822+0000 7f8d7ae00640 1 -- start start 2024-04-25T01:16:19.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.823+0000 7f8d7ae00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8d740b2e30 0x7f8d7415c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:19.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.823+0000 7f8d7ae00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d740d2920 0x7f8d740cc520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:19.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.823+0000 7f8d7ae00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d7415e270 0x7f8d740b6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:19.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.823+0000 7f8d7ae00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8d7415e650 con 0x7f8d740d2920 2024-04-25T01:16:19.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.823+0000 7f8d7ae00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8d740cfdb0 con 0x7f8d7415e270 2024-04-25T01:16:19.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.823+0000 7f8d7ae00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8d740cd850 con 0x7f8d740b2e30 2024-04-25T01:16:19.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.823+0000 7f8d6b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d740d2920 0x7f8d740cc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:19.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.824+0000 7f8d6b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d740d2920 0x7f8d740cc520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38292/0 (socket says 172.21.15.90:38292) 2024-04-25T01:16:19.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.824+0000 7f8d6b400640 1 -- 172.21.15.90:0/2346387320 learned_addr learned my addr 172.21.15.90:0/2346387320 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:19.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.824+0000 7f8d73e00640 1 --2- 172.21.15.90:0/2346387320 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8d740b2e30 0x7f8d7415c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:19.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.824+0000 7f8d78e00640 1 --2- 172.21.15.90:0/2346387320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d7415e270 0x7f8d740b6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:19.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.824+0000 7f8d6b400640 1 -- 172.21.15.90:0/2346387320 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8d740b2e30 msgr2=0x7f8d7415c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:19.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.824+0000 7f8d6b400640 1 --2- 172.21.15.90:0/2346387320 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8d740b2e30 0x7f8d7415c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:19.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.824+0000 7f8d6b400640 1 -- 172.21.15.90:0/2346387320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d7415e270 msgr2=0x7f8d740b6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:19.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.824+0000 7f8d6b400640 1 --2- 172.21.15.90:0/2346387320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d7415e270 0x7f8d740b6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:19.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.824+0000 7f8d6b400640 1 -- 172.21.15.90:0/2346387320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d740cba60 con 0x7f8d740d2920 2024-04-25T01:16:19.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.824+0000 7f8d73e00640 1 --2- 172.21.15.90:0/2346387320 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8d740b2e30 0x7f8d7415c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:16:19.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.824+0000 7f8d6b400640 1 --2- 172.21.15.90:0/2346387320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d740d2920 0x7f8d740cc520 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c000c00 tx=0x7f8d5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:19.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.825+0000 7f8d70c00640 1 -- 172.21.15.90:0/2346387320 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c037040 con 0x7f8d740d2920 2024-04-25T01:16:19.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.825+0000 7f8d70c00640 1 -- 172.21.15.90:0/2346387320 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d5c042070 con 0x7f8d740d2920 2024-04-25T01:16:19.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.825+0000 7f8d70c00640 1 -- 172.21.15.90:0/2346387320 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c030bd0 con 0x7f8d740d2920 2024-04-25T01:16:19.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.825+0000 7f8d7ae00640 1 -- 172.21.15.90:0/2346387320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d740cdaa0 con 0x7f8d740d2920 2024-04-25T01:16:19.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.825+0000 7f8d7ae00640 1 -- 172.21.15.90:0/2346387320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d740cbd20 con 0x7f8d740d2920 2024-04-25T01:16:19.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.826+0000 7f8d7ae00640 1 -- 172.21.15.90:0/2346387320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d74068180 con 0x7f8d740d2920 2024-04-25T01:16:19.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.827+0000 7f8d70c00640 1 -- 172.21.15.90:0/2346387320 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8d5c047460 con 0x7f8d740d2920 2024-04-25T01:16:19.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.828+0000 7f8d70c00640 1 --2- 172.21.15.90:0/2346387320 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f8d5006be30 0x7f8d5006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:19.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.828+0000 7f8d70c00640 1 -- 172.21.15.90:0/2346387320 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f8d5c0ab710 con 0x7f8d740d2920 2024-04-25T01:16:19.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.828+0000 7f8d73e00640 1 --2- 172.21.15.90:0/2346387320 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f8d5006be30 0x7f8d5006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:19.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.829+0000 7f8d73e00640 1 --2- 172.21.15.90:0/2346387320 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f8d5006be30 0x7f8d5006e2f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8d640012e0 tx=0x7f8d64009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:19.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:19.833+0000 7f8d70c00640 1 -- 172.21.15.90:0/2346387320 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8d5c07a370 con 0x7f8d740d2920 2024-04-25T01:16:20.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.040+0000 7f8d7ae00640 1 -- 172.21.15.90:0/2346387320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f8d740d15b0 con 0x7f8d740d2920 2024-04-25T01:16:20.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.559+0000 7f8d70c00640 1 -- 172.21.15.90:0/2346387320 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v19) ==== 138+0+0 (secure 0 0 0) 0x7f8d5c080020 con 0x7f8d740d2920 2024-04-25T01:16:20.545 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_metadata' created 2024-04-25T01:16:20.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.563+0000 7f8d7ae00640 1 -- 172.21.15.90:0/2346387320 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f8d5006be30 msgr2=0x7f8d5006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:20.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.563+0000 7f8d7ae00640 1 --2- 172.21.15.90:0/2346387320 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f8d5006be30 0x7f8d5006e2f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8d640012e0 tx=0x7f8d64009040 comp rx=0 tx=0).stop 2024-04-25T01:16:20.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.563+0000 7f8d7ae00640 1 -- 172.21.15.90:0/2346387320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d740d2920 msgr2=0x7f8d740cc520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:20.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.563+0000 7f8d7ae00640 1 --2- 172.21.15.90:0/2346387320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d740d2920 0x7f8d740cc520 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c000c00 tx=0x7f8d5c0046c0 comp rx=0 tx=0).stop 2024-04-25T01:16:20.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.564+0000 7f8d7ae00640 1 -- 172.21.15.90:0/2346387320 shutdown_connections 2024-04-25T01:16:20.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.564+0000 7f8d7ae00640 1 --2- 172.21.15.90:0/2346387320 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f8d5006be30 0x7f8d5006e2f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:20.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.564+0000 7f8d7ae00640 1 --2- 172.21.15.90:0/2346387320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d7415e270 0x7f8d740b6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:20.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.564+0000 7f8d7ae00640 1 --2- 172.21.15.90:0/2346387320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d740d2920 0x7f8d740cc520 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:20.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.564+0000 7f8d7ae00640 1 --2- 172.21.15.90:0/2346387320 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8d740b2e30 0x7f8d7415c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:20.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.564+0000 7f8d7ae00640 1 -- 172.21.15.90:0/2346387320 >> 172.21.15.90:0/2346387320 conn(0x7f8d740d3f80 msgr2=0x7f8d7406c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:20.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.564+0000 7f8d7ae00640 1 -- 172.21.15.90:0/2346387320 shutdown_connections 2024-04-25T01:16:20.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.564+0000 7f8d7ae00640 1 -- 172.21.15.90:0/2346387320 wait complete. 2024-04-25T01:16:20.575 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-25T01:16:20.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.777+0000 7f8a1b200640 1 Processor -- start 2024-04-25T01:16:20.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.777+0000 7f8a1b200640 1 -- start start 2024-04-25T01:16:20.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.778+0000 7f8a1b200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8a1c0b4a20 0x7f8a1c15f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:20.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.778+0000 7f8a1b200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8a1c06d700 0x7f8a1c15fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:20.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.778+0000 7f8a1b200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a1c160580 con 0x7f8a1c0aee00 2024-04-25T01:16:20.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.778+0000 7f8a1b200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f8a1c06c9d0 con 0x7f8a1c0b4a20 2024-04-25T01:16:20.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.778+0000 7f8a1b200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f8a1c06cc20 con 0x7f8a1c06d700 2024-04-25T01:16:20.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.779+0000 7f8a19e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f8a1c0aee00 0x7f8a1c0b4310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41656/0 (socket says 172.21.15.90:41656) 2024-04-25T01:16:20.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.779+0000 7f8a19e00640 1 -- 172.21.15.90:0/468379059 learned_addr learned my addr 172.21.15.90:0/468379059 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:20.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.779+0000 7f8a13e00640 1 --2- 172.21.15.90:0/468379059 >> v2:172.21.15.90:3300/0 conn(0x7f8a1c0b4a20 0x7f8a1c15f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:20.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.779+0000 7f8a19400640 1 -- 172.21.15.90:0/468379059 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3203148713 0 0) 0x7f8a1c160580 con 0x7f8a1c0aee00 2024-04-25T01:16:20.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.779+0000 7f8a19400640 1 -- 172.21.15.90:0/468379059 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a00003520 con 0x7f8a1c0aee00 2024-04-25T01:16:20.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.779+0000 7f8a1a800640 1 --2- 172.21.15.90:0/468379059 >> v2:172.21.15.103:3300/0 conn(0x7f8a1c06d700 0x7f8a1c15fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:20.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.779+0000 7f8a13e00640 1 -- 172.21.15.90:0/468379059 >> v2:172.21.15.103:3300/0 conn(0x7f8a1c06d700 msgr2=0x7f8a1c15fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:20.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.779+0000 7f8a13e00640 1 --2- 172.21.15.90:0/468379059 >> v2:172.21.15.103:3300/0 conn(0x7f8a1c06d700 0x7f8a1c15fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:20.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.779+0000 7f8a13e00640 1 -- 172.21.15.90:0/468379059 >> v1:172.21.15.90:6789/0 conn(0x7f8a1c0aee00 legacy=0x7f8a1c0b4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:20.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.780+0000 7f8a13e00640 1 -- 172.21.15.90:0/468379059 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a1c06cf00 con 0x7f8a1c0b4a20 2024-04-25T01:16:20.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.781+0000 7f8a13e00640 1 --2- 172.21.15.90:0/468379059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a1c0b4a20 0x7f8a1c15f810 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c06e5c0 tx=0x7f8a0402a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=addeb939bb2cb6a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:20.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.781+0000 7f8a19400640 1 -- 172.21.15.90:0/468379059 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a04037070 con 0x7f8a1c0b4a20 2024-04-25T01:16:20.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.781+0000 7f8a19400640 1 -- 172.21.15.90:0/468379059 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a0402add0 con 0x7f8a1c0b4a20 2024-04-25T01:16:20.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.781+0000 7f8a19400640 1 -- 172.21.15.90:0/468379059 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a04032660 con 0x7f8a1c0b4a20 2024-04-25T01:16:20.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.781+0000 7f8a1b200640 1 -- 172.21.15.90:0/468379059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a1c0b4a20 msgr2=0x7f8a1c15f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:20.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.781+0000 7f8a1b200640 1 --2- 172.21.15.90:0/468379059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a1c0b4a20 0x7f8a1c15f810 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c06e5c0 tx=0x7f8a0402a080 comp rx=0 tx=0).stop 2024-04-25T01:16:20.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.781+0000 7f8a1b200640 1 -- 172.21.15.90:0/468379059 shutdown_connections 2024-04-25T01:16:20.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.781+0000 7f8a1b200640 1 --2- 172.21.15.90:0/468379059 >> v2:172.21.15.103:3300/0 conn(0x7f8a1c06d700 0x7f8a1c15fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:20.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.781+0000 7f8a1b200640 1 --2- 172.21.15.90:0/468379059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a1c0b4a20 0x7f8a1c15f810 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:20.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.781+0000 7f8a1b200640 1 -- 172.21.15.90:0/468379059 >> 172.21.15.90:0/468379059 conn(0x7f8a1c06b120 msgr2=0x7f8a1c06b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:20.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.782+0000 7f8a1b200640 1 -- 172.21.15.90:0/468379059 shutdown_connections 2024-04-25T01:16:20.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.782+0000 7f8a1b200640 1 -- 172.21.15.90:0/468379059 wait complete. 2024-04-25T01:16:20.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.783+0000 7f8a1b200640 1 Processor -- start 2024-04-25T01:16:20.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.783+0000 7f8a1b200640 1 -- start start 2024-04-25T01:16:20.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.784+0000 7f8a1b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a1c06d700 0x7f8a1c142df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:20.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.784+0000 7f8a1b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a1c0aee00 0x7f8a1c1545e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:20.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.784+0000 7f8a1b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a1c0b4a20 0x7f8a1c1401c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:20.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.784+0000 7f8a1b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8a1c0af1e0 con 0x7f8a1c0b4a20 2024-04-25T01:16:20.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.784+0000 7f8a1b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8a1c160290 con 0x7f8a1c0aee00 2024-04-25T01:16:20.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.784+0000 7f8a1b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8a1c15d370 con 0x7f8a1c06d700 2024-04-25T01:16:20.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.784+0000 7f8a1a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a1c0b4a20 0x7f8a1c1401c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:20.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.784+0000 7f8a1a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a1c0b4a20 0x7f8a1c1401c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38316/0 (socket says 172.21.15.90:38316) 2024-04-25T01:16:20.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.784+0000 7f8a1a800640 1 -- 172.21.15.90:0/2210826168 learned_addr learned my addr 172.21.15.90:0/2210826168 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:20.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.785+0000 7f8a19e00640 1 --2- 172.21.15.90:0/2210826168 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a1c06d700 0x7f8a1c142df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:20.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.785+0000 7f8a1a800640 1 -- 172.21.15.90:0/2210826168 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a1c06d700 msgr2=0x7f8a1c142df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:20.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.785+0000 7f8a13e00640 1 --2- 172.21.15.90:0/2210826168 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a1c0aee00 0x7f8a1c1545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:20.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.785+0000 7f8a1a800640 1 --2- 172.21.15.90:0/2210826168 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a1c06d700 0x7f8a1c142df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:20.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.785+0000 7f8a1a800640 1 -- 172.21.15.90:0/2210826168 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a1c0aee00 msgr2=0x7f8a1c1545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:20.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.785+0000 7f8a1a800640 1 --2- 172.21.15.90:0/2210826168 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a1c0aee00 0x7f8a1c1545e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:20.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.785+0000 7f8a1a800640 1 -- 172.21.15.90:0/2210826168 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a1c1574c0 con 0x7f8a1c0b4a20 2024-04-25T01:16:20.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.785+0000 7f8a1a800640 1 --2- 172.21.15.90:0/2210826168 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a1c0b4a20 0x7f8a1c1401c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8a14007e70 tx=0x7f8a1400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:20.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.785+0000 7f8a12000640 1 -- 172.21.15.90:0/2210826168 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a14014620 con 0x7f8a1c0b4a20 2024-04-25T01:16:20.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.786+0000 7f8a12000640 1 -- 172.21.15.90:0/2210826168 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a14019070 con 0x7f8a1c0b4a20 2024-04-25T01:16:20.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.786+0000 7f8a12000640 1 -- 172.21.15.90:0/2210826168 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a14014a20 con 0x7f8a1c0b4a20 2024-04-25T01:16:20.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.786+0000 7f8a1b200640 1 -- 172.21.15.90:0/2210826168 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a1c1577a0 con 0x7f8a1c0b4a20 2024-04-25T01:16:20.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.786+0000 7f8a1b200640 1 -- 172.21.15.90:0/2210826168 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a1c157dd0 con 0x7f8a1c0b4a20 2024-04-25T01:16:20.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.787+0000 7f8a1b200640 1 -- 172.21.15.90:0/2210826168 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a1c0753c0 con 0x7f8a1c0b4a20 2024-04-25T01:16:20.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.788+0000 7f8a12000640 1 -- 172.21.15.90:0/2210826168 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8a14014bc0 con 0x7f8a1c0b4a20 2024-04-25T01:16:20.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.789+0000 7f8a12000640 1 --2- 172.21.15.90:0/2210826168 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f89f806be30 0x7f89f806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:20.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.789+0000 7f8a12000640 1 -- 172.21.15.90:0/2210826168 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5227+0+0 (secure 0 0 0) 0x7f8a1408b5e0 con 0x7f8a1c0b4a20 2024-04-25T01:16:20.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.789+0000 7f8a19e00640 1 --2- 172.21.15.90:0/2210826168 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f89f806be30 0x7f89f806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:20.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.790+0000 7f8a19e00640 1 --2- 172.21.15.90:0/2210826168 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f89f806be30 0x7f89f806e2f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c000f60 tx=0x7f8a0c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:20.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:20.794+0000 7f8a12000640 1 -- 172.21.15.90:0/2210826168 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8a1405a160 con 0x7f8a1c0b4a20 2024-04-25T01:16:21.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.024+0000 7f8a1b200640 1 -- 172.21.15.90:0/2210826168 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f8a1c06de80 con 0x7f8a1c0b4a20 2024-04-25T01:16:21.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.576+0000 7f8a12000640 1 -- 172.21.15.90:0/2210826168 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v20) ==== 171+0+0 (secure 0 0 0) 0x7f8a14059a60 con 0x7f8a1c0b4a20 2024-04-25T01:16:21.562 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data' created 2024-04-25T01:16:21.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.580+0000 7f8a1b200640 1 -- 172.21.15.90:0/2210826168 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f89f806be30 msgr2=0x7f89f806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:21.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.580+0000 7f8a1b200640 1 --2- 172.21.15.90:0/2210826168 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f89f806be30 0x7f89f806e2f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c000f60 tx=0x7f8a0c009040 comp rx=0 tx=0).stop 2024-04-25T01:16:21.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.580+0000 7f8a1b200640 1 -- 172.21.15.90:0/2210826168 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a1c0b4a20 msgr2=0x7f8a1c1401c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:21.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.580+0000 7f8a1b200640 1 --2- 172.21.15.90:0/2210826168 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a1c0b4a20 0x7f8a1c1401c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8a14007e70 tx=0x7f8a1400e4b0 comp rx=0 tx=0).stop 2024-04-25T01:16:21.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.581+0000 7f8a1b200640 1 -- 172.21.15.90:0/2210826168 shutdown_connections 2024-04-25T01:16:21.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.581+0000 7f8a1b200640 1 --2- 172.21.15.90:0/2210826168 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f89f806be30 0x7f89f806e2f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:21.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.581+0000 7f8a1b200640 1 --2- 172.21.15.90:0/2210826168 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a1c0b4a20 0x7f8a1c1401c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:21.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.581+0000 7f8a1b200640 1 --2- 172.21.15.90:0/2210826168 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a1c0aee00 0x7f8a1c1545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:21.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.581+0000 7f8a1b200640 1 --2- 172.21.15.90:0/2210826168 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a1c06d700 0x7f8a1c142df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:21.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.581+0000 7f8a1b200640 1 -- 172.21.15.90:0/2210826168 >> 172.21.15.90:0/2210826168 conn(0x7f8a1c06b120 msgr2=0x7f8a1c0747e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:21.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.581+0000 7f8a1b200640 1 -- 172.21.15.90:0/2210826168 shutdown_connections 2024-04-25T01:16:21.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.581+0000 7f8a1b200640 1 -- 172.21.15.90:0/2210826168 wait complete. 2024-04-25T01:16:21.597 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T01:16:21.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.792+0000 7f4023e00640 1 Processor -- start 2024-04-25T01:16:21.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.792+0000 7f4023e00640 1 -- start start 2024-04-25T01:16:21.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.794+0000 7f4023e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f40240d29c0 0x7f4024168bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:21.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.794+0000 7f4023e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f402415e4c0 con 0x7f40240d49b0 2024-04-25T01:16:21.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.794+0000 7f4023e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40240cebc0 con 0x7f402415e0e0 2024-04-25T01:16:21.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.794+0000 7f4023e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f40240cf4f0 con 0x7f40240d29c0 2024-04-25T01:16:21.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.794+0000 7f4023400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f40240d29c0 0x7f4024168bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:21.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.794+0000 7f4023400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f40240d29c0 0x7f4024168bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38332/0 (socket says 172.21.15.90:38332) 2024-04-25T01:16:21.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.794+0000 7f4023400640 1 -- 172.21.15.90:0/2776277388 learned_addr learned my addr 172.21.15.90:0/2776277388 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:21.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.795+0000 7f4023400640 1 -- 172.21.15.90:0/2776277388 >> v1:172.21.15.103:6789/0 conn(0x7f402415e0e0 legacy=0x7f40240d0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:21.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.795+0000 7f4023400640 1 -- 172.21.15.90:0/2776277388 >> v1:172.21.15.90:6789/0 conn(0x7f40240d49b0 legacy=0x7f40240d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:21.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.795+0000 7f4023400640 1 -- 172.21.15.90:0/2776277388 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40240ceea0 con 0x7f40240d29c0 2024-04-25T01:16:21.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.795+0000 7f4021600640 1 -- 172.21.15.90:0/2776277388 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 816776704 0 0) 0x7f402415e4c0 con 0x7f40240d49b0 2024-04-25T01:16:21.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.796+0000 7f4023400640 1 --2- 172.21.15.90:0/2776277388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40240d29c0 0x7f4024168bf0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f401c009e50 tx=0x7f401c02a760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1c5e5b787c400ac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:21.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.796+0000 7f4021600640 1 -- 172.21.15.90:0/2776277388 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f401c036070 con 0x7f40240d29c0 2024-04-25T01:16:21.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.796+0000 7f4021600640 1 -- 172.21.15.90:0/2776277388 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f401c031600 con 0x7f40240d29c0 2024-04-25T01:16:21.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.796+0000 7f4021600640 1 -- 172.21.15.90:0/2776277388 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f401c0319d0 con 0x7f40240d29c0 2024-04-25T01:16:21.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.797+0000 7f4023e00640 1 -- 172.21.15.90:0/2776277388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40240d29c0 msgr2=0x7f4024168bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:21.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.797+0000 7f4023e00640 1 --2- 172.21.15.90:0/2776277388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40240d29c0 0x7f4024168bf0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f401c009e50 tx=0x7f401c02a760 comp rx=0 tx=0).stop 2024-04-25T01:16:21.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.797+0000 7f4023e00640 1 -- 172.21.15.90:0/2776277388 shutdown_connections 2024-04-25T01:16:21.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.797+0000 7f4023e00640 1 --2- 172.21.15.90:0/2776277388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40240d29c0 0x7f4024168bf0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:21.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.797+0000 7f4023e00640 1 -- 172.21.15.90:0/2776277388 >> 172.21.15.90:0/2776277388 conn(0x7f40240d3cd0 msgr2=0x7f40240d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:21.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.797+0000 7f4023e00640 1 -- 172.21.15.90:0/2776277388 shutdown_connections 2024-04-25T01:16:21.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.797+0000 7f4023e00640 1 -- 172.21.15.90:0/2776277388 wait complete. 2024-04-25T01:16:21.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.798+0000 7f4023e00640 1 Processor -- start 2024-04-25T01:16:21.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.799+0000 7f4023e00640 1 -- start start 2024-04-25T01:16:21.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.799+0000 7f4023e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40240d29c0 0x7f40241574c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:21.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.800+0000 7f4023e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40240d49b0 0x7f40240cd520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:21.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.800+0000 7f4023e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f402415e0e0 0x7f40240cda60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:21.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.800+0000 7f4023e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f40240ce7e0 con 0x7f40240d29c0 2024-04-25T01:16:21.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.800+0000 7f4023e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f40240d1420 con 0x7f402415e0e0 2024-04-25T01:16:21.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.800+0000 7f4023e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f40240ce960 con 0x7f40240d49b0 2024-04-25T01:16:21.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.800+0000 7f4022a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40240d29c0 0x7f40241574c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:21.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.800+0000 7f4022a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40240d29c0 0x7f40241574c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38340/0 (socket says 172.21.15.90:38340) 2024-04-25T01:16:21.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.800+0000 7f4022a00640 1 -- 172.21.15.90:0/467470038 learned_addr learned my addr 172.21.15.90:0/467470038 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:21.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.800+0000 7f4022a00640 1 -- 172.21.15.90:0/467470038 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40240d49b0 msgr2=0x7f40240cd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:21.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.800+0000 7f4023400640 1 --2- 172.21.15.90:0/467470038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f402415e0e0 0x7f40240cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:21.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.800+0000 7f4022000640 1 --2- 172.21.15.90:0/467470038 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40240d49b0 0x7f40240cd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:21.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.800+0000 7f4022a00640 1 --2- 172.21.15.90:0/467470038 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40240d49b0 0x7f40240cd520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:21.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.800+0000 7f4022a00640 1 -- 172.21.15.90:0/467470038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f402415e0e0 msgr2=0x7f40240cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:21.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.800+0000 7f4022a00640 1 --2- 172.21.15.90:0/467470038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f402415e0e0 0x7f40240cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:21.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.800+0000 7f4022a00640 1 -- 172.21.15.90:0/467470038 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40240b7b50 con 0x7f40240d29c0 2024-04-25T01:16:21.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.800+0000 7f4023400640 1 --2- 172.21.15.90:0/467470038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f402415e0e0 0x7f40240cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:21.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.801+0000 7f4022a00640 1 --2- 172.21.15.90:0/467470038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40240d29c0 0x7f40241574c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f401400c3f0 tx=0x7f401400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:21.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.801+0000 7f401aa00640 1 -- 172.21.15.90:0/467470038 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4014011790 con 0x7f40240d29c0 2024-04-25T01:16:21.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.801+0000 7f401aa00640 1 -- 172.21.15.90:0/467470038 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4014017070 con 0x7f40240d29c0 2024-04-25T01:16:21.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.801+0000 7f401aa00640 1 -- 172.21.15.90:0/467470038 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4014011b90 con 0x7f40240d29c0 2024-04-25T01:16:21.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.801+0000 7f4023e00640 1 -- 172.21.15.90:0/467470038 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40240b7e30 con 0x7f40240d29c0 2024-04-25T01:16:21.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.801+0000 7f4023e00640 1 -- 172.21.15.90:0/467470038 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40240b83f0 con 0x7f40240d29c0 2024-04-25T01:16:21.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.802+0000 7f4023e00640 1 -- 172.21.15.90:0/467470038 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4024068180 con 0x7f40240d29c0 2024-04-25T01:16:21.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.803+0000 7f401aa00640 1 -- 172.21.15.90:0/467470038 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f401401c460 con 0x7f40240d29c0 2024-04-25T01:16:21.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.804+0000 7f401aa00640 1 --2- 172.21.15.90:0/467470038 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f400006be30 0x7f400006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:21.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.804+0000 7f401aa00640 1 -- 172.21.15.90:0/467470038 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5609+0+0 (secure 0 0 0) 0x7f401408ab80 con 0x7f40240d29c0 2024-04-25T01:16:21.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.805+0000 7f4022000640 1 --2- 172.21.15.90:0/467470038 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f400006be30 0x7f400006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:21.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.805+0000 7f4022000640 1 --2- 172.21.15.90:0/467470038 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f400006be30 0x7f400006e2f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f402415b220 tx=0x7f400c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:21.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:21.809+0000 7f401aa00640 1 -- 172.21.15.90:0/467470038 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4024068180 con 0x7f40240d29c0 2024-04-25T01:16:22.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.056+0000 7f4023e00640 1 -- 172.21.15.90:0/467470038 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f4024075350 con 0x7f40240d29c0 2024-04-25T01:16:22.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.599+0000 7f401aa00640 1 -- 172.21.15.90:0/467470038 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7f4014059620 con 0x7f40240d29c0 2024-04-25T01:16:22.589 INFO:teuthology.orchestra.run.smithi090.stderr:new fs with metadata pool 2 and data pool 3 2024-04-25T01:16:22.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.609+0000 7f4023e00640 1 -- 172.21.15.90:0/467470038 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f400006be30 msgr2=0x7f400006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:22.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.609+0000 7f4023e00640 1 --2- 172.21.15.90:0/467470038 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f400006be30 0x7f400006e2f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f402415b220 tx=0x7f400c009040 comp rx=0 tx=0).stop 2024-04-25T01:16:22.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.609+0000 7f4023e00640 1 -- 172.21.15.90:0/467470038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40240d29c0 msgr2=0x7f40241574c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:22.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.609+0000 7f4023e00640 1 --2- 172.21.15.90:0/467470038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40240d29c0 0x7f40241574c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f401400c3f0 tx=0x7f401400c8b0 comp rx=0 tx=0).stop 2024-04-25T01:16:22.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.614+0000 7f4023e00640 1 -- 172.21.15.90:0/467470038 shutdown_connections 2024-04-25T01:16:22.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.614+0000 7f4023e00640 1 --2- 172.21.15.90:0/467470038 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f400006be30 0x7f400006e2f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:22.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.614+0000 7f4023e00640 1 --2- 172.21.15.90:0/467470038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f402415e0e0 0x7f40240cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:22.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.614+0000 7f4023e00640 1 --2- 172.21.15.90:0/467470038 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40240d49b0 0x7f40240cd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:22.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.614+0000 7f4023e00640 1 --2- 172.21.15.90:0/467470038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40240d29c0 0x7f40241574c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:22.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.614+0000 7f4023e00640 1 -- 172.21.15.90:0/467470038 >> 172.21.15.90:0/467470038 conn(0x7f40240d3cd0 msgr2=0x7f402415d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:22.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.616+0000 7f4023e00640 1 -- 172.21.15.90:0/467470038 shutdown_connections 2024-04-25T01:16:22.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.616+0000 7f4023e00640 1 -- 172.21.15.90:0/467470038 wait complete. 2024-04-25T01:16:22.637 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-25T01:16:22.637 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-25T01:16:22.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.828+0000 7f5896400640 1 Processor -- start 2024-04-25T01:16:22.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.828+0000 7f5896400640 1 -- start start 2024-04-25T01:16:22.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.830+0000 7f5896400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f58900cfd10 0x7f58900d2880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:22.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.830+0000 7f5896400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f58900cf110 0x7f58900d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:22.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.830+0000 7f5896400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f589015fbb0 con 0x7f589015c4e0 2024-04-25T01:16:22.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.830+0000 7f5896400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f589015fe90 con 0x7f58900cf110 2024-04-25T01:16:22.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.830+0000 7f5896400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5890160170 con 0x7f58900cfd10 2024-04-25T01:16:22.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.830+0000 7f588fe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f589015c4e0 0x7f589015c940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41676/0 (socket says 172.21.15.90:41676) 2024-04-25T01:16:22.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.830+0000 7f588fe00640 1 -- 172.21.15.90:0/990891542 learned_addr learned my addr 172.21.15.90:0/990891542 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:22.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.831+0000 7f588ea00640 1 --2- 172.21.15.90:0/990891542 >> v2:172.21.15.103:3300/0 conn(0x7f58900cf110 0x7f58900d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:22.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.831+0000 7f588e000640 1 -- 172.21.15.90:0/990891542 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2550503778 0 0) 0x7f589015fbb0 con 0x7f589015c4e0 2024-04-25T01:16:22.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.831+0000 7f588e000640 1 -- 172.21.15.90:0/990891542 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5874003520 con 0x7f589015c4e0 2024-04-25T01:16:22.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.831+0000 7f588f400640 1 --2- 172.21.15.90:0/990891542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f58900cfd10 0x7f58900d2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:22.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.831+0000 7f588e000640 1 -- 172.21.15.90:0/990891542 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4020973722 0 0) 0x7f5874003520 con 0x7f589015c4e0 2024-04-25T01:16:22.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.831+0000 7f588e000640 1 -- 172.21.15.90:0/990891542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f58900cfd10 msgr2=0x7f58900d2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:22.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.831+0000 7f588e000640 1 --2- 172.21.15.90:0/990891542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f58900cfd10 0x7f58900d2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:22.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.831+0000 7f588e000640 1 -- 172.21.15.90:0/990891542 >> v2:172.21.15.103:3300/0 conn(0x7f58900cf110 msgr2=0x7f58900d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:22.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.831+0000 7f588e000640 1 --2- 172.21.15.90:0/990891542 >> v2:172.21.15.103:3300/0 conn(0x7f58900cf110 0x7f58900d00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:22.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.831+0000 7f588e000640 1 -- 172.21.15.90:0/990891542 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5890160450 con 0x7f589015c4e0 2024-04-25T01:16:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.831+0000 7f588e000640 1 -- 172.21.15.90:0/990891542 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f58880019f0 con 0x7f589015c4e0 2024-04-25T01:16:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.831+0000 7f588e000640 1 -- 172.21.15.90:0/990891542 >> v1:172.21.15.90:6789/0 conn(0x7f589015c4e0 legacy=0x7f589015c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.831+0000 7f588e000640 1 --2- 172.21.15.90:0/990891542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f58900cfd10 0x7f58740039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.831+0000 7f588e000640 1 --2- 172.21.15.90:0/990891542 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5874003f30 0x7f5874004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.831+0000 7f588e000640 1 --2- 172.21.15.90:0/990891542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5874004d40 0x7f58740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.831+0000 7f588e000640 1 -- 172.21.15.90:0/990891542 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f589015fe90 con 0x7f5874003f30 2024-04-25T01:16:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.831+0000 7f588e000640 1 -- 172.21.15.90:0/990891542 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5890160170 con 0x7f5874004d40 2024-04-25T01:16:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.831+0000 7f588e000640 1 -- 172.21.15.90:0/990891542 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f587400ba10 con 0x7f58900cfd10 2024-04-25T01:16:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.831+0000 7f588ea00640 1 --2- 172.21.15.90:0/990891542 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5874003f30 0x7f5874004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.832+0000 7f588ea00640 1 -- 172.21.15.90:0/990891542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f58900cfd10 msgr2=0x7f58740039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.832+0000 7f588ea00640 1 --2- 172.21.15.90:0/990891542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f58900cfd10 0x7f58740039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.832+0000 7f588ea00640 1 -- 172.21.15.90:0/990891542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5874004d40 msgr2=0x7f58740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.832+0000 7f588fe00640 1 --2- 172.21.15.90:0/990891542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5874004d40 0x7f58740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.832+0000 7f588ea00640 1 --2- 172.21.15.90:0/990891542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5874004d40 0x7f58740051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:22.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.832+0000 7f588ea00640 1 -- 172.21.15.90:0/990891542 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5874003520 con 0x7f5874003f30 2024-04-25T01:16:22.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.832+0000 7f588fe00640 1 --2- 172.21.15.90:0/990891542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5874004d40 0x7f58740051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:22.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.833+0000 7f588ea00640 1 --2- 172.21.15.90:0/990891542 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5874003f30 0x7f5874004800 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5878009ac0 tx=0x7f5874005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e8f4ad148b5881e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:22.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.833+0000 7f588e000640 1 -- 172.21.15.90:0/990891542 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5878036070 con 0x7f5874003f30 2024-04-25T01:16:22.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.833+0000 7f588e000640 1 -- 172.21.15.90:0/990891542 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5878031400 con 0x7f5874003f30 2024-04-25T01:16:22.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.833+0000 7f588e000640 1 -- 172.21.15.90:0/990891542 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5878009690 con 0x7f5874003f30 2024-04-25T01:16:22.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.833+0000 7f5896400640 1 -- 172.21.15.90:0/990891542 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5874003f30 msgr2=0x7f5874004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:22.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.833+0000 7f5896400640 1 --2- 172.21.15.90:0/990891542 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5874003f30 0x7f5874004800 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5878009ac0 tx=0x7f5874005830 comp rx=0 tx=0).stop 2024-04-25T01:16:22.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.834+0000 7f588fe00640 1 -- 172.21.15.90:0/990891542 reap_dead start 2024-04-25T01:16:22.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.834+0000 7f5896400640 1 -- 172.21.15.90:0/990891542 shutdown_connections 2024-04-25T01:16:22.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.834+0000 7f5896400640 1 -- 172.21.15.90:0/990891542 >> 172.21.15.90:0/990891542 conn(0x7f58900d1cf0 msgr2=0x7f58900d2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:22.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.834+0000 7f5896400640 1 -- 172.21.15.90:0/990891542 shutdown_connections 2024-04-25T01:16:22.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.834+0000 7f5896400640 1 -- 172.21.15.90:0/990891542 wait complete. 2024-04-25T01:16:22.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.835+0000 7f5896400640 1 Processor -- start 2024-04-25T01:16:22.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.836+0000 7f5896400640 1 -- start start 2024-04-25T01:16:22.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.836+0000 7f5896400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5874003f30 0x7f58900c78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:22.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.837+0000 7f5896400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58900b7200 0x7f58900b0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:22.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.837+0000 7f5896400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58900c4320 0x7f58900c47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:22.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.837+0000 7f5896400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f58900d2880 con 0x7f58900c4320 2024-04-25T01:16:22.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.837+0000 7f5896400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f58900cf530 con 0x7f58900b7200 2024-04-25T01:16:22.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.837+0000 7f5896400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f58900d2a00 con 0x7f5874003f30 2024-04-25T01:16:22.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.837+0000 7f588fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58900c4320 0x7f58900c47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:22.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.837+0000 7f588fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58900c4320 0x7f58900c47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38368/0 (socket says 172.21.15.90:38368) 2024-04-25T01:16:22.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.837+0000 7f588fe00640 1 -- 172.21.15.90:0/734509547 learned_addr learned my addr 172.21.15.90:0/734509547 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:22.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.837+0000 7f588f400640 1 --2- 172.21.15.90:0/734509547 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5874003f30 0x7f58900c78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:22.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.837+0000 7f588ea00640 1 --2- 172.21.15.90:0/734509547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58900b7200 0x7f58900b0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:22.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.837+0000 7f588fe00640 1 -- 172.21.15.90:0/734509547 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5874003f30 msgr2=0x7f58900c78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:22.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.837+0000 7f588fe00640 1 --2- 172.21.15.90:0/734509547 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5874003f30 0x7f58900c78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:22.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.837+0000 7f588fe00640 1 -- 172.21.15.90:0/734509547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58900b7200 msgr2=0x7f58900b0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:22.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.837+0000 7f588fe00640 1 --2- 172.21.15.90:0/734509547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58900b7200 0x7f58900b0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:22.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.837+0000 7f588fe00640 1 -- 172.21.15.90:0/734509547 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58900c4ce0 con 0x7f58900c4320 2024-04-25T01:16:22.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.837+0000 7f588f400640 1 --2- 172.21.15.90:0/734509547 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5874003f30 0x7f58900c78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:22.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.837+0000 7f588ea00640 1 --2- 172.21.15.90:0/734509547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58900b7200 0x7f58900b0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:22.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.838+0000 7f588fe00640 1 --2- 172.21.15.90:0/734509547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58900c4320 0x7f58900c47a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5888000f80 tx=0x7f588800df90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:22.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.838+0000 7f5887400640 1 -- 172.21.15.90:0/734509547 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588800bb10 con 0x7f58900c4320 2024-04-25T01:16:22.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.838+0000 7f5887400640 1 -- 172.21.15.90:0/734509547 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f588800bcb0 con 0x7f58900c4320 2024-04-25T01:16:22.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.838+0000 7f5896400640 1 -- 172.21.15.90:0/734509547 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58900c7e90 con 0x7f58900c4320 2024-04-25T01:16:22.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.838+0000 7f5887400640 1 -- 172.21.15.90:0/734509547 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58880072b0 con 0x7f58900c4320 2024-04-25T01:16:22.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.838+0000 7f5896400640 1 -- 172.21.15.90:0/734509547 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58900c8450 con 0x7f58900c4320 2024-04-25T01:16:22.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.839+0000 7f5896400640 1 -- 172.21.15.90:0/734509547 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5890068180 con 0x7f58900c4320 2024-04-25T01:16:22.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.840+0000 7f5887400640 1 -- 172.21.15.90:0/734509547 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f588801b020 con 0x7f58900c4320 2024-04-25T01:16:22.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.841+0000 7f5887400640 1 --2- 172.21.15.90:0/734509547 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f586c06be60 0x7f586c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:22.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.841+0000 7f5887400640 1 -- 172.21.15.90:0/734509547 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f588808b200 con 0x7f58900c4320 2024-04-25T01:16:22.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.841+0000 7f588f400640 1 --2- 172.21.15.90:0/734509547 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f586c06be60 0x7f586c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:22.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.842+0000 7f588f400640 1 --2- 172.21.15.90:0/734509547 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f586c06be60 0x7f586c06e320 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5880004770 tx=0x7f5880008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:22.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:22.845+0000 7f5887400640 1 -- 172.21.15.90:0/734509547 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f588800be60 con 0x7f58900c4320 2024-04-25T01:16:23.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.041+0000 7f5896400640 1 -- 172.21.15.90:0/734509547 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f58900ce4e0 con 0x7f58900c4320 2024-04-25T01:16:23.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.607+0000 7f5887400640 1 -- 172.21.15.90:0/734509547 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v22) ==== 157+0+0 (secure 0 0 0) 0x7f5888059a80 con 0x7f58900c4320 2024-04-25T01:16:23.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.612+0000 7f5896400640 1 -- 172.21.15.90:0/734509547 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f586c06be60 msgr2=0x7f586c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:23.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.612+0000 7f5896400640 1 --2- 172.21.15.90:0/734509547 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f586c06be60 0x7f586c06e320 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5880004770 tx=0x7f5880008040 comp rx=0 tx=0).stop 2024-04-25T01:16:23.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.612+0000 7f5896400640 1 -- 172.21.15.90:0/734509547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58900c4320 msgr2=0x7f58900c47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:23.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.612+0000 7f5896400640 1 --2- 172.21.15.90:0/734509547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58900c4320 0x7f58900c47a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5888000f80 tx=0x7f588800df90 comp rx=0 tx=0).stop 2024-04-25T01:16:23.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.612+0000 7f5896400640 1 -- 172.21.15.90:0/734509547 shutdown_connections 2024-04-25T01:16:23.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.612+0000 7f5896400640 1 --2- 172.21.15.90:0/734509547 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f586c06be60 0x7f586c06e320 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:23.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.612+0000 7f5896400640 1 --2- 172.21.15.90:0/734509547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58900c4320 0x7f58900c47a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:23.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.612+0000 7f5896400640 1 --2- 172.21.15.90:0/734509547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58900b7200 0x7f58900b0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:23.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.612+0000 7f5896400640 1 --2- 172.21.15.90:0/734509547 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5874003f30 0x7f58900c78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:23.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.612+0000 7f5896400640 1 -- 172.21.15.90:0/734509547 >> 172.21.15.90:0/734509547 conn(0x7f58900d1cf0 msgr2=0x7f58900cd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:23.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.612+0000 7f5896400640 1 -- 172.21.15.90:0/734509547 shutdown_connections 2024-04-25T01:16:23.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.612+0000 7f5896400640 1 -- 172.21.15.90:0/734509547 wait complete. 2024-04-25T01:16:23.619 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T01:16:23.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.843+0000 7fe521600640 1 Processor -- start 2024-04-25T01:16:23.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.844+0000 7fe521600640 1 -- start start 2024-04-25T01:16:23.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.845+0000 7fe521600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe51c0b9a90 0x7fe51c0d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:23.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.845+0000 7fe521600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe51c0d27b0 0x7fe51c0d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:23.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.845+0000 7fe521600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe51c0cea40 con 0x7fe51c0d23d0 2024-04-25T01:16:23.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.845+0000 7fe521600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fe51c0cece0 con 0x7fe51c0b9a90 2024-04-25T01:16:23.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.845+0000 7fe521600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe51c0cefc0 con 0x7fe51c0d27b0 2024-04-25T01:16:23.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.846+0000 7fe51a000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe51c0b9a90 0x7fe51c0d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:23.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.846+0000 7fe51a000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe51c0b9a90 0x7fe51c0d4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38372/0 (socket says 172.21.15.90:38372) 2024-04-25T01:16:23.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.846+0000 7fe51a000640 1 -- 172.21.15.90:0/832629190 learned_addr learned my addr 172.21.15.90:0/832629190 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:23.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.846+0000 7fe51aa00640 1 --1- 172.21.15.90:0/832629190 >> v1:172.21.15.90:6789/0 conn(0x7fe51c0d23d0 0x7fe51c0b9380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41684/0 (socket says 172.21.15.90:41684) 2024-04-25T01:16:23.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.846+0000 7fe51b400640 1 --2- 172.21.15.90:0/832629190 >> v2:172.21.15.103:3300/0 conn(0x7fe51c0d27b0 0x7fe51c0d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:23.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.846+0000 7fe51a000640 1 -- 172.21.15.90:0/832629190 >> v2:172.21.15.103:3300/0 conn(0x7fe51c0d27b0 msgr2=0x7fe51c0d0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:23.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.846+0000 7fe51a000640 1 --2- 172.21.15.90:0/832629190 >> v2:172.21.15.103:3300/0 conn(0x7fe51c0d27b0 0x7fe51c0d0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:23.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.846+0000 7fe51a000640 1 -- 172.21.15.90:0/832629190 >> v1:172.21.15.90:6789/0 conn(0x7fe51c0d23d0 legacy=0x7fe51c0b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:23.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.846+0000 7fe51a000640 1 -- 172.21.15.90:0/832629190 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe51c0cf2a0 con 0x7fe51c0b9a90 2024-04-25T01:16:23.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.846+0000 7fe519600640 1 -- 172.21.15.90:0/832629190 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 336558039 0 0) 0x7fe51c0cea40 con 0x7fe51c0d23d0 2024-04-25T01:16:23.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.848+0000 7fe51a000640 1 --2- 172.21.15.90:0/832629190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe51c0b9a90 0x7fe51c0d4920 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe51c0d4e60 tx=0x7fe50802a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0c67fdf0cd3cc29 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:23.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.848+0000 7fe519600640 1 -- 172.21.15.90:0/832629190 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe508037070 con 0x7fe51c0b9a90 2024-04-25T01:16:23.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.848+0000 7fe519600640 1 -- 172.21.15.90:0/832629190 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe50802ae50 con 0x7fe51c0b9a90 2024-04-25T01:16:23.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.848+0000 7fe519600640 1 -- 172.21.15.90:0/832629190 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe508032860 con 0x7fe51c0b9a90 2024-04-25T01:16:23.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.849+0000 7fe521600640 1 -- 172.21.15.90:0/832629190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe51c0b9a90 msgr2=0x7fe51c0d4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:23.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.849+0000 7fe521600640 1 --2- 172.21.15.90:0/832629190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe51c0b9a90 0x7fe51c0d4920 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe51c0d4e60 tx=0x7fe50802a300 comp rx=0 tx=0).stop 2024-04-25T01:16:23.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.849+0000 7fe521600640 1 -- 172.21.15.90:0/832629190 shutdown_connections 2024-04-25T01:16:23.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.849+0000 7fe521600640 1 --2- 172.21.15.90:0/832629190 >> v2:172.21.15.103:3300/0 conn(0x7fe51c0d27b0 0x7fe51c0d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:23.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.849+0000 7fe521600640 1 --2- 172.21.15.90:0/832629190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe51c0b9a90 0x7fe51c0d4920 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:23.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.849+0000 7fe521600640 1 -- 172.21.15.90:0/832629190 >> 172.21.15.90:0/832629190 conn(0x7fe51c0d3e30 msgr2=0x7fe51c0d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:23.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.849+0000 7fe521600640 1 -- 172.21.15.90:0/832629190 shutdown_connections 2024-04-25T01:16:23.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.849+0000 7fe521600640 1 -- 172.21.15.90:0/832629190 wait complete. 2024-04-25T01:16:23.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.851+0000 7fe521600640 1 Processor -- start 2024-04-25T01:16:23.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.851+0000 7fe521600640 1 -- start start 2024-04-25T01:16:23.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.852+0000 7fe521600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe51c0b9a90 0x7fe51c0a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:23.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.852+0000 7fe521600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe51c0d23d0 0x7fe51c06bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:23.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.852+0000 7fe521600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe51c0d27b0 0x7fe51c09fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:23.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.852+0000 7fe521600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c0cfd80 con 0x7fe51c0d27b0 2024-04-25T01:16:23.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.852+0000 7fe521600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c0cf860 con 0x7fe51c0d23d0 2024-04-25T01:16:23.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.852+0000 7fe521600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe51c0b6c80 con 0x7fe51c0b9a90 2024-04-25T01:16:23.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.852+0000 7fe51b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe51c0d27b0 0x7fe51c09fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:23.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.853+0000 7fe51b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe51c0d27b0 0x7fe51c09fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38378/0 (socket says 172.21.15.90:38378) 2024-04-25T01:16:23.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.853+0000 7fe51b400640 1 -- 172.21.15.90:0/1767134918 learned_addr learned my addr 172.21.15.90:0/1767134918 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:23.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.853+0000 7fe51a000640 1 --2- 172.21.15.90:0/1767134918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe51c0d23d0 0x7fe51c06bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:23.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.853+0000 7fe51aa00640 1 --2- 172.21.15.90:0/1767134918 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe51c0b9a90 0x7fe51c0a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:23.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.853+0000 7fe51b400640 1 -- 172.21.15.90:0/1767134918 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe51c0b9a90 msgr2=0x7fe51c0a11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:23.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.853+0000 7fe51b400640 1 --2- 172.21.15.90:0/1767134918 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe51c0b9a90 0x7fe51c0a11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:23.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.853+0000 7fe51b400640 1 -- 172.21.15.90:0/1767134918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe51c0d23d0 msgr2=0x7fe51c06bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:23.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.853+0000 7fe51b400640 1 --2- 172.21.15.90:0/1767134918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe51c0d23d0 0x7fe51c06bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:23.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.853+0000 7fe51b400640 1 -- 172.21.15.90:0/1767134918 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe51c0a1a10 con 0x7fe51c0d27b0 2024-04-25T01:16:23.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.853+0000 7fe51b400640 1 --2- 172.21.15.90:0/1767134918 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe51c0d27b0 0x7fe51c09fb30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe51400d5d0 tx=0x7fe51400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:23.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.853+0000 7fe512a00640 1 -- 172.21.15.90:0/1767134918 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5140125f0 con 0x7fe51c0d27b0 2024-04-25T01:16:23.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.853+0000 7fe512a00640 1 -- 172.21.15.90:0/1767134918 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe514019070 con 0x7fe51c0d27b0 2024-04-25T01:16:23.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.853+0000 7fe512a00640 1 -- 172.21.15.90:0/1767134918 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5140129f0 con 0x7fe51c0d27b0 2024-04-25T01:16:23.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.853+0000 7fe521600640 1 -- 172.21.15.90:0/1767134918 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe51c1f4ea0 con 0x7fe51c0d27b0 2024-04-25T01:16:23.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.854+0000 7fe521600640 1 -- 172.21.15.90:0/1767134918 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe51c1f5740 con 0x7fe51c0d27b0 2024-04-25T01:16:23.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.855+0000 7fe521600640 1 -- 172.21.15.90:0/1767134918 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe51c1f5070 con 0x7fe51c0d27b0 2024-04-25T01:16:23.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.856+0000 7fe512a00640 1 -- 172.21.15.90:0/1767134918 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe514012b90 con 0x7fe51c0d27b0 2024-04-25T01:16:23.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.857+0000 7fe512a00640 1 --2- 172.21.15.90:0/1767134918 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe4fc06be30 0x7fe4fc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:23.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.857+0000 7fe512a00640 1 -- 172.21.15.90:0/1767134918 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 6007+0+0 (secure 0 0 0) 0x7fe51408b360 con 0x7fe51c0d27b0 2024-04-25T01:16:23.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.857+0000 7fe51aa00640 1 --2- 172.21.15.90:0/1767134918 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe4fc06be30 0x7fe4fc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:23.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.857+0000 7fe51aa00640 1 --2- 172.21.15.90:0/1767134918 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe4fc06be30 0x7fe4fc06e2f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe50c0012e0 tx=0x7fe50c007ef0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:23.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:23.861+0000 7fe512a00640 1 -- 172.21.15.90:0/1767134918 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe514059bf0 con 0x7fe51c0d27b0 2024-04-25T01:16:24.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:24.048+0000 7fe521600640 1 -- 172.21.15.90:0/1767134918 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fe51c068fb0 con 0x7fe51c0d27b0 2024-04-25T01:16:25.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.616+0000 7fe512a00640 1 -- 172.21.15.90:0/1767134918 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v24) ==== 228+0+0 (secure 0 0 0) 0x7fe5140594f0 con 0x7fe51c0d27b0 2024-04-25T01:16:25.603 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data_ec' created 2024-04-25T01:16:25.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.620+0000 7fe521600640 1 -- 172.21.15.90:0/1767134918 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe4fc06be30 msgr2=0x7fe4fc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:25.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.620+0000 7fe521600640 1 --2- 172.21.15.90:0/1767134918 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe4fc06be30 0x7fe4fc06e2f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe50c0012e0 tx=0x7fe50c007ef0 comp rx=0 tx=0).stop 2024-04-25T01:16:25.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.620+0000 7fe521600640 1 -- 172.21.15.90:0/1767134918 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe51c0d27b0 msgr2=0x7fe51c09fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:25.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.620+0000 7fe521600640 1 --2- 172.21.15.90:0/1767134918 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe51c0d27b0 0x7fe51c09fb30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe51400d5d0 tx=0x7fe51400da90 comp rx=0 tx=0).stop 2024-04-25T01:16:25.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.620+0000 7fe521600640 1 -- 172.21.15.90:0/1767134918 shutdown_connections 2024-04-25T01:16:25.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.620+0000 7fe521600640 1 --2- 172.21.15.90:0/1767134918 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe4fc06be30 0x7fe4fc06e2f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:25.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.620+0000 7fe521600640 1 --2- 172.21.15.90:0/1767134918 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe51c0d27b0 0x7fe51c09fb30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:25.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.620+0000 7fe521600640 1 --2- 172.21.15.90:0/1767134918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe51c0d23d0 0x7fe51c06bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:25.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.620+0000 7fe521600640 1 --2- 172.21.15.90:0/1767134918 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe51c0b9a90 0x7fe51c0a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:25.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.620+0000 7fe521600640 1 -- 172.21.15.90:0/1767134918 >> 172.21.15.90:0/1767134918 conn(0x7fe51c0d3e30 msgr2=0x7fe51c0b8610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:25.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.621+0000 7fe521600640 1 -- 172.21.15.90:0/1767134918 shutdown_connections 2024-04-25T01:16:25.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.621+0000 7fe521600640 1 -- 172.21.15.90:0/1767134918 wait complete. 2024-04-25T01:16:25.633 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-25T01:16:25.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.829+0000 7feb1f800640 1 Processor -- start 2024-04-25T01:16:25.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.829+0000 7feb1f800640 1 -- start start 2024-04-25T01:16:25.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.831+0000 7feb1f800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb180c8e30 0x7feb180c9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:25.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.831+0000 7feb1f800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7feb180ce090 0x7feb180ca130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:25.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.831+0000 7feb1f800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb18147b00 con 0x7feb180ca670 2024-04-25T01:16:25.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.831+0000 7feb1f800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7feb18071e10 con 0x7feb180ce090 2024-04-25T01:16:25.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.831+0000 7feb1f800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7feb18071800 con 0x7feb180c8e30 2024-04-25T01:16:25.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.832+0000 7feb1ce00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb180c8e30 0x7feb180c9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:25.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.832+0000 7feb0fe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7feb180ce090 0x7feb180ca130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:25.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.832+0000 7feb1ce00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb180c8e30 0x7feb180c9210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.90:54974/0 (socket says 172.21.15.90:54974) 2024-04-25T01:16:25.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.832+0000 7feb1ce00640 1 -- 172.21.15.90:0/1236805348 learned_addr learned my addr 172.21.15.90:0/1236805348 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:25.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.833+0000 7feb1ce00640 1 -- 172.21.15.90:0/1236805348 >> v2:172.21.15.103:3300/0 conn(0x7feb180ce090 msgr2=0x7feb180ca130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:25.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.833+0000 7feb1ce00640 1 --2- 172.21.15.90:0/1236805348 >> v2:172.21.15.103:3300/0 conn(0x7feb180ce090 0x7feb180ca130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:25.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.833+0000 7feb1ce00640 1 -- 172.21.15.90:0/1236805348 >> v1:172.21.15.103:6789/0 conn(0x7feb180ca670 legacy=0x7feb180722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:25.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.833+0000 7feb1ce00640 1 -- 172.21.15.90:0/1236805348 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb180c9750 con 0x7feb180c8e30 2024-04-25T01:16:25.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.834+0000 7feb1ce00640 1 --2- 172.21.15.90:0/1236805348 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb180c8e30 0x7feb180c9210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7feb08009870 tx=0x7feb0802a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3141bfad6217e67a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:25.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.834+0000 7feb17e00640 1 -- 172.21.15.90:0/1236805348 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb08037070 con 0x7feb180c8e30 2024-04-25T01:16:25.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.834+0000 7feb17e00640 1 -- 172.21.15.90:0/1236805348 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb08032440 con 0x7feb180c8e30 2024-04-25T01:16:25.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.834+0000 7feb17e00640 1 -- 172.21.15.90:0/1236805348 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb08032810 con 0x7feb180c8e30 2024-04-25T01:16:25.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.835+0000 7feb1f800640 1 -- 172.21.15.90:0/1236805348 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb180c8e30 msgr2=0x7feb180c9210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:25.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.835+0000 7feb1f800640 1 --2- 172.21.15.90:0/1236805348 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb180c8e30 0x7feb180c9210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7feb08009870 tx=0x7feb0802a250 comp rx=0 tx=0).stop 2024-04-25T01:16:25.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.835+0000 7feb1f800640 1 -- 172.21.15.90:0/1236805348 shutdown_connections 2024-04-25T01:16:25.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.835+0000 7feb1f800640 1 --2- 172.21.15.90:0/1236805348 >> v2:172.21.15.103:3300/0 conn(0x7feb180ce090 0x7feb180ca130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:25.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.835+0000 7feb1f800640 1 --2- 172.21.15.90:0/1236805348 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb180c8e30 0x7feb180c9210 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:25.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.835+0000 7feb1f800640 1 -- 172.21.15.90:0/1236805348 >> 172.21.15.90:0/1236805348 conn(0x7feb180cfae0 msgr2=0x7feb180cfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:25.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.835+0000 7feb1f800640 1 -- 172.21.15.90:0/1236805348 shutdown_connections 2024-04-25T01:16:25.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.835+0000 7feb1f800640 1 -- 172.21.15.90:0/1236805348 wait complete. 2024-04-25T01:16:25.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.837+0000 7feb1f800640 1 Processor -- start 2024-04-25T01:16:25.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.837+0000 7feb1f800640 1 -- start start 2024-04-25T01:16:25.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.838+0000 7feb1f800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb180c8e30 0x7feb181f4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:25.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.838+0000 7feb1f800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb180ca670 0x7feb1815d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:25.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.838+0000 7feb1f800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb180ce090 0x7feb181e96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:25.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.838+0000 7feb1f800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7feb180cf080 con 0x7feb180ca670 2024-04-25T01:16:25.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.838+0000 7feb1f800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7feb180ceda0 con 0x7feb180c8e30 2024-04-25T01:16:25.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.838+0000 7feb1f800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7feb18071aa0 con 0x7feb180ce090 2024-04-25T01:16:25.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.838+0000 7feb0fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb180ca670 0x7feb1815d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:25.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.838+0000 7feb0fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb180ca670 0x7feb1815d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54942/0 (socket says 172.21.15.90:54942) 2024-04-25T01:16:25.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.838+0000 7feb0fe00640 1 -- 172.21.15.90:0/2006712371 learned_addr learned my addr 172.21.15.90:0/2006712371 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:25.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.838+0000 7feb1ce00640 1 --2- 172.21.15.90:0/2006712371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb180c8e30 0x7feb181f4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:25.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.838+0000 7feb0fe00640 1 -- 172.21.15.90:0/2006712371 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb180ce090 msgr2=0x7feb181e96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:25.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.838+0000 7feb0fe00640 1 --2- 172.21.15.90:0/2006712371 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb180ce090 0x7feb181e96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:25.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.838+0000 7feb0fe00640 1 -- 172.21.15.90:0/2006712371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb180c8e30 msgr2=0x7feb181f4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:25.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.838+0000 7feb0fe00640 1 --2- 172.21.15.90:0/2006712371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb180c8e30 0x7feb181f4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:25.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.838+0000 7feb0fe00640 1 -- 172.21.15.90:0/2006712371 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb1815da40 con 0x7feb180ca670 2024-04-25T01:16:25.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.839+0000 7feb0fe00640 1 --2- 172.21.15.90:0/2006712371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb180ca670 0x7feb1815d500 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7feb0000e870 tx=0x7feb0000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:25.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.839+0000 7feb15600640 1 -- 172.21.15.90:0/2006712371 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb00017070 con 0x7feb180ca670 2024-04-25T01:16:25.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.839+0000 7feb15600640 1 -- 172.21.15.90:0/2006712371 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb0000c8e0 con 0x7feb180ca670 2024-04-25T01:16:25.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.839+0000 7feb1f800640 1 -- 172.21.15.90:0/2006712371 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb1806d0a0 con 0x7feb180ca670 2024-04-25T01:16:25.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.839+0000 7feb15600640 1 -- 172.21.15.90:0/2006712371 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0000ccd0 con 0x7feb180ca670 2024-04-25T01:16:25.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.839+0000 7feb1f800640 1 -- 172.21.15.90:0/2006712371 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb1806d660 con 0x7feb180ca670 2024-04-25T01:16:25.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.840+0000 7feb1f800640 1 -- 172.21.15.90:0/2006712371 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb18068fb0 con 0x7feb180ca670 2024-04-25T01:16:25.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.841+0000 7feb15600640 1 -- 172.21.15.90:0/2006712371 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7feb0001a070 con 0x7feb180ca670 2024-04-25T01:16:25.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.842+0000 7feb15600640 1 --2- 172.21.15.90:0/2006712371 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7feaf406be30 0x7feaf406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:25.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.842+0000 7feb15600640 1 -- 172.21.15.90:0/2006712371 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6500+0+0 (secure 0 0 0) 0x7feb0008afc0 con 0x7feb180ca670 2024-04-25T01:16:25.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.842+0000 7feb1ce00640 1 --2- 172.21.15.90:0/2006712371 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7feaf406be30 0x7feaf406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:25.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.843+0000 7feb1ce00640 1 --2- 172.21.15.90:0/2006712371 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7feaf406be30 0x7feaf406e2f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7feb08034000 tx=0x7feb0802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:25.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:25.848+0000 7feb15600640 1 -- 172.21.15.90:0/2006712371 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7feb00059640 con 0x7feb180ca670 2024-04-25T01:16:26.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.044+0000 7feb1f800640 1 -- 172.21.15.90:0/2006712371 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7feb18147880 con 0x7feb180ca670 2024-04-25T01:16:26.031 INFO:tasks.ceph.mds.d.smithi103.stderr:2024-04-25T01:16:26.047+0000 7fd4bc400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T01:16:26.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.620+0000 7feb15600640 1 -- 172.21.15.90:0/2006712371 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v25) ==== 169+0+0 (secure 0 0 0) 0x7feb00058f40 con 0x7feb180ca670 2024-04-25T01:16:26.606 INFO:teuthology.orchestra.run.smithi090.stderr:set pool 4 allow_ec_overwrites to true 2024-04-25T01:16:26.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.624+0000 7feb1f800640 1 -- 172.21.15.90:0/2006712371 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7feaf406be30 msgr2=0x7feaf406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:26.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.624+0000 7feb1f800640 1 --2- 172.21.15.90:0/2006712371 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7feaf406be30 0x7feaf406e2f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7feb08034000 tx=0x7feb0802e000 comp rx=0 tx=0).stop 2024-04-25T01:16:26.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.624+0000 7feb1f800640 1 -- 172.21.15.90:0/2006712371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb180ca670 msgr2=0x7feb1815d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:26.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.624+0000 7feb1f800640 1 --2- 172.21.15.90:0/2006712371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb180ca670 0x7feb1815d500 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7feb0000e870 tx=0x7feb0000ed30 comp rx=0 tx=0).stop 2024-04-25T01:16:26.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.624+0000 7feb1f800640 1 -- 172.21.15.90:0/2006712371 shutdown_connections 2024-04-25T01:16:26.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.624+0000 7feb1f800640 1 --2- 172.21.15.90:0/2006712371 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7feaf406be30 0x7feaf406e2f0 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7feb08034000 tx=0x7feb0802e000 comp rx=0 tx=0).stop 2024-04-25T01:16:26.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.624+0000 7feb1f800640 1 --2- 172.21.15.90:0/2006712371 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb180ce090 0x7feb181e96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:26.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.624+0000 7feb1f800640 1 --2- 172.21.15.90:0/2006712371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb180ca670 0x7feb1815d500 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:26.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.624+0000 7feb1f800640 1 --2- 172.21.15.90:0/2006712371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb180c8e30 0x7feb181f4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:26.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.624+0000 7feb1f800640 1 -- 172.21.15.90:0/2006712371 >> 172.21.15.90:0/2006712371 conn(0x7feb180cfae0 msgr2=0x7feb180af720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:26.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.625+0000 7feb1f800640 1 -- 172.21.15.90:0/2006712371 shutdown_connections 2024-04-25T01:16:26.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.625+0000 7feb1f800640 1 -- 172.21.15.90:0/2006712371 wait complete. 2024-04-25T01:16:26.653 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-25T01:16:26.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.838+0000 7f6075e00640 1 Processor -- start 2024-04-25T01:16:26.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.838+0000 7f6075e00640 1 -- start start 2024-04-25T01:16:26.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.839+0000 7f6075e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6070158a60 0x7f607006df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:26.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.839+0000 7f6075e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f607015f4c0 0x7f60700b3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:26.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.839+0000 7f6075e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f607006d140 con 0x7f6070158680 2024-04-25T01:16:26.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.839+0000 7f6075e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f607006d420 con 0x7f607015f4c0 2024-04-25T01:16:26.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.839+0000 7f6075e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f607006d700 con 0x7f6070158a60 2024-04-25T01:16:26.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.839+0000 7f606f400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f607015f4c0 0x7f60700b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:26.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.840+0000 7f606f400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f607015f4c0 0x7f60700b3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54956/0 (socket says 172.21.15.90:54956) 2024-04-25T01:16:26.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.840+0000 7f606f400640 1 -- 172.21.15.90:0/227314188 learned_addr learned my addr 172.21.15.90:0/227314188 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:26.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.840+0000 7f6066000640 1 --2- 172.21.15.90:0/227314188 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6070158a60 0x7f607006df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:26.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.840+0000 7f606f400640 1 -- 172.21.15.90:0/227314188 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6070158a60 msgr2=0x7f607006df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:26.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.840+0000 7f606f400640 1 --2- 172.21.15.90:0/227314188 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6070158a60 0x7f607006df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:26.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.840+0000 7f606f400640 1 -- 172.21.15.90:0/227314188 >> v1:172.21.15.103:6789/0 conn(0x7f6070158680 legacy=0x7f607006e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:26.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.840+0000 7f606f400640 1 -- 172.21.15.90:0/227314188 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f607006d9e0 con 0x7f607015f4c0 2024-04-25T01:16:26.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.841+0000 7f606f400640 1 --2- 172.21.15.90:0/227314188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f607015f4c0 0x7f60700b3550 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6068009e50 tx=0x7f606802a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3e87484f4d4832d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:26.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.841+0000 7f606e000640 1 -- 172.21.15.90:0/227314188 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6068036070 con 0x7f607015f4c0 2024-04-25T01:16:26.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.841+0000 7f606e000640 1 -- 172.21.15.90:0/227314188 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6068031600 con 0x7f607015f4c0 2024-04-25T01:16:26.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.841+0000 7f606e000640 1 -- 172.21.15.90:0/227314188 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60680319d0 con 0x7f607015f4c0 2024-04-25T01:16:26.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.842+0000 7f6075e00640 1 -- 172.21.15.90:0/227314188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f607015f4c0 msgr2=0x7f60700b3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:26.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.842+0000 7f6075e00640 1 --2- 172.21.15.90:0/227314188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f607015f4c0 0x7f60700b3550 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6068009e50 tx=0x7f606802a640 comp rx=0 tx=0).stop 2024-04-25T01:16:26.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.842+0000 7f6075e00640 1 -- 172.21.15.90:0/227314188 shutdown_connections 2024-04-25T01:16:26.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.842+0000 7f6075e00640 1 --2- 172.21.15.90:0/227314188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f607015f4c0 0x7f60700b3550 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:26.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.842+0000 7f6075e00640 1 --2- 172.21.15.90:0/227314188 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6070158a60 0x7f607006df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:26.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.842+0000 7f6075e00640 1 -- 172.21.15.90:0/227314188 >> 172.21.15.90:0/227314188 conn(0x7f607006b720 msgr2=0x7f60700b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:26.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.842+0000 7f6075e00640 1 -- 172.21.15.90:0/227314188 shutdown_connections 2024-04-25T01:16:26.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.842+0000 7f6075e00640 1 -- 172.21.15.90:0/227314188 wait complete. 2024-04-25T01:16:26.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.844+0000 7f6075e00640 1 Processor -- start 2024-04-25T01:16:26.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.844+0000 7f6075e00640 1 -- start start 2024-04-25T01:16:26.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.845+0000 7f6075e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6070158680 0x7f60701f86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:26.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.845+0000 7f6075e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6070158a60 0x7f607014b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:26.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.845+0000 7f6075e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f607015f4c0 0x7f6070154c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:26.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.845+0000 7f6075e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f60700b3c10 con 0x7f6070158a60 2024-04-25T01:16:26.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.845+0000 7f6075e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f60700b3a90 con 0x7f607015f4c0 2024-04-25T01:16:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.845+0000 7f6075e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f60700b3d90 con 0x7f6070158680 2024-04-25T01:16:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.845+0000 7f6066000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6070158a60 0x7f607014b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.845+0000 7f6066000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6070158a60 0x7f607014b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54968/0 (socket says 172.21.15.90:54968) 2024-04-25T01:16:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.845+0000 7f6066000640 1 -- 172.21.15.90:0/1041992886 learned_addr learned my addr 172.21.15.90:0/1041992886 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.845+0000 7f606f400640 1 --2- 172.21.15.90:0/1041992886 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f607015f4c0 0x7f6070154c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.845+0000 7f6066000640 1 -- 172.21.15.90:0/1041992886 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6070158680 msgr2=0x7f60701f86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.845+0000 7f6066000640 1 --2- 172.21.15.90:0/1041992886 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6070158680 0x7f60701f86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.845+0000 7f6066000640 1 -- 172.21.15.90:0/1041992886 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f607015f4c0 msgr2=0x7f6070154c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.845+0000 7f6066000640 1 --2- 172.21.15.90:0/1041992886 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f607015f4c0 0x7f6070154c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.845+0000 7f6066000640 1 -- 172.21.15.90:0/1041992886 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f607014e9e0 con 0x7f6070158a60 2024-04-25T01:16:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.846+0000 7f6066000640 1 --2- 172.21.15.90:0/1041992886 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6070158a60 0x7f607014b5a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f605800c7f0 tx=0x7f605800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.846+0000 7f6067e00640 1 -- 172.21.15.90:0/1041992886 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6058015070 con 0x7f6070158a60 2024-04-25T01:16:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.846+0000 7f6067e00640 1 -- 172.21.15.90:0/1041992886 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6058007820 con 0x7f6070158a60 2024-04-25T01:16:26.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.846+0000 7f6067e00640 1 -- 172.21.15.90:0/1041992886 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6058007bd0 con 0x7f6070158a60 2024-04-25T01:16:26.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.846+0000 7f6075e00640 1 -- 172.21.15.90:0/1041992886 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f607014fa10 con 0x7f6070158a60 2024-04-25T01:16:26.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.846+0000 7f6075e00640 1 -- 172.21.15.90:0/1041992886 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f607014ef30 con 0x7f6070158a60 2024-04-25T01:16:26.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.847+0000 7f6075e00640 1 -- 172.21.15.90:0/1041992886 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6070068fb0 con 0x7f6070158a60 2024-04-25T01:16:26.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.848+0000 7f6067e00640 1 -- 172.21.15.90:0/1041992886 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6058018070 con 0x7f6070158a60 2024-04-25T01:16:26.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.849+0000 7f6067e00640 1 --2- 172.21.15.90:0/1041992886 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f604c06be30 0x7f604c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:26.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.849+0000 7f6067e00640 1 -- 172.21.15.90:0/1041992886 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6500+0+0 (secure 0 0 0) 0x7f605808ae80 con 0x7f6070158a60 2024-04-25T01:16:26.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.849+0000 7f606ea00640 1 --2- 172.21.15.90:0/1041992886 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f604c06be30 0x7f604c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:26.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.850+0000 7f606ea00640 1 --2- 172.21.15.90:0/1041992886 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f604c06be30 0x7f604c06e2f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6060006fd0 tx=0x7f6060006bc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:26.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:26.854+0000 7f6067e00640 1 -- 172.21.15.90:0/1041992886 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6058059500 con 0x7f6070158a60 2024-04-25T01:16:27.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.090+0000 7f6075e00640 1 -- 172.21.15.90:0/1041992886 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f60700712a0 con 0x7f6070158a60 2024-04-25T01:16:27.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.698+0000 7f6067e00640 1 -- 172.21.15.90:0/1041992886 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v7) ==== 137+0+0 (secure 0 0 0) 0x7f6058058e00 con 0x7f6070158a60 2024-04-25T01:16:27.685 INFO:teuthology.orchestra.run.smithi090.stderr:added data pool 4 to fsmap 2024-04-25T01:16:27.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.701+0000 7f6075e00640 1 -- 172.21.15.90:0/1041992886 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f604c06be30 msgr2=0x7f604c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:27.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.701+0000 7f6075e00640 1 --2- 172.21.15.90:0/1041992886 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f604c06be30 0x7f604c06e2f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6060006fd0 tx=0x7f6060006bc0 comp rx=0 tx=0).stop 2024-04-25T01:16:27.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.701+0000 7f6075e00640 1 -- 172.21.15.90:0/1041992886 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6070158a60 msgr2=0x7f607014b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:27.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.701+0000 7f6075e00640 1 --2- 172.21.15.90:0/1041992886 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6070158a60 0x7f607014b5a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f605800c7f0 tx=0x7f605800ccb0 comp rx=0 tx=0).stop 2024-04-25T01:16:27.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.702+0000 7f6075e00640 1 -- 172.21.15.90:0/1041992886 shutdown_connections 2024-04-25T01:16:27.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.702+0000 7f6075e00640 1 --2- 172.21.15.90:0/1041992886 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f604c06be30 0x7f604c06e2f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:27.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.702+0000 7f6075e00640 1 --2- 172.21.15.90:0/1041992886 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f607015f4c0 0x7f6070154c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:27.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.702+0000 7f6075e00640 1 --2- 172.21.15.90:0/1041992886 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6070158a60 0x7f607014b5a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:27.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.702+0000 7f6075e00640 1 --2- 172.21.15.90:0/1041992886 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6070158680 0x7f60701f86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:27.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.702+0000 7f6075e00640 1 -- 172.21.15.90:0/1041992886 >> 172.21.15.90:0/1041992886 conn(0x7f607006b720 msgr2=0x7f60700b08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:27.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.702+0000 7f6075e00640 1 -- 172.21.15.90:0/1041992886 shutdown_connections 2024-04-25T01:16:27.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.702+0000 7f6075e00640 1 -- 172.21.15.90:0/1041992886 wait complete. 2024-04-25T01:16:27.708 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:16:27.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.918+0000 7f657dc00640 1 Processor -- start 2024-04-25T01:16:27.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.918+0000 7f657dc00640 1 -- start start 2024-04-25T01:16:27.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.920+0000 7f657dc00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f657806c840 0x7f65780cd060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:27.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.920+0000 7f657dc00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65780b4fd0 con 0x7f65780b4a10 2024-04-25T01:16:27.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.920+0000 7f657dc00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f657806d520 con 0x7f657815a570 2024-04-25T01:16:27.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.920+0000 7f657dc00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f65780b52b0 con 0x7f657806c840 2024-04-25T01:16:27.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.920+0000 7f656e000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f65780b4a10 0x7f657806c130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:46894/0 (socket says 172.21.15.90:46894) 2024-04-25T01:16:27.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.920+0000 7f656e000640 1 -- 172.21.15.90:0/1515208444 learned_addr learned my addr 172.21.15.90:0/1515208444 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:27.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.921+0000 7f6576000640 1 -- 172.21.15.90:0/1515208444 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 782504629 0 0) 0x7f65780b4fd0 con 0x7f65780b4a10 2024-04-25T01:16:27.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.921+0000 7f6577400640 1 --2- 172.21.15.90:0/1515208444 >> v2:172.21.15.103:3300/0 conn(0x7f657806c840 0x7f65780cd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:27.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.921+0000 7f6576000640 1 -- 172.21.15.90:0/1515208444 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f655c003520 con 0x7f65780b4a10 2024-04-25T01:16:27.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.921+0000 7f6576000640 1 -- 172.21.15.90:0/1515208444 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1530901391 0 0) 0x7f655c003520 con 0x7f65780b4a10 2024-04-25T01:16:27.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.921+0000 7f6576000640 1 -- 172.21.15.90:0/1515208444 >> v2:172.21.15.103:3300/0 conn(0x7f657806c840 msgr2=0x7f65780cd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:27.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.921+0000 7f6576000640 1 --2- 172.21.15.90:0/1515208444 >> v2:172.21.15.103:3300/0 conn(0x7f657806c840 0x7f65780cd060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:27.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.921+0000 7f6576000640 1 -- 172.21.15.90:0/1515208444 >> v1:172.21.15.103:6789/0 conn(0x7f657815a570 legacy=0x7f65780b3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:27.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.921+0000 7f6576000640 1 -- 172.21.15.90:0/1515208444 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f657806d800 con 0x7f65780b4a10 2024-04-25T01:16:27.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.921+0000 7f6576000640 1 -- 172.21.15.90:0/1515208444 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f6560001670 con 0x7f65780b4a10 2024-04-25T01:16:27.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.921+0000 7f6576000640 1 -- 172.21.15.90:0/1515208444 >> v1:172.21.15.90:6789/0 conn(0x7f65780b4a10 legacy=0x7f657806c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:27.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.921+0000 7f6576000640 1 --2- 172.21.15.90:0/1515208444 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f655c0039b0 0x7f655c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:27.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.921+0000 7f6576000640 1 --2- 172.21.15.90:0/1515208444 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f655c004800 0x7f655c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:27.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.921+0000 7f6576000640 1 --2- 172.21.15.90:0/1515208444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f655c0051c0 0x7f655c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:27.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.921+0000 7f6576000640 1 -- 172.21.15.90:0/1515208444 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f65780b52b0 con 0x7f655c0039b0 2024-04-25T01:16:27.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.921+0000 7f6576000640 1 -- 172.21.15.90:0/1515208444 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f655c00c160 con 0x7f655c0051c0 2024-04-25T01:16:27.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.921+0000 7f6576000640 1 -- 172.21.15.90:0/1515208444 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f655c00c360 con 0x7f655c004800 2024-04-25T01:16:27.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.921+0000 7f6576a00640 1 --2- 172.21.15.90:0/1515208444 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f655c0039b0 0x7f655c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:27.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.922+0000 7f6576a00640 1 -- 172.21.15.90:0/1515208444 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f655c004800 msgr2=0x7f655c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:27.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.922+0000 7f6576a00640 1 --2- 172.21.15.90:0/1515208444 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f655c004800 0x7f655c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:27.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.922+0000 7f6576a00640 1 -- 172.21.15.90:0/1515208444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f655c0051c0 msgr2=0x7f655c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:27.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.922+0000 7f6576a00640 1 --2- 172.21.15.90:0/1515208444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f655c0051c0 0x7f655c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:27.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.922+0000 7f6576a00640 1 -- 172.21.15.90:0/1515208444 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f655c003520 con 0x7f655c0039b0 2024-04-25T01:16:27.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.922+0000 7f6577400640 1 -- 172.21.15.90:0/1515208444 reap_dead start 2024-04-25T01:16:27.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.923+0000 7f6576a00640 1 --2- 172.21.15.90:0/1515208444 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f655c0039b0 0x7f655c003dd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6568001e50 tx=0x7f655c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26df8ab22097d8a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:27.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.923+0000 7f6576000640 1 -- 172.21.15.90:0/1515208444 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6568001a70 con 0x7f655c0039b0 2024-04-25T01:16:27.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.923+0000 7f6576000640 1 -- 172.21.15.90:0/1515208444 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6568003970 con 0x7f655c0039b0 2024-04-25T01:16:27.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.923+0000 7f6576000640 1 -- 172.21.15.90:0/1515208444 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6568003d60 con 0x7f655c0039b0 2024-04-25T01:16:27.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.923+0000 7f657dc00640 1 -- 172.21.15.90:0/1515208444 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f655c0039b0 msgr2=0x7f655c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:27.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.924+0000 7f657dc00640 1 --2- 172.21.15.90:0/1515208444 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f655c0039b0 0x7f655c003dd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6568001e50 tx=0x7f655c005830 comp rx=0 tx=0).stop 2024-04-25T01:16:27.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.924+0000 7f657dc00640 1 -- 172.21.15.90:0/1515208444 shutdown_connections 2024-04-25T01:16:27.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.924+0000 7f657dc00640 1 --2- 172.21.15.90:0/1515208444 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f655c0039b0 0x7f655c003dd0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:27.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.924+0000 7f657dc00640 1 -- 172.21.15.90:0/1515208444 >> 172.21.15.90:0/1515208444 conn(0x7f6578071550 msgr2=0x7f6578071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:27.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.924+0000 7f657dc00640 1 -- 172.21.15.90:0/1515208444 shutdown_connections 2024-04-25T01:16:27.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.924+0000 7f657dc00640 1 -- 172.21.15.90:0/1515208444 wait complete. 2024-04-25T01:16:27.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.925+0000 7f657dc00640 1 Processor -- start 2024-04-25T01:16:27.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.926+0000 7f657dc00640 1 -- start start 2024-04-25T01:16:27.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.926+0000 7f657dc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f655c0039b0 0x7f6578144dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:27.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.927+0000 7f657dc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f657815cf60 0x7f657814d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:27.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.927+0000 7f657dc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f657815d3e0 0x7f657814d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:27.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.927+0000 7f657dc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f65780cd3f0 con 0x7f657815cf60 2024-04-25T01:16:27.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.927+0000 7f657dc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f65780cd270 con 0x7f657815d3e0 2024-04-25T01:16:27.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.927+0000 7f657dc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f657806db70 con 0x7f655c0039b0 2024-04-25T01:16:27.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.927+0000 7f656e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f657815cf60 0x7f657814d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:27.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.927+0000 7f656e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f657815cf60 0x7f657814d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54982/0 (socket says 172.21.15.90:54982) 2024-04-25T01:16:27.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.927+0000 7f656e000640 1 -- 172.21.15.90:0/2120904316 learned_addr learned my addr 172.21.15.90:0/2120904316 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:27.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.927+0000 7f6576a00640 1 --2- 172.21.15.90:0/2120904316 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f655c0039b0 0x7f6578144dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:27.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.927+0000 7f6577400640 1 --2- 172.21.15.90:0/2120904316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f657815d3e0 0x7f657814d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:27.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.927+0000 7f656e000640 1 -- 172.21.15.90:0/2120904316 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f655c0039b0 msgr2=0x7f6578144dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:27.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.927+0000 7f656e000640 1 --2- 172.21.15.90:0/2120904316 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f655c0039b0 0x7f6578144dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:27.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.927+0000 7f656e000640 1 -- 172.21.15.90:0/2120904316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f657815d3e0 msgr2=0x7f657814d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:27.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.927+0000 7f656e000640 1 --2- 172.21.15.90:0/2120904316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f657815d3e0 0x7f657814d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:27.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.927+0000 7f656e000640 1 -- 172.21.15.90:0/2120904316 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6578145310 con 0x7f657815cf60 2024-04-25T01:16:27.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.927+0000 7f6576a00640 1 --2- 172.21.15.90:0/2120904316 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f655c0039b0 0x7f6578144dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:27.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.927+0000 7f656e000640 1 --2- 172.21.15.90:0/2120904316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f657815cf60 0x7f657814d3e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6560001eb0 tx=0x7f656000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:27.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.928+0000 7f656f400640 1 -- 172.21.15.90:0/2120904316 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6560018070 con 0x7f657815cf60 2024-04-25T01:16:27.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.928+0000 7f656f400640 1 -- 172.21.15.90:0/2120904316 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65600072d0 con 0x7f657815cf60 2024-04-25T01:16:27.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.928+0000 7f656f400640 1 -- 172.21.15.90:0/2120904316 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65600047e0 con 0x7f657815cf60 2024-04-25T01:16:27.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.928+0000 7f657dc00640 1 -- 172.21.15.90:0/2120904316 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f657815b500 con 0x7f657815cf60 2024-04-25T01:16:27.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.928+0000 7f657dc00640 1 -- 172.21.15.90:0/2120904316 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f657815bac0 con 0x7f657815cf60 2024-04-25T01:16:27.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.929+0000 7f657dc00640 1 -- 172.21.15.90:0/2120904316 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6578068fb0 con 0x7f657815cf60 2024-04-25T01:16:27.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.930+0000 7f656f400640 1 -- 172.21.15.90:0/2120904316 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f656001b070 con 0x7f657815cf60 2024-04-25T01:16:27.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.931+0000 7f656f400640 1 --2- 172.21.15.90:0/2120904316 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f655406be60 0x7f655406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:27.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.931+0000 7f656f400640 1 -- 172.21.15.90:0/2120904316 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f656008a8a0 con 0x7f657815cf60 2024-04-25T01:16:27.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.931+0000 7f6576a00640 1 --2- 172.21.15.90:0/2120904316 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f655406be60 0x7f655406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:27.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.932+0000 7f6576a00640 1 --2- 172.21.15.90:0/2120904316 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f655406be60 0x7f655406e320 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6568032000 tx=0x7f656802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:27.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:27.935+0000 7f656f400640 1 -- 172.21.15.90:0/2120904316 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6578145310 con 0x7f657815cf60 2024-04-25T01:16:28.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.159+0000 7f657dc00640 1 -- 172.21.15.90:0/2120904316 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f657815dce0 con 0x7f657815cf60 2024-04-25T01:16:28.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.160+0000 7f656f400640 1 -- 172.21.15.90:0/2120904316 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5180 (secure 0 0 0) 0x7f6560058f10 con 0x7f657815cf60 2024-04-25T01:16:28.145 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:16:28.145 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6834/2605945289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":2605945289},{"type":"v1","addr":"172.21.15.103:6834","nonce":2605945289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6836/704639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":704639426},{"type":"v1","addr":"172.21.15.90:6836","nonce":704639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4245,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3477867755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6835","nonce":3477867755},{"type":"v1","addr":"172.21.15.90:6837","nonce":3477867755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:16:22.060018+0000","modified":"2024-04-25T01:16:27.093210+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6835/4291578198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6833","nonce":4291578198},{"type":"v1","addr":"172.21.15.103:6835","nonce":4291578198}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1}]} 2024-04-25T01:16:28.146 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 7 2024-04-25T01:16:28.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.162+0000 7f657dc00640 1 -- 172.21.15.90:0/2120904316 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f655406be60 msgr2=0x7f655406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:28.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.162+0000 7f657dc00640 1 --2- 172.21.15.90:0/2120904316 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f655406be60 0x7f655406e320 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6568032000 tx=0x7f656802e000 comp rx=0 tx=0).stop 2024-04-25T01:16:28.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.162+0000 7f657dc00640 1 -- 172.21.15.90:0/2120904316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f657815cf60 msgr2=0x7f657814d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:28.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.162+0000 7f657dc00640 1 --2- 172.21.15.90:0/2120904316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f657815cf60 0x7f657814d3e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6560001eb0 tx=0x7f656000cb90 comp rx=0 tx=0).stop 2024-04-25T01:16:28.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.163+0000 7f657dc00640 1 -- 172.21.15.90:0/2120904316 shutdown_connections 2024-04-25T01:16:28.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.163+0000 7f657dc00640 1 --2- 172.21.15.90:0/2120904316 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f655406be60 0x7f655406e320 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:28.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.163+0000 7f657dc00640 1 --2- 172.21.15.90:0/2120904316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f657815d3e0 0x7f657814d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:28.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.163+0000 7f657dc00640 1 --2- 172.21.15.90:0/2120904316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f657815cf60 0x7f657814d3e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:28.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.163+0000 7f657dc00640 1 --2- 172.21.15.90:0/2120904316 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f655c0039b0 0x7f6578144dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:28.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.163+0000 7f657dc00640 1 -- 172.21.15.90:0/2120904316 >> 172.21.15.90:0/2120904316 conn(0x7f6578071550 msgr2=0x7f657806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:28.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.163+0000 7f657dc00640 1 -- 172.21.15.90:0/2120904316 shutdown_connections 2024-04-25T01:16:28.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.163+0000 7f657dc00640 1 -- 172.21.15.90:0/2120904316 wait complete. 2024-04-25T01:16:28.165 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:16:28.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.368+0000 7fd942800640 1 Processor -- start 2024-04-25T01:16:28.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.369+0000 7fd942800640 1 -- start start 2024-04-25T01:16:28.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.370+0000 7fd942800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd93c0ae730 0x7fd93c0cd900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:28.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.370+0000 7fd942800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd93c15ca10 0x7fd93c15cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:28.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.370+0000 7fd942800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd93c15c220 con 0x7fd93c06b120 2024-04-25T01:16:28.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.370+0000 7fd942800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd93c0cef40 con 0x7fd93c0ae730 2024-04-25T01:16:28.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.370+0000 7fd942800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd93c15c580 con 0x7fd93c15ca10 2024-04-25T01:16:28.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.370+0000 7fd941400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd93c06b120 0x7fd93c06b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:46910/0 (socket says 172.21.15.90:46910) 2024-04-25T01:16:28.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.370+0000 7fd941400640 1 -- 172.21.15.90:0/31787741 learned_addr learned my addr 172.21.15.90:0/31787741 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:28.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.371+0000 7fd93be00640 1 -- 172.21.15.90:0/31787741 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3928026529 0 0) 0x7fd93c15c220 con 0x7fd93c06b120 2024-04-25T01:16:28.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.371+0000 7fd93be00640 1 -- 172.21.15.90:0/31787741 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd924003520 con 0x7fd93c06b120 2024-04-25T01:16:28.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.371+0000 7fd940a00640 1 --2- 172.21.15.90:0/31787741 >> v2:172.21.15.103:3300/0 conn(0x7fd93c0ae730 0x7fd93c0cd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:28.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.371+0000 7fd941e00640 1 --2- 172.21.15.90:0/31787741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd93c15ca10 0x7fd93c15cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:28.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.371+0000 7fd93be00640 1 -- 172.21.15.90:0/31787741 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3072552054 0 0) 0x7fd924003520 con 0x7fd93c06b120 2024-04-25T01:16:28.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.371+0000 7fd93be00640 1 -- 172.21.15.90:0/31787741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd93c15ca10 msgr2=0x7fd93c15cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:28.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.371+0000 7fd93be00640 1 --2- 172.21.15.90:0/31787741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd93c15ca10 0x7fd93c15cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:28.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.371+0000 7fd93be00640 1 -- 172.21.15.90:0/31787741 >> v2:172.21.15.103:3300/0 conn(0x7fd93c0ae730 msgr2=0x7fd93c0cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:28.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.371+0000 7fd93be00640 1 --2- 172.21.15.90:0/31787741 >> v2:172.21.15.103:3300/0 conn(0x7fd93c0ae730 0x7fd93c0cd900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:28.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.371+0000 7fd93be00640 1 -- 172.21.15.90:0/31787741 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd93c0aec80 con 0x7fd93c06b120 2024-04-25T01:16:28.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.371+0000 7fd93be00640 1 -- 172.21.15.90:0/31787741 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7fd930001670 con 0x7fd93c06b120 2024-04-25T01:16:28.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.371+0000 7fd93be00640 1 -- 172.21.15.90:0/31787741 >> v1:172.21.15.90:6789/0 conn(0x7fd93c06b120 legacy=0x7fd93c06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:28.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.371+0000 7fd93be00640 1 --2- 172.21.15.90:0/31787741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9240039b0 0x7fd924003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:28.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.371+0000 7fd93be00640 1 --2- 172.21.15.90:0/31787741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd924004800 0x7fd924004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:28.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.372+0000 7fd93be00640 1 --2- 172.21.15.90:0/31787741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd93c15ca10 0x7fd9240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:28.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.372+0000 7fd93be00640 1 -- 172.21.15.90:0/31787741 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd93c15c580 con 0x7fd924004800 2024-04-25T01:16:28.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.372+0000 7fd93be00640 1 -- 172.21.15.90:0/31787741 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd92400ba20 con 0x7fd9240039b0 2024-04-25T01:16:28.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.372+0000 7fd93be00640 1 -- 172.21.15.90:0/31787741 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd92400bc20 con 0x7fd93c15ca10 2024-04-25T01:16:28.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.372+0000 7fd940a00640 1 --2- 172.21.15.90:0/31787741 >> v2:172.21.15.103:3300/0 conn(0x7fd93c0ae730 0x7fd93c0cd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:16:28.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.372+0000 7fd940a00640 1 --2- 172.21.15.90:0/31787741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd924004800 0x7fd924004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:28.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.372+0000 7fd941400640 1 --2- 172.21.15.90:0/31787741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9240039b0 0x7fd924003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:28.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.372+0000 7fd940a00640 1 -- 172.21.15.90:0/31787741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd93c15ca10 msgr2=0x7fd9240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:28.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.372+0000 7fd940a00640 1 --2- 172.21.15.90:0/31787741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd93c15ca10 0x7fd9240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:28.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.372+0000 7fd940a00640 1 -- 172.21.15.90:0/31787741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9240039b0 msgr2=0x7fd924003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:28.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.372+0000 7fd940a00640 1 --2- 172.21.15.90:0/31787741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9240039b0 0x7fd924003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:28.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.372+0000 7fd940a00640 1 -- 172.21.15.90:0/31787741 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd924003520 con 0x7fd924004800 2024-04-25T01:16:28.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.372+0000 7fd941400640 1 --2- 172.21.15.90:0/31787741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9240039b0 0x7fd924003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:28.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.373+0000 7fd940a00640 1 --2- 172.21.15.90:0/31787741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd924004800 0x7fd924004c80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd928006fd0 tx=0x7fd924005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7de6564342bb4366 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:28.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.373+0000 7fd93be00640 1 -- 172.21.15.90:0/31787741 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd928037070 con 0x7fd924004800 2024-04-25T01:16:28.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.373+0000 7fd93be00640 1 -- 172.21.15.90:0/31787741 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd92802ace0 con 0x7fd924004800 2024-04-25T01:16:28.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.373+0000 7fd93be00640 1 -- 172.21.15.90:0/31787741 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd92800f040 con 0x7fd924004800 2024-04-25T01:16:28.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.374+0000 7fd942800640 1 -- 172.21.15.90:0/31787741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd924004800 msgr2=0x7fd924004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:28.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.374+0000 7fd942800640 1 --2- 172.21.15.90:0/31787741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd924004800 0x7fd924004c80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd928006fd0 tx=0x7fd924005830 comp rx=0 tx=0).stop 2024-04-25T01:16:28.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.374+0000 7fd941e00640 1 -- 172.21.15.90:0/31787741 reap_dead start 2024-04-25T01:16:28.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.374+0000 7fd942800640 1 -- 172.21.15.90:0/31787741 shutdown_connections 2024-04-25T01:16:28.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.374+0000 7fd942800640 1 -- 172.21.15.90:0/31787741 >> 172.21.15.90:0/31787741 conn(0x7fd93c0b1660 msgr2=0x7fd93c0b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:28.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.374+0000 7fd942800640 1 -- 172.21.15.90:0/31787741 shutdown_connections 2024-04-25T01:16:28.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.374+0000 7fd942800640 1 -- 172.21.15.90:0/31787741 wait complete. 2024-04-25T01:16:28.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.376+0000 7fd942800640 1 Processor -- start 2024-04-25T01:16:28.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.376+0000 7fd942800640 1 -- start start 2024-04-25T01:16:28.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.377+0000 7fd942800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd924004800 0x7fd93c140f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:28.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.377+0000 7fd942800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd93c0a9f70 0x7fd93c1416a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:28.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.377+0000 7fd942800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd93c142f20 0x7fd93c150310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:28.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.377+0000 7fd942800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd93c153080 con 0x7fd93c0a9f70 2024-04-25T01:16:28.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.377+0000 7fd942800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd93c0af240 con 0x7fd93c142f20 2024-04-25T01:16:28.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.377+0000 7fd942800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd93c141be0 con 0x7fd924004800 2024-04-25T01:16:28.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.377+0000 7fd940a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd93c0a9f70 0x7fd93c1416a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:28.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.377+0000 7fd940a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd93c0a9f70 0x7fd93c1416a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55000/0 (socket says 172.21.15.90:55000) 2024-04-25T01:16:28.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.377+0000 7fd940a00640 1 -- 172.21.15.90:0/64788433 learned_addr learned my addr 172.21.15.90:0/64788433 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:28.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.377+0000 7fd940a00640 1 -- 172.21.15.90:0/64788433 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd924004800 msgr2=0x7fd93c140f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:28.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.377+0000 7fd940a00640 1 --2- 172.21.15.90:0/64788433 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd924004800 0x7fd93c140f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:28.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.377+0000 7fd940a00640 1 -- 172.21.15.90:0/64788433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd93c142f20 msgr2=0x7fd93c150310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:28.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.377+0000 7fd940a00640 1 --2- 172.21.15.90:0/64788433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd93c142f20 0x7fd93c150310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:28.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.377+0000 7fd940a00640 1 -- 172.21.15.90:0/64788433 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd93c141e80 con 0x7fd93c0a9f70 2024-04-25T01:16:28.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.378+0000 7fd940a00640 1 --2- 172.21.15.90:0/64788433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd93c0a9f70 0x7fd93c1416a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd92802a7d0 tx=0x7fd928004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:28.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.378+0000 7fd939600640 1 -- 172.21.15.90:0/64788433 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd928037050 con 0x7fd93c0a9f70 2024-04-25T01:16:28.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.378+0000 7fd939600640 1 -- 172.21.15.90:0/64788433 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd928043070 con 0x7fd93c0a9f70 2024-04-25T01:16:28.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.378+0000 7fd942800640 1 -- 172.21.15.90:0/64788433 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd93c142160 con 0x7fd93c0a9f70 2024-04-25T01:16:28.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.378+0000 7fd939600640 1 -- 172.21.15.90:0/64788433 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd92803c830 con 0x7fd93c0a9f70 2024-04-25T01:16:28.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.378+0000 7fd942800640 1 -- 172.21.15.90:0/64788433 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd93c142720 con 0x7fd93c0a9f70 2024-04-25T01:16:28.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.379+0000 7fd942800640 1 -- 172.21.15.90:0/64788433 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd93c075350 con 0x7fd93c0a9f70 2024-04-25T01:16:28.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.380+0000 7fd939600640 1 -- 172.21.15.90:0/64788433 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd92803c9d0 con 0x7fd93c0a9f70 2024-04-25T01:16:28.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.381+0000 7fd939600640 1 --2- 172.21.15.90:0/64788433 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd91c06be30 0x7fd91c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:28.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.381+0000 7fd939600640 1 -- 172.21.15.90:0/64788433 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fd9280abc60 con 0x7fd93c0a9f70 2024-04-25T01:16:28.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.381+0000 7fd941400640 1 --2- 172.21.15.90:0/64788433 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd91c06be30 0x7fd91c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:28.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.382+0000 7fd941400640 1 --2- 172.21.15.90:0/64788433 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd91c06be30 0x7fd91c06e2f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd9300017e0 tx=0x7fd930009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:28.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.386+0000 7fd939600640 1 -- 172.21.15.90:0/64788433 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd92807a1a0 con 0x7fd93c0a9f70 2024-04-25T01:16:28.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.585+0000 7fd942800640 1 -- 172.21.15.90:0/64788433 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd93c06ccd0 con 0x7fd93c0a9f70 2024-04-25T01:16:28.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.589+0000 7fd939600640 1 -- 172.21.15.90:0/64788433 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18482 (secure 0 0 0) 0x7fd928032e20 con 0x7fd93c0a9f70 2024-04-25T01:16:28.574 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:16:28.575 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":26,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:16:27.092855+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:16:20.094512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:16:21.078268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:16:24.670642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:16:28.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.591+0000 7fd942800640 1 -- 172.21.15.90:0/64788433 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd91c06be30 msgr2=0x7fd91c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:28.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.591+0000 7fd942800640 1 --2- 172.21.15.90:0/64788433 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd91c06be30 0x7fd91c06e2f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd9300017e0 tx=0x7fd930009040 comp rx=0 tx=0).stop 2024-04-25T01:16:28.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.591+0000 7fd942800640 1 -- 172.21.15.90:0/64788433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd93c0a9f70 msgr2=0x7fd93c1416a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:28.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.591+0000 7fd942800640 1 --2- 172.21.15.90:0/64788433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd93c0a9f70 0x7fd93c1416a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd92802a7d0 tx=0x7fd928004730 comp rx=0 tx=0).stop 2024-04-25T01:16:28.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.591+0000 7fd942800640 1 -- 172.21.15.90:0/64788433 shutdown_connections 2024-04-25T01:16:28.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.591+0000 7fd942800640 1 --2- 172.21.15.90:0/64788433 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd91c06be30 0x7fd91c06e2f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:28.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.591+0000 7fd942800640 1 --2- 172.21.15.90:0/64788433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd93c142f20 0x7fd93c150310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:28.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.591+0000 7fd942800640 1 --2- 172.21.15.90:0/64788433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd93c0a9f70 0x7fd93c1416a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:28.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.591+0000 7fd942800640 1 --2- 172.21.15.90:0/64788433 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd924004800 0x7fd93c140f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:28.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.591+0000 7fd942800640 1 -- 172.21.15.90:0/64788433 >> 172.21.15.90:0/64788433 conn(0x7fd93c0b1660 msgr2=0x7fd93c06e100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:28.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.592+0000 7fd942800640 1 -- 172.21.15.90:0/64788433 shutdown_connections 2024-04-25T01:16:28.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.592+0000 7fd942800640 1 -- 172.21.15.90:0/64788433 wait complete. 2024-04-25T01:16:28.596 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:16:28.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.803+0000 7faff9200640 1 Processor -- start 2024-04-25T01:16:28.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.803+0000 7faff9200640 1 -- start start 2024-04-25T01:16:28.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.805+0000 7faff9200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7faff40b50e0 0x7faff4161010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:28.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.805+0000 7faff9200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faff40cf3e0 con 0x7faff40d0ca0 2024-04-25T01:16:28.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.805+0000 7faff9200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faff40cf6c0 con 0x7faff40d08c0 2024-04-25T01:16:28.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.805+0000 7faff9200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7faff40d2e80 con 0x7faff40b50e0 2024-04-25T01:16:28.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.805+0000 7faff3e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7faff40d0ca0 0x7faff4161550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:46912/0 (socket says 172.21.15.90:46912) 2024-04-25T01:16:28.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.805+0000 7faff3e00640 1 -- 172.21.15.90:0/1011736382 learned_addr learned my addr 172.21.15.90:0/1011736382 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:28.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.805+0000 7faff2a00640 1 --2- 172.21.15.90:0/1011736382 >> v2:172.21.15.103:3300/0 conn(0x7faff40b50e0 0x7faff4161010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:28.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.805+0000 7faff2000640 1 -- 172.21.15.90:0/1011736382 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3615437620 0 0) 0x7faff40cf3e0 con 0x7faff40d0ca0 2024-04-25T01:16:28.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.805+0000 7faff2000640 1 -- 172.21.15.90:0/1011736382 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafd8003520 con 0x7faff40d0ca0 2024-04-25T01:16:28.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.806+0000 7faff2000640 1 -- 172.21.15.90:0/1011736382 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2054521793 0 0) 0x7fafd8003520 con 0x7faff40d0ca0 2024-04-25T01:16:28.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.806+0000 7faff2000640 1 -- 172.21.15.90:0/1011736382 >> v2:172.21.15.103:3300/0 conn(0x7faff40b50e0 msgr2=0x7faff4161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:28.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.806+0000 7faff2000640 1 --2- 172.21.15.90:0/1011736382 >> v2:172.21.15.103:3300/0 conn(0x7faff40b50e0 0x7faff4161010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:28.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.806+0000 7faff2000640 1 -- 172.21.15.90:0/1011736382 >> v1:172.21.15.103:6789/0 conn(0x7faff40d08c0 legacy=0x7faff40ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:28.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.806+0000 7faff2000640 1 -- 172.21.15.90:0/1011736382 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faff40cf9a0 con 0x7faff40d0ca0 2024-04-25T01:16:28.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.806+0000 7faff2000640 1 -- 172.21.15.90:0/1011736382 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7fafec0019f0 con 0x7faff40d0ca0 2024-04-25T01:16:28.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.806+0000 7faff2000640 1 -- 172.21.15.90:0/1011736382 >> v1:172.21.15.90:6789/0 conn(0x7faff40d0ca0 legacy=0x7faff4161550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:28.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.806+0000 7faff2000640 1 --2- 172.21.15.90:0/1011736382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafd80039b0 0x7fafd8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:28.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.806+0000 7faff2000640 1 --2- 172.21.15.90:0/1011736382 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fafd8004800 0x7fafd8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:28.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.806+0000 7faff2000640 1 --2- 172.21.15.90:0/1011736382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafd80051c0 0x7fafd800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:28.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.806+0000 7faff2000640 1 -- 172.21.15.90:0/1011736382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7faff40d2e80 con 0x7fafd80051c0 2024-04-25T01:16:28.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.806+0000 7faff2000640 1 -- 172.21.15.90:0/1011736382 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fafd800c060 con 0x7fafd80039b0 2024-04-25T01:16:28.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.806+0000 7faff2000640 1 -- 172.21.15.90:0/1011736382 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fafd800c260 con 0x7fafd8004800 2024-04-25T01:16:28.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.806+0000 7faff3e00640 1 --2- 172.21.15.90:0/1011736382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafd80051c0 0x7fafd800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:28.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.807+0000 7faff3400640 1 --2- 172.21.15.90:0/1011736382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafd80039b0 0x7fafd8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:28.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.807+0000 7faff3e00640 1 -- 172.21.15.90:0/1011736382 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fafd8004800 msgr2=0x7fafd8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:28.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.807+0000 7faff3e00640 1 --2- 172.21.15.90:0/1011736382 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fafd8004800 0x7fafd8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:28.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.807+0000 7faff3e00640 1 -- 172.21.15.90:0/1011736382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafd80039b0 msgr2=0x7fafd8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:28.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.807+0000 7faff3e00640 1 --2- 172.21.15.90:0/1011736382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafd80039b0 0x7fafd8003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:28.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.807+0000 7faff3e00640 1 -- 172.21.15.90:0/1011736382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd8003520 con 0x7fafd80051c0 2024-04-25T01:16:28.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.807+0000 7faff3e00640 1 -- 172.21.15.90:0/1011736382 reap_dead start 2024-04-25T01:16:28.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.807+0000 7faff3e00640 1 --2- 172.21.15.90:0/1011736382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafd80051c0 0x7fafd800b9c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fafec00a860 tx=0x7fafd8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1f198edf11a7074 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:28.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.808+0000 7faff2000640 1 -- 172.21.15.90:0/1011736382 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafec00a480 con 0x7fafd80051c0 2024-04-25T01:16:28.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.808+0000 7faff2000640 1 -- 172.21.15.90:0/1011736382 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafec02f070 con 0x7fafd80051c0 2024-04-25T01:16:28.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.808+0000 7faff2000640 1 -- 172.21.15.90:0/1011736382 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafec02bcc0 con 0x7fafd80051c0 2024-04-25T01:16:28.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.808+0000 7faff9200640 1 -- 172.21.15.90:0/1011736382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafd80051c0 msgr2=0x7fafd800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:28.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.808+0000 7faff9200640 1 --2- 172.21.15.90:0/1011736382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafd80051c0 0x7fafd800b9c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fafec00a860 tx=0x7fafd8005830 comp rx=0 tx=0).stop 2024-04-25T01:16:28.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.808+0000 7faff9200640 1 -- 172.21.15.90:0/1011736382 shutdown_connections 2024-04-25T01:16:28.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.808+0000 7faff9200640 1 --2- 172.21.15.90:0/1011736382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafd80051c0 0x7fafd800b9c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:28.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.808+0000 7faff9200640 1 -- 172.21.15.90:0/1011736382 >> 172.21.15.90:0/1011736382 conn(0x7faff409d070 msgr2=0x7faff40d3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:28.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.808+0000 7faff9200640 1 -- 172.21.15.90:0/1011736382 shutdown_connections 2024-04-25T01:16:28.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.808+0000 7faff9200640 1 -- 172.21.15.90:0/1011736382 wait complete. 2024-04-25T01:16:28.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.810+0000 7faff9200640 1 Processor -- start 2024-04-25T01:16:28.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.810+0000 7faff9200640 1 -- start start 2024-04-25T01:16:28.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.811+0000 7faff9200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafd80051c0 0x7faff41f3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:28.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.811+0000 7faff9200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faff41554e0 0x7faff4158580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:28.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.811+0000 7faff9200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faff4155940 0x7faff4158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:28.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.811+0000 7faff9200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7faff40b6bd0 con 0x7faff4155940 2024-04-25T01:16:28.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.811+0000 7faff9200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faff40d1950 con 0x7fafd80051c0 2024-04-25T01:16:28.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.811+0000 7faff9200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7faff40b6d50 con 0x7faff41554e0 2024-04-25T01:16:28.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.811+0000 7faff3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faff4155940 0x7faff4158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:28.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.811+0000 7faff3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faff4155940 0x7faff4158ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55020/0 (socket says 172.21.15.90:55020) 2024-04-25T01:16:28.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.811+0000 7faff3e00640 1 -- 172.21.15.90:0/2832687065 learned_addr learned my addr 172.21.15.90:0/2832687065 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:28.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.812+0000 7faff3400640 1 --2- 172.21.15.90:0/2832687065 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafd80051c0 0x7faff41f3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:28.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.812+0000 7faff2a00640 1 --2- 172.21.15.90:0/2832687065 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faff41554e0 0x7faff4158580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:28.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.812+0000 7faff3e00640 1 -- 172.21.15.90:0/2832687065 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faff41554e0 msgr2=0x7faff4158580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:28.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.812+0000 7faff3e00640 1 --2- 172.21.15.90:0/2832687065 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faff41554e0 0x7faff4158580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:28.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.812+0000 7faff3e00640 1 -- 172.21.15.90:0/2832687065 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafd80051c0 msgr2=0x7faff41f3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:28.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.812+0000 7faff3e00640 1 --2- 172.21.15.90:0/2832687065 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafd80051c0 0x7faff41f3f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:28.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.812+0000 7faff3e00640 1 -- 172.21.15.90:0/2832687065 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faff41e67e0 con 0x7faff4155940 2024-04-25T01:16:28.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.812+0000 7faff3e00640 1 --2- 172.21.15.90:0/2832687065 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faff4155940 0x7faff4158ac0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fafec00a980 tx=0x7faff40cf5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:28.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.812+0000 7fafeb400640 1 -- 172.21.15.90:0/2832687065 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafec044070 con 0x7faff4155940 2024-04-25T01:16:28.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.812+0000 7fafeb400640 1 -- 172.21.15.90:0/2832687065 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafec02bdc0 con 0x7faff4155940 2024-04-25T01:16:28.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.812+0000 7fafeb400640 1 -- 172.21.15.90:0/2832687065 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafec00a480 con 0x7faff4155940 2024-04-25T01:16:28.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.812+0000 7faff9200640 1 -- 172.21.15.90:0/2832687065 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faff41e6ac0 con 0x7faff4155940 2024-04-25T01:16:28.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.812+0000 7faff9200640 1 -- 172.21.15.90:0/2832687065 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faff41e7080 con 0x7faff4155940 2024-04-25T01:16:28.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.813+0000 7faff9200640 1 -- 172.21.15.90:0/2832687065 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faff4075350 con 0x7faff4155940 2024-04-25T01:16:28.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.814+0000 7fafeb400640 1 -- 172.21.15.90:0/2832687065 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fafec047070 con 0x7faff4155940 2024-04-25T01:16:28.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.815+0000 7fafeb400640 1 --2- 172.21.15.90:0/2832687065 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fafd006be30 0x7fafd006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:28.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.815+0000 7fafeb400640 1 -- 172.21.15.90:0/2832687065 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fafec0aade0 con 0x7faff4155940 2024-04-25T01:16:28.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.815+0000 7faff3400640 1 --2- 172.21.15.90:0/2832687065 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fafd006be30 0x7fafd006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:28.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.816+0000 7faff3400640 1 --2- 172.21.15.90:0/2832687065 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fafd006be30 0x7fafd006e2f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fafe4001510 tx=0x7fafe4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:28.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:28.820+0000 7fafeb400640 1 -- 172.21.15.90:0/2832687065 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fafec078d60 con 0x7faff4155940 2024-04-25T01:16:28.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.006+0000 7faff9200640 1 -- 172.21.15.90:0/2832687065 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faff40d0040 con 0x7faff4155940 2024-04-25T01:16:28.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.008+0000 7fafeb400640 1 -- 172.21.15.90:0/2832687065 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18482 (secure 0 0 0) 0x7fafec07d060 con 0x7faff4155940 2024-04-25T01:16:28.994 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:16:28.994 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":26,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:16:27.092855+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:16:20.094512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:16:21.078268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:16:24.670642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:16:28.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.011+0000 7faff9200640 1 -- 172.21.15.90:0/2832687065 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fafd006be30 msgr2=0x7fafd006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:28.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.011+0000 7faff9200640 1 --2- 172.21.15.90:0/2832687065 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fafd006be30 0x7fafd006e2f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fafe4001510 tx=0x7fafe4009040 comp rx=0 tx=0).stop 2024-04-25T01:16:28.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.011+0000 7faff9200640 1 -- 172.21.15.90:0/2832687065 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faff4155940 msgr2=0x7faff4158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:28.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.011+0000 7faff9200640 1 --2- 172.21.15.90:0/2832687065 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faff4155940 0x7faff4158ac0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fafec00a980 tx=0x7faff40cf5b0 comp rx=0 tx=0).stop 2024-04-25T01:16:28.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.011+0000 7faff9200640 1 -- 172.21.15.90:0/2832687065 shutdown_connections 2024-04-25T01:16:28.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.011+0000 7faff9200640 1 --2- 172.21.15.90:0/2832687065 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fafd006be30 0x7fafd006e2f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:28.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.011+0000 7faff9200640 1 --2- 172.21.15.90:0/2832687065 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faff4155940 0x7faff4158ac0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:28.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.011+0000 7faff9200640 1 --2- 172.21.15.90:0/2832687065 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faff41554e0 0x7faff4158580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:28.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.011+0000 7faff9200640 1 --2- 172.21.15.90:0/2832687065 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafd80051c0 0x7faff41f3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:28.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.011+0000 7faff9200640 1 -- 172.21.15.90:0/2832687065 >> 172.21.15.90:0/2832687065 conn(0x7faff409d070 msgr2=0x7faff4157400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:28.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.011+0000 7faff9200640 1 -- 172.21.15.90:0/2832687065 shutdown_connections 2024-04-25T01:16:28.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.011+0000 7faff9200640 1 -- 172.21.15.90:0/2832687065 wait complete. 2024-04-25T01:16:29.029 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:16:29.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.240+0000 7f401e000640 1 Processor -- start 2024-04-25T01:16:29.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.240+0000 7f401e000640 1 -- start start 2024-04-25T01:16:29.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.241+0000 7f401e000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f401806b120 0x7f401806b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:29.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.241+0000 7f401e000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f40180ae730 0x7f40180cd900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:29.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.241+0000 7f401e000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f401806ba40 con 0x7f401815ca10 2024-04-25T01:16:29.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.241+0000 7f401e000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f40180cef40 con 0x7f401806b120 2024-04-25T01:16:29.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.241+0000 7f401e000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f401815c580 con 0x7f40180ae730 2024-04-25T01:16:29.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.241+0000 7f401cc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f401806b120 0x7f401806b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:29.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.241+0000 7f401d600640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f401815ca10 0x7f40180cc030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:46914/0 (socket says 172.21.15.90:46914) 2024-04-25T01:16:29.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.241+0000 7f401d600640 1 -- 172.21.15.90:0/723126852 learned_addr learned my addr 172.21.15.90:0/723126852 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:29.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.242+0000 7f400fe00640 1 --2- 172.21.15.90:0/723126852 >> v2:172.21.15.103:3300/0 conn(0x7f40180ae730 0x7f40180cd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:29.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.242+0000 7f401cc00640 1 -- 172.21.15.90:0/723126852 >> v2:172.21.15.103:3300/0 conn(0x7f40180ae730 msgr2=0x7f40180cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:29.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.242+0000 7f401cc00640 1 --2- 172.21.15.90:0/723126852 >> v2:172.21.15.103:3300/0 conn(0x7f40180ae730 0x7f40180cd900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:29.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.242+0000 7f401cc00640 1 -- 172.21.15.90:0/723126852 >> v1:172.21.15.90:6789/0 conn(0x7f401815ca10 legacy=0x7f40180cc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:29.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.242+0000 7f401cc00640 1 -- 172.21.15.90:0/723126852 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f401815c220 con 0x7f401806b120 2024-04-25T01:16:29.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.242+0000 7f4017e00640 1 -- 172.21.15.90:0/723126852 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1880085809 0 0) 0x7f401806ba40 con 0x7f401815ca10 2024-04-25T01:16:29.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.242+0000 7f400fe00640 1 --2- 172.21.15.90:0/723126852 >> v2:172.21.15.103:3300/0 conn(0x7f40180ae730 0x7f40180cd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:29.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.243+0000 7f401cc00640 1 --2- 172.21.15.90:0/723126852 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f401806b120 0x7f401806b500 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f40180b2680 tx=0x7f400802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=738c1b8b989a1797 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:29.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.243+0000 7f4017e00640 1 -- 172.21.15.90:0/723126852 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4008037070 con 0x7f401806b120 2024-04-25T01:16:29.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.243+0000 7f4017e00640 1 -- 172.21.15.90:0/723126852 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4008032440 con 0x7f401806b120 2024-04-25T01:16:29.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.243+0000 7f4017e00640 1 -- 172.21.15.90:0/723126852 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4008032810 con 0x7f401806b120 2024-04-25T01:16:29.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.244+0000 7f401e000640 1 -- 172.21.15.90:0/723126852 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f401806b120 msgr2=0x7f401806b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:29.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.244+0000 7f401e000640 1 --2- 172.21.15.90:0/723126852 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f401806b120 0x7f401806b500 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f40180b2680 tx=0x7f400802a1a0 comp rx=0 tx=0).stop 2024-04-25T01:16:29.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.244+0000 7f401e000640 1 -- 172.21.15.90:0/723126852 shutdown_connections 2024-04-25T01:16:29.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.244+0000 7f401e000640 1 --2- 172.21.15.90:0/723126852 >> v2:172.21.15.103:3300/0 conn(0x7f40180ae730 0x7f40180cd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:29.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.244+0000 7f401e000640 1 --2- 172.21.15.90:0/723126852 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f401806b120 0x7f401806b500 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:29.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.244+0000 7f401e000640 1 -- 172.21.15.90:0/723126852 >> 172.21.15.90:0/723126852 conn(0x7f40180b1660 msgr2=0x7f40180b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:29.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.244+0000 7f401e000640 1 -- 172.21.15.90:0/723126852 shutdown_connections 2024-04-25T01:16:29.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.244+0000 7f401e000640 1 -- 172.21.15.90:0/723126852 wait complete. 2024-04-25T01:16:29.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.246+0000 7f401e000640 1 Processor -- start 2024-04-25T01:16:29.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.246+0000 7f401e000640 1 -- start start 2024-04-25T01:16:29.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.247+0000 7f401e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f401806b120 0x7f4018145360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:29.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.247+0000 7f401e000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40180ae730 0x7f40180ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:29.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.247+0000 7f401e000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f401815ca10 0x7f40180ade20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:29.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.247+0000 7f401e000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f40180af240 con 0x7f401806b120 2024-04-25T01:16:29.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.247+0000 7f401e000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f40180aef60 con 0x7f401815ca10 2024-04-25T01:16:29.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.247+0000 7f401e000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4018153080 con 0x7f40180ae730 2024-04-25T01:16:29.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.247+0000 7f401cc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f401806b120 0x7f4018145360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:29.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.247+0000 7f401cc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f401806b120 0x7f4018145360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55030/0 (socket says 172.21.15.90:55030) 2024-04-25T01:16:29.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.247+0000 7f401cc00640 1 -- 172.21.15.90:0/3822214314 learned_addr learned my addr 172.21.15.90:0/3822214314 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:29.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.247+0000 7f401cc00640 1 -- 172.21.15.90:0/3822214314 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40180ae730 msgr2=0x7f40180ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:29.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.247+0000 7f401cc00640 1 --2- 172.21.15.90:0/3822214314 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40180ae730 0x7f40180ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:29.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.247+0000 7f401cc00640 1 -- 172.21.15.90:0/3822214314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f401815ca10 msgr2=0x7f40180ade20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:29.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.247+0000 7f401cc00640 1 --2- 172.21.15.90:0/3822214314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f401815ca10 0x7f40180ade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:29.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.247+0000 7f401cc00640 1 -- 172.21.15.90:0/3822214314 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f401814c060 con 0x7f401806b120 2024-04-25T01:16:29.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.248+0000 7f401cc00640 1 --2- 172.21.15.90:0/3822214314 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f401806b120 0x7f4018145360 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4008009890 tx=0x7f40080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:29.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.248+0000 7f4015600640 1 -- 172.21.15.90:0/3822214314 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4008037040 con 0x7f401806b120 2024-04-25T01:16:29.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.248+0000 7f4015600640 1 -- 172.21.15.90:0/3822214314 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4008042070 con 0x7f401806b120 2024-04-25T01:16:29.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.248+0000 7f4015600640 1 -- 172.21.15.90:0/3822214314 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4008030a50 con 0x7f401806b120 2024-04-25T01:16:29.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.248+0000 7f401e000640 1 -- 172.21.15.90:0/3822214314 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4018147d40 con 0x7f401806b120 2024-04-25T01:16:29.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.248+0000 7f401e000640 1 -- 172.21.15.90:0/3822214314 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4018148300 con 0x7f401806b120 2024-04-25T01:16:29.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.249+0000 7f401e000640 1 -- 172.21.15.90:0/3822214314 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4018075350 con 0x7f401806b120 2024-04-25T01:16:29.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.250+0000 7f4015600640 1 -- 172.21.15.90:0/3822214314 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4008030bf0 con 0x7f401806b120 2024-04-25T01:16:29.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.251+0000 7f4015600640 1 --2- 172.21.15.90:0/3822214314 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f3ff406be30 0x7f3ff406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:29.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.252+0000 7f4015600640 1 -- 172.21.15.90:0/3822214314 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f40080abc60 con 0x7f401806b120 2024-04-25T01:16:29.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.252+0000 7f400fe00640 1 --2- 172.21.15.90:0/3822214314 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f3ff406be30 0x7f3ff406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:29.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.252+0000 7f400fe00640 1 --2- 172.21.15.90:0/3822214314 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f3ff406be30 0x7f3ff406e2f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4000006fd0 tx=0x7f4000008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:29.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.256+0000 7f4015600640 1 -- 172.21.15.90:0/3822214314 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f400807a2c0 con 0x7f401806b120 2024-04-25T01:16:29.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.479+0000 7f401e000640 1 -- 172.21.15.90:0/3822214314 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f40180726f0 con 0x7f401806b120 2024-04-25T01:16:29.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.480+0000 7f4015600640 1 -- 172.21.15.90:0/3822214314 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5180 (secure 0 0 0) 0x7f4008079bc0 con 0x7f401806b120 2024-04-25T01:16:29.466 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:16:29.466 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6834/2605945289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":2605945289},{"type":"v1","addr":"172.21.15.103:6834","nonce":2605945289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6836/704639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":704639426},{"type":"v1","addr":"172.21.15.90:6836","nonce":704639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4245,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3477867755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6835","nonce":3477867755},{"type":"v1","addr":"172.21.15.90:6837","nonce":3477867755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:16:22.060018+0000","modified":"2024-04-25T01:16:27.093210+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6835/4291578198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6833","nonce":4291578198},{"type":"v1","addr":"172.21.15.103:6835","nonce":4291578198}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1}]} 2024-04-25T01:16:29.466 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 7 2024-04-25T01:16:29.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.483+0000 7f401e000640 1 -- 172.21.15.90:0/3822214314 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f3ff406be30 msgr2=0x7f3ff406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:29.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.483+0000 7f401e000640 1 --2- 172.21.15.90:0/3822214314 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f3ff406be30 0x7f3ff406e2f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4000006fd0 tx=0x7f4000008040 comp rx=0 tx=0).stop 2024-04-25T01:16:29.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.483+0000 7f401e000640 1 -- 172.21.15.90:0/3822214314 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f401806b120 msgr2=0x7f4018145360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:29.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.483+0000 7f401e000640 1 --2- 172.21.15.90:0/3822214314 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f401806b120 0x7f4018145360 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4008009890 tx=0x7f40080046c0 comp rx=0 tx=0).stop 2024-04-25T01:16:29.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.483+0000 7f401e000640 1 -- 172.21.15.90:0/3822214314 shutdown_connections 2024-04-25T01:16:29.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.483+0000 7f401e000640 1 --2- 172.21.15.90:0/3822214314 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f3ff406be30 0x7f3ff406e2f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:29.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.483+0000 7f401e000640 1 --2- 172.21.15.90:0/3822214314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f401815ca10 0x7f40180ade20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:29.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.483+0000 7f401e000640 1 --2- 172.21.15.90:0/3822214314 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40180ae730 0x7f40180ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:29.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.483+0000 7f401e000640 1 --2- 172.21.15.90:0/3822214314 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f401806b120 0x7f4018145360 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:29.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.483+0000 7f401e000640 1 -- 172.21.15.90:0/3822214314 >> 172.21.15.90:0/3822214314 conn(0x7f40180b1660 msgr2=0x7f401806e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:29.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.483+0000 7f401e000640 1 -- 172.21.15.90:0/3822214314 shutdown_connections 2024-04-25T01:16:29.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.483+0000 7f401e000640 1 -- 172.21.15.90:0/3822214314 wait complete. 2024-04-25T01:16:29.488 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:16:29.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.700+0000 7fdddca00640 1 Processor -- start 2024-04-25T01:16:29.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.700+0000 7fdddca00640 1 -- start start 2024-04-25T01:16:29.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.702+0000 7fdddca00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fddd8161760 0x7fddd80d2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:29.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.702+0000 7fdddca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fddd80b5340 0x7fddd80b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:29.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.702+0000 7fdddca00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddd80cebb0 con 0x7fddd80d0850 2024-04-25T01:16:29.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.702+0000 7fdddca00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fddd80b57a0 con 0x7fddd8161760 2024-04-25T01:16:29.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.702+0000 7fdddca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fddd80d46d0 con 0x7fddd80b5340 2024-04-25T01:16:29.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.702+0000 7fddd7400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fddd8161760 0x7fddd80d2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:29.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.702+0000 7fddd7e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fddd80d0850 0x7fddd80cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:46920/0 (socket says 172.21.15.90:46920) 2024-04-25T01:16:29.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.702+0000 7fddd7400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fddd8161760 0x7fddd80d2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55036/0 (socket says 172.21.15.90:55036) 2024-04-25T01:16:29.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.702+0000 7fddd7e00640 1 -- 172.21.15.90:0/2745068573 learned_addr learned my addr 172.21.15.90:0/2745068573 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:29.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.703+0000 7fddd6a00640 1 --2- 172.21.15.90:0/2745068573 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fddd80b5340 0x7fddd80b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:29.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.703+0000 7fddd7400640 1 -- 172.21.15.90:0/2745068573 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fddd80b5340 msgr2=0x7fddd80b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:29.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.703+0000 7fddd7400640 1 --2- 172.21.15.90:0/2745068573 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fddd80b5340 0x7fddd80b3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:29.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.703+0000 7fddd7400640 1 -- 172.21.15.90:0/2745068573 >> v1:172.21.15.90:6789/0 conn(0x7fddd80d0850 legacy=0x7fddd80cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:29.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.703+0000 7fddd7400640 1 -- 172.21.15.90:0/2745068573 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddd80d49b0 con 0x7fddd8161760 2024-04-25T01:16:29.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.703+0000 7fddd6000640 1 -- 172.21.15.90:0/2745068573 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2988204765 0 0) 0x7fddd80cebb0 con 0x7fddd80d0850 2024-04-25T01:16:29.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.703+0000 7fddd6a00640 1 --2- 172.21.15.90:0/2745068573 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fddd80b5340 0x7fddd80b3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:29.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.704+0000 7fddd7400640 1 --2- 172.21.15.90:0/2745068573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fddd8161760 0x7fddd80d2b10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fddc80097a0 tx=0x7fddc802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bde2d77777d73f02 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:29.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.704+0000 7fddd6000640 1 -- 172.21.15.90:0/2745068573 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc8037070 con 0x7fddd8161760 2024-04-25T01:16:29.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.704+0000 7fddd6000640 1 -- 172.21.15.90:0/2745068573 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddc802ae50 con 0x7fddd8161760 2024-04-25T01:16:29.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.704+0000 7fddd6000640 1 -- 172.21.15.90:0/2745068573 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc8032660 con 0x7fddd8161760 2024-04-25T01:16:29.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.704+0000 7fdddca00640 1 -- 172.21.15.90:0/2745068573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fddd8161760 msgr2=0x7fddd80d2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:29.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.704+0000 7fdddca00640 1 --2- 172.21.15.90:0/2745068573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fddd8161760 0x7fddd80d2b10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fddc80097a0 tx=0x7fddc802a100 comp rx=0 tx=0).stop 2024-04-25T01:16:29.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.705+0000 7fdddca00640 1 -- 172.21.15.90:0/2745068573 shutdown_connections 2024-04-25T01:16:29.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.705+0000 7fdddca00640 1 --2- 172.21.15.90:0/2745068573 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fddd80b5340 0x7fddd80b3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:29.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.705+0000 7fdddca00640 1 --2- 172.21.15.90:0/2745068573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fddd8161760 0x7fddd80d2b10 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:29.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.705+0000 7fdddca00640 1 -- 172.21.15.90:0/2745068573 >> 172.21.15.90:0/2745068573 conn(0x7fddd809d170 msgr2=0x7fddd80d3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:29.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.705+0000 7fdddca00640 1 -- 172.21.15.90:0/2745068573 shutdown_connections 2024-04-25T01:16:29.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.705+0000 7fdddca00640 1 -- 172.21.15.90:0/2745068573 wait complete. 2024-04-25T01:16:29.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.706+0000 7fdddca00640 1 Processor -- start 2024-04-25T01:16:29.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.707+0000 7fdddca00640 1 -- start start 2024-04-25T01:16:29.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.707+0000 7fdddca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fddd80b5340 0x7fddd815c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:29.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.707+0000 7fdddca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fddd80d0850 0x7fddd80cc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:29.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.708+0000 7fdddca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddd8161760 0x7fddd815b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:29.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.708+0000 7fdddca00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fddd8161b40 con 0x7fddd80b5340 2024-04-25T01:16:29.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.708+0000 7fdddca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fddd80d4c90 con 0x7fddd8161760 2024-04-25T01:16:29.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.708+0000 7fdddca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fddd80cb9e0 con 0x7fddd80d0850 2024-04-25T01:16:29.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.708+0000 7fddd7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fddd80b5340 0x7fddd815c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:29.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.708+0000 7fddd7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fddd80b5340 0x7fddd815c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55048/0 (socket says 172.21.15.90:55048) 2024-04-25T01:16:29.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.708+0000 7fddd7400640 1 -- 172.21.15.90:0/2657439182 learned_addr learned my addr 172.21.15.90:0/2657439182 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:29.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.708+0000 7fddd7400640 1 -- 172.21.15.90:0/2657439182 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fddd80d0850 msgr2=0x7fddd80cc7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:29.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.708+0000 7fddd7400640 1 --2- 172.21.15.90:0/2657439182 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fddd80d0850 0x7fddd80cc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:29.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.708+0000 7fddd7400640 1 -- 172.21.15.90:0/2657439182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddd8161760 msgr2=0x7fddd815b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:29.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.708+0000 7fddd7400640 1 --2- 172.21.15.90:0/2657439182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddd8161760 0x7fddd815b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:29.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.708+0000 7fddd7400640 1 -- 172.21.15.90:0/2657439182 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddd80cb0b0 con 0x7fddd80b5340 2024-04-25T01:16:29.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.708+0000 7fddd7400640 1 --2- 172.21.15.90:0/2657439182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fddd80b5340 0x7fddd815c0c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fddc8035040 tx=0x7fddc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:29.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.709+0000 7fddcf400640 1 -- 172.21.15.90:0/2657439182 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc8041070 con 0x7fddd80b5340 2024-04-25T01:16:29.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.709+0000 7fddcf400640 1 -- 172.21.15.90:0/2657439182 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddc803c670 con 0x7fddd80b5340 2024-04-25T01:16:29.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.709+0000 7fdddca00640 1 -- 172.21.15.90:0/2657439182 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddd80cb390 con 0x7fddd80b5340 2024-04-25T01:16:29.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.709+0000 7fddcf400640 1 -- 172.21.15.90:0/2657439182 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc8037040 con 0x7fddd80b5340 2024-04-25T01:16:29.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.709+0000 7fdddca00640 1 -- 172.21.15.90:0/2657439182 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddd80cbe70 con 0x7fddd80b5340 2024-04-25T01:16:29.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.710+0000 7fdddca00640 1 -- 172.21.15.90:0/2657439182 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddd8075350 con 0x7fddd80b5340 2024-04-25T01:16:29.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.711+0000 7fddcf400640 1 -- 172.21.15.90:0/2657439182 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fddc803a070 con 0x7fddd80b5340 2024-04-25T01:16:29.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.712+0000 7fddcf400640 1 --2- 172.21.15.90:0/2657439182 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fddb406be30 0x7fddb406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:29.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.712+0000 7fddcf400640 1 -- 172.21.15.90:0/2657439182 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fddc80ab4c0 con 0x7fddd80b5340 2024-04-25T01:16:29.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.712+0000 7fddd6a00640 1 --2- 172.21.15.90:0/2657439182 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fddb406be30 0x7fddb406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:29.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.713+0000 7fddd6a00640 1 --2- 172.21.15.90:0/2657439182 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fddb406be30 0x7fddb406e2f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fddc0006fd0 tx=0x7fddc0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:29.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.716+0000 7fddcf400640 1 -- 172.21.15.90:0/2657439182 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fddc8079a90 con 0x7fddd80b5340 2024-04-25T01:16:29.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.916+0000 7fdddca00640 1 -- 172.21.15.90:0/2657439182 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fddd80678f0 con 0x7fddd80b5340 2024-04-25T01:16:29.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.919+0000 7fddcf400640 1 -- 172.21.15.90:0/2657439182 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18482 (secure 0 0 0) 0x7fddc8079390 con 0x7fddd80b5340 2024-04-25T01:16:29.905 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:16:29.905 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":26,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:16:27.092855+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:16:20.094512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:16:21.078268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:16:24.670642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:16:29.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.922+0000 7fdddca00640 1 -- 172.21.15.90:0/2657439182 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fddb406be30 msgr2=0x7fddb406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:29.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.922+0000 7fdddca00640 1 --2- 172.21.15.90:0/2657439182 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fddb406be30 0x7fddb406e2f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fddc0006fd0 tx=0x7fddc0008040 comp rx=0 tx=0).stop 2024-04-25T01:16:29.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.922+0000 7fdddca00640 1 -- 172.21.15.90:0/2657439182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fddd80b5340 msgr2=0x7fddd815c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:29.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.922+0000 7fdddca00640 1 --2- 172.21.15.90:0/2657439182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fddd80b5340 0x7fddd815c0c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fddc8035040 tx=0x7fddc80046c0 comp rx=0 tx=0).stop 2024-04-25T01:16:29.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.922+0000 7fdddca00640 1 -- 172.21.15.90:0/2657439182 shutdown_connections 2024-04-25T01:16:29.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.922+0000 7fdddca00640 1 --2- 172.21.15.90:0/2657439182 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fddb406be30 0x7fddb406e2f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:29.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.922+0000 7fdddca00640 1 --2- 172.21.15.90:0/2657439182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddd8161760 0x7fddd815b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:29.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.922+0000 7fdddca00640 1 --2- 172.21.15.90:0/2657439182 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fddd80d0850 0x7fddd80cc7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:29.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.922+0000 7fdddca00640 1 --2- 172.21.15.90:0/2657439182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fddd80b5340 0x7fddd815c0c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:29.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.922+0000 7fdddca00640 1 -- 172.21.15.90:0/2657439182 >> 172.21.15.90:0/2657439182 conn(0x7fddd809d170 msgr2=0x7fddd80d3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:29.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.922+0000 7fdddca00640 1 -- 172.21.15.90:0/2657439182 shutdown_connections 2024-04-25T01:16:29.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:29.922+0000 7fdddca00640 1 -- 172.21.15.90:0/2657439182 wait complete. 2024-04-25T01:16:30.328 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T01:16:22.060018+0000', 'modified': '2024-04-25T01:16:27.093210+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4254}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.103:6835/4291578198', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6833', 'nonce': 4291578198}, {'type': 'v1', 'addr': '172.21.15.103:6835', 'nonce': 4291578198}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4254, 'qdb_cluster': [4254]} 2024-04-25T01:16:30.328 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T01:16:30.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-25T01:16:30.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.552+0000 7f4073e00640 1 Processor -- start 2024-04-25T01:16:30.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.552+0000 7f4073e00640 1 -- start start 2024-04-25T01:16:30.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.554+0000 7f4073e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f407415ad40 0x7f407406b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.554+0000 7f4073e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f407406d910 0x7f40740cd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.554+0000 7f4073e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f407406ba90 con 0x7f40740cf520 2024-04-25T01:16:30.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.554+0000 7f4073e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f407406cb50 con 0x7f407415ad40 2024-04-25T01:16:30.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.554+0000 7f4073e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f407406ce30 con 0x7f407406d910 2024-04-25T01:16:30.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.555+0000 7f4072a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f407415ad40 0x7f407406b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:30.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.555+0000 7f4072a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f407415ad40 0x7f407406b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55060/0 (socket says 172.21.15.90:55060) 2024-04-25T01:16:30.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.555+0000 7f4072a00640 1 -- 172.21.15.90:0/3775083338 learned_addr learned my addr 172.21.15.90:0/3775083338 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:30.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.555+0000 7f4073400640 1 --2- 172.21.15.90:0/3775083338 >> v2:172.21.15.103:3300/0 conn(0x7f407406d910 0x7f40740cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:30.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.555+0000 7f4072a00640 1 -- 172.21.15.90:0/3775083338 >> v2:172.21.15.103:3300/0 conn(0x7f407406d910 msgr2=0x7f40740cd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:30.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.555+0000 7f4072a00640 1 --2- 172.21.15.90:0/3775083338 >> v2:172.21.15.103:3300/0 conn(0x7f407406d910 0x7f40740cd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.555+0000 7f4072a00640 1 -- 172.21.15.90:0/3775083338 >> v1:172.21.15.103:6789/0 conn(0x7f40740cf520 legacy=0x7f40740ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:30.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.555+0000 7f4072a00640 1 -- 172.21.15.90:0/3775083338 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f407406d110 con 0x7f407415ad40 2024-04-25T01:16:30.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.557+0000 7f4072a00640 1 --2- 172.21.15.90:0/3775083338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f407415ad40 0x7f407406b120 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f40740b3f30 tx=0x7f405c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c62ae76b3cdf3e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:30.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.557+0000 7f4071600640 1 -- 172.21.15.90:0/3775083338 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c037070 con 0x7f407415ad40 2024-04-25T01:16:30.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.557+0000 7f4071600640 1 -- 172.21.15.90:0/3775083338 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f405c032440 con 0x7f407415ad40 2024-04-25T01:16:30.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.557+0000 7f4071600640 1 -- 172.21.15.90:0/3775083338 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c032810 con 0x7f407415ad40 2024-04-25T01:16:30.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.558+0000 7f4073e00640 1 -- 172.21.15.90:0/3775083338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f407415ad40 msgr2=0x7f407406b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:30.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.558+0000 7f4073e00640 1 --2- 172.21.15.90:0/3775083338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f407415ad40 0x7f407406b120 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f40740b3f30 tx=0x7f405c02a230 comp rx=0 tx=0).stop 2024-04-25T01:16:30.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.558+0000 7f4073e00640 1 -- 172.21.15.90:0/3775083338 shutdown_connections 2024-04-25T01:16:30.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.558+0000 7f4073e00640 1 --2- 172.21.15.90:0/3775083338 >> v2:172.21.15.103:3300/0 conn(0x7f407406d910 0x7f40740cd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.558+0000 7f4073e00640 1 --2- 172.21.15.90:0/3775083338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f407415ad40 0x7f407406b120 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.558+0000 7f4073e00640 1 -- 172.21.15.90:0/3775083338 >> 172.21.15.90:0/3775083338 conn(0x7f40740b1660 msgr2=0x7f40740b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:30.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.558+0000 7f4073e00640 1 -- 172.21.15.90:0/3775083338 shutdown_connections 2024-04-25T01:16:30.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.558+0000 7f4073e00640 1 -- 172.21.15.90:0/3775083338 wait complete. 2024-04-25T01:16:30.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.560+0000 7f4073e00640 1 Processor -- start 2024-04-25T01:16:30.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.560+0000 7f4073e00640 1 -- start start 2024-04-25T01:16:30.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.561+0000 7f4073e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f407406d910 0x7f407415d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.561+0000 7f4073e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40740cf520 0x7f4074144770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.561+0000 7f4073e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f407415ad40 0x7f407415b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.561+0000 7f4073e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f407406b7e0 con 0x7f407406d910 2024-04-25T01:16:30.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.561+0000 7f4073e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f407406b660 con 0x7f407415ad40 2024-04-25T01:16:30.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.561+0000 7f4073e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f40740aee40 con 0x7f40740cf520 2024-04-25T01:16:30.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.561+0000 7f4072a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f407406d910 0x7f407415d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:30.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.561+0000 7f4072a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f407406d910 0x7f407415d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55064/0 (socket says 172.21.15.90:55064) 2024-04-25T01:16:30.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.561+0000 7f4072a00640 1 -- 172.21.15.90:0/1408267293 learned_addr learned my addr 172.21.15.90:0/1408267293 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:30.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.561+0000 7f4072a00640 1 -- 172.21.15.90:0/1408267293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40740cf520 msgr2=0x7f4074144770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:30.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.561+0000 7f4072a00640 1 --2- 172.21.15.90:0/1408267293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40740cf520 0x7f4074144770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.561+0000 7f4072a00640 1 -- 172.21.15.90:0/1408267293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f407415ad40 msgr2=0x7f407415b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:30.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.561+0000 7f4073400640 1 --2- 172.21.15.90:0/1408267293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f407415ad40 0x7f407415b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:30.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.561+0000 7f4072a00640 1 --2- 172.21.15.90:0/1408267293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f407415ad40 0x7f407415b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.561+0000 7f4072a00640 1 -- 172.21.15.90:0/1408267293 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f407415b890 con 0x7f407406d910 2024-04-25T01:16:30.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.562+0000 7f4072a00640 1 --2- 172.21.15.90:0/1408267293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f407406d910 0x7f407415d9d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f405c000c00 tx=0x7f405c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:30.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.562+0000 7f406aa00640 1 -- 172.21.15.90:0/1408267293 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c041020 con 0x7f407406d910 2024-04-25T01:16:30.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.562+0000 7f406aa00640 1 -- 172.21.15.90:0/1408267293 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f405c0306c0 con 0x7f407406d910 2024-04-25T01:16:30.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.562+0000 7f406aa00640 1 -- 172.21.15.90:0/1408267293 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c037040 con 0x7f407406d910 2024-04-25T01:16:30.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.562+0000 7f4073e00640 1 -- 172.21.15.90:0/1408267293 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f407415bb70 con 0x7f407406d910 2024-04-25T01:16:30.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.562+0000 7f4073e00640 1 -- 172.21.15.90:0/1408267293 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f407415c130 con 0x7f407406d910 2024-04-25T01:16:30.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.564+0000 7f406aa00640 1 -- 172.21.15.90:0/1408267293 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f405c044020 con 0x7f407406d910 2024-04-25T01:16:30.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.565+0000 7f406aa00640 1 --2- 172.21.15.90:0/1408267293 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f405006be30 0x7f405006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.565+0000 7f406aa00640 1 -- 172.21.15.90:0/1408267293 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f405c0ab9a0 con 0x7f407406d910 2024-04-25T01:16:30.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.565+0000 7f4072000640 1 --2- 172.21.15.90:0/1408267293 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f405006be30 0x7f405006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:30.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.566+0000 7f407c872740 1 Processor -- start 2024-04-25T01:16:30.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.566+0000 7f4072000640 1 --2- 172.21.15.90:0/1408267293 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f405006be30 0x7f405006e2f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f40741684c0 tx=0x7f4064007d50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:30.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.566+0000 7f407c872740 1 -- start start 2024-04-25T01:16:30.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.567+0000 7f407c872740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d93300a720 0x55d93300ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.567+0000 7f407c872740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d933027fc0 0x55d933028510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.567+0000 7f407c872740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d932fe0400 0x55d933026680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.567+0000 7f407c872740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d932edfed0 con 0x55d93300a720 2024-04-25T01:16:30.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.567+0000 7f407c872740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d932f42df0 con 0x55d932fe0400 2024-04-25T01:16:30.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.567+0000 7f407c872740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d932eeb8d0 con 0x55d933027fc0 2024-04-25T01:16:30.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.567+0000 7f4072a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d93300a720 0x55d93300ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:30.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.567+0000 7f4072a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d93300a720 0x55d93300ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55068/0 (socket says 172.21.15.90:55068) 2024-04-25T01:16:30.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.567+0000 7f4072a00640 1 -- 172.21.15.90:0/643537922 learned_addr learned my addr 172.21.15.90:0/643537922 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:30.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.568+0000 7f4072a00640 1 -- 172.21.15.90:0/643537922 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d933027fc0 msgr2=0x55d933028510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:30.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.568+0000 7f4072a00640 1 --2- 172.21.15.90:0/643537922 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d933027fc0 0x55d933028510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.568+0000 7f4072a00640 1 -- 172.21.15.90:0/643537922 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d932fe0400 msgr2=0x55d933026680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:30.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.568+0000 7f4072a00640 1 --2- 172.21.15.90:0/643537922 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d932fe0400 0x55d933026680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.568+0000 7f4072a00640 1 -- 172.21.15.90:0/643537922 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d932c572d0 con 0x55d93300a720 2024-04-25T01:16:30.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.568+0000 7f4072a00640 1 --2- 172.21.15.90:0/643537922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d93300a720 0x55d93300ab00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f405c035040 tx=0x7f405c075ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98b0bec294b713e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:30.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.568+0000 7f4063400640 1 -- 172.21.15.90:0/643537922 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c07dcc0 con 0x55d93300a720 2024-04-25T01:16:30.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.568+0000 7f4063400640 1 -- 172.21.15.90:0/643537922 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f405c07ee50 con 0x55d93300a720 2024-04-25T01:16:30.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.568+0000 7f4063400640 1 -- 172.21.15.90:0/643537922 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c0824f0 con 0x55d93300a720 2024-04-25T01:16:30.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.569+0000 7f407c872740 1 -- 172.21.15.90:0/643537922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d93300a720 msgr2=0x55d93300ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:30.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.569+0000 7f407c872740 1 --2- 172.21.15.90:0/643537922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d93300a720 0x55d93300ab00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f405c035040 tx=0x7f405c075ce0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.569+0000 7f407c872740 1 -- 172.21.15.90:0/643537922 shutdown_connections 2024-04-25T01:16:30.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.569+0000 7f407c872740 1 --2- 172.21.15.90:0/643537922 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d932fe0400 0x55d933026680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.569+0000 7f407c872740 1 --2- 172.21.15.90:0/643537922 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d933027fc0 0x55d933028510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.569+0000 7f407c872740 1 --2- 172.21.15.90:0/643537922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d93300a720 0x55d93300ab00 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.569+0000 7f407c872740 1 -- 172.21.15.90:0/643537922 >> 172.21.15.90:0/643537922 conn(0x55d932fdcd30 msgr2=0x55d933025530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:30.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.569+0000 7f407c872740 1 -- 172.21.15.90:0/643537922 shutdown_connections 2024-04-25T01:16:30.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.569+0000 7f407c872740 1 -- 172.21.15.90:0/643537922 wait complete. 2024-04-25T01:16:30.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.570+0000 7f407c872740 1 -- start start 2024-04-25T01:16:30.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.570+0000 7f407c872740 1 Processor -- start 2024-04-25T01:16:30.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.571+0000 7f407c872740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d93300a720 0x55d93301c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.571+0000 7f407c872740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d933027fc0 0x55d93301d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.571+0000 7f407c872740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d9330062f0 0x55d932fd9890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.571+0000 7f407c872740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d932efed20 con 0x55d93300a720 2024-04-25T01:16:30.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.571+0000 7f407c872740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d932f3edb0 con 0x55d9330062f0 2024-04-25T01:16:30.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.571+0000 7f407c872740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d932edfed0 con 0x55d933027fc0 2024-04-25T01:16:30.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.571+0000 7f407c872740 10 client.0 _ll_register_callbacks cb 0x55d932f39f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:16:30.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.571+0000 7f4072a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d93300a720 0x55d93301c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:30.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.571+0000 7f4072a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d93300a720 0x55d93301c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55076/0 (socket says 172.21.15.90:55076) 2024-04-25T01:16:30.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.571+0000 7f4072a00640 1 -- 172.21.15.90:0/2176406875 learned_addr learned my addr 172.21.15.90:0/2176406875 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:30.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.571+0000 7f4072a00640 1 -- 172.21.15.90:0/2176406875 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d933027fc0 msgr2=0x55d93301d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:30.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.571+0000 7f4072a00640 1 --2- 172.21.15.90:0/2176406875 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d933027fc0 0x55d93301d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.571+0000 7f4072a00640 1 -- 172.21.15.90:0/2176406875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d9330062f0 msgr2=0x55d932fd9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:30.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.571+0000 7f4072a00640 1 --2- 172.21.15.90:0/2176406875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d9330062f0 0x55d932fd9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.571+0000 7f4072a00640 1 -- 172.21.15.90:0/2176406875 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d932d216b0 con 0x55d93300a720 2024-04-25T01:16:30.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.571+0000 7f4072a00640 1 --2- 172.21.15.90:0/2176406875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d93300a720 0x55d93301c5e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f405c0a9be0 tx=0x7f405c0ab000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d836c2328564cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:30.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.571+0000 7f4057e00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T01:16:30.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.572+0000 7f4057e00640 1 -- 172.21.15.90:0/2176406875 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c0a9070 con 0x55d93300a720 2024-04-25T01:16:30.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.572+0000 7f4057e00640 1 -- 172.21.15.90:0/2176406875 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f405c082060 con 0x55d93300a720 2024-04-25T01:16:30.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.572+0000 7f407c872740 1 -- 172.21.15.90:0/2176406875 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f405c07ee50 con 0x55d93300a720 2024-04-25T01:16:30.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.572+0000 7f4057e00640 1 -- 172.21.15.90:0/2176406875 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7f405c0824f0 con 0x55d93300a720 2024-04-25T01:16:30.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.572+0000 7f407c872740 10 client.4401 fetch_fsmap learned FSMap version 7 2024-04-25T01:16:30.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.572+0000 7f407c872740 1 -- 172.21.15.90:0/2176406875 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x55d9330393f0 con 0x55d93300a720 2024-04-25T01:16:30.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.572+0000 7f4057e00640 1 -- 172.21.15.90:0/2176406875 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f405c07ee50 con 0x55d93300a720 2024-04-25T01:16:30.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.572+0000 7f4057e00640 1 -- 172.21.15.90:0/2176406875 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 7) ==== 3502+0+0 (secure 0 0 0) 0x7f405c03a030 con 0x55d93300a720 2024-04-25T01:16:30.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.572+0000 7f407c872740 10 client.4401 fetch_fsmap finished waiting for FSMap version 7 2024-04-25T01:16:30.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.572+0000 7f407c872740 10 client.4401 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-25T01:16:30.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.572+0000 7f407c872740 20 client.4401 populate_metadata read hostname 'smithi090' 2024-04-25T01:16:30.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.572+0000 7f407c872740 1 --2- 172.21.15.90:0/2176406875 >> [v2:172.21.15.103:6833/4291578198,v1:172.21.15.103:6835/4291578198] conn(0x55d933041060 0x55d933043450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.572+0000 7f407c872740 4 client.4401 mds_command: new command op to 4254 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:16:30.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.572+0000 7f407c872740 1 -- 172.21.15.90:0/2176406875 --> [v2:172.21.15.103:6833/4291578198,v1:172.21.15.103:6835/4291578198] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d932f50330 con 0x55d933041060 2024-04-25T01:16:30.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.573+0000 7f4072000640 1 --2- 172.21.15.90:0/2176406875 >> [v2:172.21.15.103:6833/4291578198,v1:172.21.15.103:6835/4291578198] conn(0x55d933041060 0x55d933043450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:30.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.574+0000 7f4072000640 1 --2- 172.21.15.90:0/2176406875 >> [v2:172.21.15.103:6833/4291578198,v1:172.21.15.103:6835/4291578198] conn(0x55d933041060 0x55d933043450 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f035b522eb3b0da6 server_cookie=f316475daaf94502 in_seq=0 out_seq=0 2024-04-25T01:16:30.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.574+0000 7f4057e00640 10 client.4401 ms_handle_connect on v2:172.21.15.103:6833/4291578198 2024-04-25T01:16:30.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.576+0000 7f4057e00640 1 -- 172.21.15.90:0/2176406875 <== mds.0 v2:172.21.15.103:6833/4291578198 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d932f50330 con 0x55d933041060 2024-04-25T01:16:30.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.576+0000 7f4057e00640 10 client.4401 handle_command_reply: tid=0 2024-04-25T01:16:30.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.576+0000 7f407c872740 1 client.4401 shutdown 2024-04-25T01:16:30.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.576+0000 7f407c872740 1 -- 172.21.15.90:0/2176406875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d93300a720 msgr2=0x55d93301c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:30.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.576+0000 7f407c872740 1 --2- 172.21.15.90:0/2176406875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d93300a720 0x55d93301c5e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f405c0a9be0 tx=0x7f405c0ab000 comp rx=0 tx=0).stop 2024-04-25T01:16:30.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.576+0000 7f407c872740 1 -- 172.21.15.90:0/2176406875 shutdown_connections 2024-04-25T01:16:30.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.576+0000 7f407c872740 1 --2- 172.21.15.90:0/2176406875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d9330062f0 0x55d932fd9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.576+0000 7f407c872740 1 --2- 172.21.15.90:0/2176406875 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d933027fc0 0x55d93301d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.576+0000 7f407c872740 1 --2- 172.21.15.90:0/2176406875 >> [v2:172.21.15.103:6833/4291578198,v1:172.21.15.103:6835/4291578198] conn(0x55d933041060 0x55d933043450 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.576+0000 7f407c872740 1 --2- 172.21.15.90:0/2176406875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d93300a720 0x55d93301c5e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.576+0000 7f407c872740 1 -- 172.21.15.90:0/2176406875 >> 172.21.15.90:0/2176406875 conn(0x55d932fe0400 msgr2=0x55d933024cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:30.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.576+0000 7f4057e00640 0 client.4401 ms_handle_reset on v2:172.21.15.103:6833/4291578198 2024-04-25T01:16:30.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.577+0000 7f407c872740 1 -- 172.21.15.90:0/2176406875 shutdown_connections 2024-04-25T01:16:30.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.577+0000 7f407c872740 1 -- 172.21.15.90:0/2176406875 wait complete. 2024-04-25T01:16:30.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.577+0000 7f407c872740 20 client.4401 trim_cache size 0 max 16384 2024-04-25T01:16:30.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.596+0000 7f407c872740 1 Processor -- start 2024-04-25T01:16:30.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.596+0000 7f407c872740 1 -- start start 2024-04-25T01:16:30.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.597+0000 7f407c872740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d9330062f0 0x55d933025530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.597+0000 7f407c872740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d93300a720 0x55d933025a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.597+0000 7f407c872740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d933027fc0 0x55d933048480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.597+0000 7f407c872740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4074200ac0 con 0x55d933027fc0 2024-04-25T01:16:30.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.597+0000 7f407c872740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f40741ffe50 con 0x55d9330062f0 2024-04-25T01:16:30.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.597+0000 7f407c872740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4074200470 con 0x55d93300a720 2024-04-25T01:16:30.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.598+0000 7f4073400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d933027fc0 0x55d933048480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:30.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.598+0000 7f4073400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d933027fc0 0x55d933048480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55088/0 (socket says 172.21.15.90:55088) 2024-04-25T01:16:30.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.598+0000 7f4073400640 1 -- 172.21.15.90:0/1599970757 learned_addr learned my addr 172.21.15.90:0/1599970757 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:30.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.598+0000 7f4072000640 1 --2- 172.21.15.90:0/1599970757 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d93300a720 0x55d933025a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:30.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.598+0000 7f4073400640 1 -- 172.21.15.90:0/1599970757 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d93300a720 msgr2=0x55d933025a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:30.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.598+0000 7f4073400640 1 --2- 172.21.15.90:0/1599970757 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d93300a720 0x55d933025a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.598+0000 7f4073400640 1 -- 172.21.15.90:0/1599970757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d9330062f0 msgr2=0x55d933025530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:30.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.598+0000 7f4072a00640 1 --2- 172.21.15.90:0/1599970757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d9330062f0 0x55d933025530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:30.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.598+0000 7f4073400640 1 --2- 172.21.15.90:0/1599970757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d9330062f0 0x55d933025530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.598+0000 7f4073400640 1 -- 172.21.15.90:0/1599970757 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d933027c80 con 0x55d933027fc0 2024-04-25T01:16:30.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.598+0000 7f4072000640 1 --2- 172.21.15.90:0/1599970757 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d93300a720 0x55d933025a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:30.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.598+0000 7f4073400640 1 --2- 172.21.15.90:0/1599970757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d933027fc0 0x55d933048480 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f40740b3fa0 tx=0x7f406c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a01e2068334d477e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:30.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.598+0000 7f4062a00640 1 -- 172.21.15.90:0/1599970757 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406c017030 con 0x55d933027fc0 2024-04-25T01:16:30.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.598+0000 7f4062a00640 1 -- 172.21.15.90:0/1599970757 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f406c009d80 con 0x55d933027fc0 2024-04-25T01:16:30.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.599+0000 7f4062a00640 1 -- 172.21.15.90:0/1599970757 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406c00f040 con 0x55d933027fc0 2024-04-25T01:16:30.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.599+0000 7f407c872740 1 -- 172.21.15.90:0/1599970757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d933027fc0 msgr2=0x55d933048480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:30.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.599+0000 7f407c872740 1 --2- 172.21.15.90:0/1599970757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d933027fc0 0x55d933048480 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f40740b3fa0 tx=0x7f406c009b30 comp rx=0 tx=0).stop 2024-04-25T01:16:30.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.599+0000 7f407c872740 1 -- 172.21.15.90:0/1599970757 shutdown_connections 2024-04-25T01:16:30.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.599+0000 7f407c872740 1 --2- 172.21.15.90:0/1599970757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d933027fc0 0x55d933048480 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.599+0000 7f407c872740 1 --2- 172.21.15.90:0/1599970757 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d93300a720 0x55d933025a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.599+0000 7f407c872740 1 --2- 172.21.15.90:0/1599970757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d9330062f0 0x55d933025530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.599+0000 7f407c872740 1 -- 172.21.15.90:0/1599970757 >> 172.21.15.90:0/1599970757 conn(0x55d932fe0400 msgr2=0x55d93301d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:30.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.599+0000 7f407c872740 1 -- 172.21.15.90:0/1599970757 shutdown_connections 2024-04-25T01:16:30.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.599+0000 7f407c872740 1 -- 172.21.15.90:0/1599970757 wait complete. 2024-04-25T01:16:30.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.600+0000 7f407c872740 1 -- start start 2024-04-25T01:16:30.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.600+0000 7f407c872740 1 Processor -- start 2024-04-25T01:16:30.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.601+0000 7f407c872740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d9330062f0 0x55d93309e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.601+0000 7f407c872740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d93300a720 0x55d932fde620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.601+0000 7f407c872740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d933027fc0 0x55d932fdeb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.601+0000 7f407c872740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d932efed20 con 0x55d9330062f0 2024-04-25T01:16:30.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.601+0000 7f407c872740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d932edfed0 con 0x55d93300a720 2024-04-25T01:16:30.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.601+0000 7f407c872740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55d932f3edb0 con 0x55d933027fc0 2024-04-25T01:16:30.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.601+0000 7f407c872740 10 client.0 _ll_register_callbacks cb 0x7f405c090070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:16:30.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.601+0000 7f4072a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d9330062f0 0x55d93309e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:30.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.601+0000 7f4072a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d9330062f0 0x55d93309e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55092/0 (socket says 172.21.15.90:55092) 2024-04-25T01:16:30.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.601+0000 7f4072a00640 1 -- 172.21.15.90:0/3891206619 learned_addr learned my addr 172.21.15.90:0/3891206619 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:30.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.602+0000 7f4072a00640 1 -- 172.21.15.90:0/3891206619 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d933027fc0 msgr2=0x55d932fdeb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:30.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.602+0000 7f4072a00640 1 --2- 172.21.15.90:0/3891206619 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d933027fc0 0x55d932fdeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.602+0000 7f4072a00640 1 -- 172.21.15.90:0/3891206619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d93300a720 msgr2=0x55d932fde620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:30.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.602+0000 7f4072a00640 1 --2- 172.21.15.90:0/3891206619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d93300a720 0x55d932fde620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.602+0000 7f4072a00640 1 -- 172.21.15.90:0/3891206619 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d933033b70 con 0x55d9330062f0 2024-04-25T01:16:30.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.602+0000 7f4072a00640 1 --2- 172.21.15.90:0/3891206619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d9330062f0 0x55d93309e9e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f405c08f010 tx=0x7f405c0ab070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3970c3c6cec82abe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:30.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.602+0000 7f4057e00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T01:16:30.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.602+0000 7f4057e00640 1 -- 172.21.15.90:0/3891206619 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c0a2020 con 0x55d9330062f0 2024-04-25T01:16:30.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.602+0000 7f4057e00640 1 -- 172.21.15.90:0/3891206619 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f405c091d80 con 0x55d9330062f0 2024-04-25T01:16:30.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.602+0000 7f407c872740 1 -- 172.21.15.90:0/3891206619 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f406c009d80 con 0x55d9330062f0 2024-04-25T01:16:30.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.602+0000 7f4057e00640 1 -- 172.21.15.90:0/3891206619 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7f405c0a9040 con 0x55d9330062f0 2024-04-25T01:16:30.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.602+0000 7f407c872740 10 client.4407 fetch_fsmap learned FSMap version 7 2024-04-25T01:16:30.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.602+0000 7f407c872740 1 -- 172.21.15.90:0/3891206619 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x55d933033b70 con 0x55d9330062f0 2024-04-25T01:16:30.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.602+0000 7f4057e00640 1 -- 172.21.15.90:0/3891206619 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f406c009d80 con 0x55d9330062f0 2024-04-25T01:16:30.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.603+0000 7f4057e00640 1 -- 172.21.15.90:0/3891206619 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 7) ==== 3502+0+0 (secure 0 0 0) 0x7f405c078d60 con 0x55d9330062f0 2024-04-25T01:16:30.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.603+0000 7f407c872740 10 client.4407 fetch_fsmap finished waiting for FSMap version 7 2024-04-25T01:16:30.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.603+0000 7f407c872740 10 client.4407 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-25T01:16:30.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.603+0000 7f407c872740 20 client.4407 populate_metadata read hostname 'smithi090' 2024-04-25T01:16:30.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.603+0000 7f407c872740 1 --2- 172.21.15.90:0/3891206619 >> [v2:172.21.15.103:6833/4291578198,v1:172.21.15.103:6835/4291578198] conn(0x55d932ffa990 0x55d933026430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.603+0000 7f407c872740 4 client.4407 mds_command: new command op to 4254 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:16:30.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.603+0000 7f407c872740 1 -- 172.21.15.90:0/3891206619 --> [v2:172.21.15.103:6833/4291578198,v1:172.21.15.103:6835/4291578198] -- command(tid 0: {"prefix": "status"}) -- 0x55d932f4a680 con 0x55d932ffa990 2024-04-25T01:16:30.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.603+0000 7f4072000640 1 --2- 172.21.15.90:0/3891206619 >> [v2:172.21.15.103:6833/4291578198,v1:172.21.15.103:6835/4291578198] conn(0x55d932ffa990 0x55d933026430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:30.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.604+0000 7f4072000640 1 --2- 172.21.15.90:0/3891206619 >> [v2:172.21.15.103:6833/4291578198,v1:172.21.15.103:6835/4291578198] conn(0x55d932ffa990 0x55d933026430 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98b23e57d96f759c server_cookie=a54b8e8d71cc5425 in_seq=0 out_seq=0 2024-04-25T01:16:30.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.604+0000 7f4057e00640 10 client.4407 ms_handle_connect on v2:172.21.15.103:6833/4291578198 2024-04-25T01:16:30.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.605+0000 7f4057e00640 1 -- 172.21.15.90:0/3891206619 <== mds.0 v2:172.21.15.103:6833/4291578198 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x55d932f4a680 con 0x55d932ffa990 2024-04-25T01:16:30.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.605+0000 7f4057e00640 10 client.4407 handle_command_reply: tid=0 2024-04-25T01:16:30.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.605+0000 7f407c872740 1 client.4407 shutdown 2024-04-25T01:16:30.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.605+0000 7f407c872740 1 -- 172.21.15.90:0/3891206619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d9330062f0 msgr2=0x55d93309e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:30.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.605+0000 7f407c872740 1 --2- 172.21.15.90:0/3891206619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d9330062f0 0x55d93309e9e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f405c08f010 tx=0x7f405c0ab070 comp rx=0 tx=0).stop 2024-04-25T01:16:30.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.605+0000 7f407c872740 1 -- 172.21.15.90:0/3891206619 shutdown_connections 2024-04-25T01:16:30.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.605+0000 7f407c872740 1 --2- 172.21.15.90:0/3891206619 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55d933027fc0 0x55d932fdeb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.605+0000 7f407c872740 1 --2- 172.21.15.90:0/3891206619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d93300a720 0x55d932fde620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.605+0000 7f407c872740 1 --2- 172.21.15.90:0/3891206619 >> [v2:172.21.15.103:6833/4291578198,v1:172.21.15.103:6835/4291578198] conn(0x55d932ffa990 0x55d933026430 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.605+0000 7f407c872740 1 --2- 172.21.15.90:0/3891206619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d9330062f0 0x55d93309e9e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.605+0000 7f407c872740 1 -- 172.21.15.90:0/3891206619 >> 172.21.15.90:0/3891206619 conn(0x55d932fe0400 msgr2=0x55d9330375e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:30.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.605+0000 7f4057e00640 0 client.4407 ms_handle_reset on v2:172.21.15.103:6833/4291578198 2024-04-25T01:16:30.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.605+0000 7f407c872740 1 -- 172.21.15.90:0/3891206619 shutdown_connections 2024-04-25T01:16:30.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.605+0000 7f407c872740 1 -- 172.21.15.90:0/3891206619 wait complete. 2024-04-25T01:16:30.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.606+0000 7f407c872740 20 client.4407 trim_cache size 0 max 16384 2024-04-25T01:16:30.597 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T01:16:30.597 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21", 2024-04-25T01:16:30.597 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 0, 2024-04-25T01:16:30.597 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 4254, 2024-04-25T01:16:30.597 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-04-25T01:16:30.597 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-04-25T01:16:30.597 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-04-25T01:16:30.597 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 8.0043681279999994, 2024-04-25T01:16:30.597 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 7, 2024-04-25T01:16:30.598 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 26, 2024-04-25T01:16:30.598 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 26, 2024-04-25T01:16:30.598 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 15.560767523999999 2024-04-25T01:16:30.598 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T01:16:30.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.606+0000 7f4073e00640 1 -- 172.21.15.90:0/1408267293 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f405006be30 msgr2=0x7f405006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:30.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.606+0000 7f4073e00640 1 --2- 172.21.15.90:0/1408267293 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f405006be30 0x7f405006e2f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f40741684c0 tx=0x7f4064007d50 comp rx=0 tx=0).stop 2024-04-25T01:16:30.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.606+0000 7f4073e00640 1 -- 172.21.15.90:0/1408267293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f407406d910 msgr2=0x7f407415d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:30.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.606+0000 7f4073e00640 1 --2- 172.21.15.90:0/1408267293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f407406d910 0x7f407415d9d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f405c000c00 tx=0x7f405c0046c0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.606+0000 7f4073e00640 1 -- 172.21.15.90:0/1408267293 shutdown_connections 2024-04-25T01:16:30.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.606+0000 7f4073e00640 1 --2- 172.21.15.90:0/1408267293 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f405006be30 0x7f405006e2f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.606+0000 7f4073e00640 1 --2- 172.21.15.90:0/1408267293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f407415ad40 0x7f407415b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.606+0000 7f4073e00640 1 --2- 172.21.15.90:0/1408267293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40740cf520 0x7f4074144770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.606+0000 7f4073e00640 1 --2- 172.21.15.90:0/1408267293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f407406d910 0x7f407415d9d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.606+0000 7f4073e00640 1 -- 172.21.15.90:0/1408267293 >> 172.21.15.90:0/1408267293 conn(0x7f40740b1660 msgr2=0x7f40740cc3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:30.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.606+0000 7f4073e00640 1 -- 172.21.15.90:0/1408267293 shutdown_connections 2024-04-25T01:16:30.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.606+0000 7f4073e00640 1 -- 172.21.15.90:0/1408267293 wait complete. 2024-04-25T01:16:30.612 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T01:16:30.612 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:16:30.613 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T01:16:30.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.817+0000 7f72dbc00640 1 Processor -- start 2024-04-25T01:16:30.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.818+0000 7f72dbc00640 1 -- start start 2024-04-25T01:16:30.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.819+0000 7f72dbc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72d4143940 0x7f72d406e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.819+0000 7f72dbc00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f72d4143d20 0x7f72d406eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.819+0000 7f72dbc00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72d4070e60 con 0x7f72d415ecf0 2024-04-25T01:16:30.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.819+0000 7f72dbc00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f72d4071140 con 0x7f72d4143d20 2024-04-25T01:16:30.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.819+0000 7f72dbc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f72d4071420 con 0x7f72d4143940 2024-04-25T01:16:30.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.820+0000 7f72d9c00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f72d415ecf0 0x7f72d415f180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:46930/0 (socket says 172.21.15.90:46930) 2024-04-25T01:16:30.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.820+0000 7f72d9c00640 1 -- 172.21.15.90:0/4209517421 learned_addr learned my addr 172.21.15.90:0/4209517421 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:30.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.820+0000 7f72d9200640 1 --2- 172.21.15.90:0/4209517421 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72d4143940 0x7f72d406e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:30.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.820+0000 7f72cbe00640 1 --2- 172.21.15.90:0/4209517421 >> v2:172.21.15.103:3300/0 conn(0x7f72d4143d20 0x7f72d406eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:30.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.820+0000 7f72d3e00640 1 -- 172.21.15.90:0/4209517421 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1060763223 0 0) 0x7f72d4070e60 con 0x7f72d415ecf0 2024-04-25T01:16:30.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.820+0000 7f72d3e00640 1 -- 172.21.15.90:0/4209517421 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72b4003520 con 0x7f72d415ecf0 2024-04-25T01:16:30.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.821+0000 7f72d3e00640 1 -- 172.21.15.90:0/4209517421 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 168989296 0 0) 0x7f72b4003520 con 0x7f72d415ecf0 2024-04-25T01:16:30.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.821+0000 7f72d3e00640 1 -- 172.21.15.90:0/4209517421 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72d4143940 msgr2=0x7f72d406e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:30.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.821+0000 7f72d3e00640 1 --2- 172.21.15.90:0/4209517421 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72d4143940 0x7f72d406e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.821+0000 7f72d3e00640 1 -- 172.21.15.90:0/4209517421 >> v2:172.21.15.103:3300/0 conn(0x7f72d4143d20 msgr2=0x7f72d406eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:30.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.821+0000 7f72d3e00640 1 --2- 172.21.15.90:0/4209517421 >> v2:172.21.15.103:3300/0 conn(0x7f72d4143d20 0x7f72d406eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.821+0000 7f72d3e00640 1 -- 172.21.15.90:0/4209517421 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d4071700 con 0x7f72d415ecf0 2024-04-25T01:16:30.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.821+0000 7f72d3e00640 1 -- 172.21.15.90:0/4209517421 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f72cc0019f0 con 0x7f72d415ecf0 2024-04-25T01:16:30.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.821+0000 7f72d3e00640 1 -- 172.21.15.90:0/4209517421 >> v1:172.21.15.90:6789/0 conn(0x7f72d415ecf0 legacy=0x7f72d415f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:30.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.821+0000 7f72d3e00640 1 --2- 172.21.15.90:0/4209517421 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72b40039b0 0x7f72b4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.821+0000 7f72d3e00640 1 --2- 172.21.15.90:0/4209517421 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72d4143940 0x7f72b4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.821+0000 7f72d3e00640 1 --2- 172.21.15.90:0/4209517421 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72b4004d40 0x7f72b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.821+0000 7f72d3e00640 1 -- 172.21.15.90:0/4209517421 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f72d4071140 con 0x7f72b40039b0 2024-04-25T01:16:30.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.821+0000 7f72d3e00640 1 -- 172.21.15.90:0/4209517421 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f72d4071420 con 0x7f72b4004d40 2024-04-25T01:16:30.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.821+0000 7f72d3e00640 1 -- 172.21.15.90:0/4209517421 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f72b400b9c0 con 0x7f72d4143940 2024-04-25T01:16:30.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.821+0000 7f72d9200640 1 --2- 172.21.15.90:0/4209517421 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72b40039b0 0x7f72b4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:30.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.821+0000 7f72d9200640 1 -- 172.21.15.90:0/4209517421 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72d4143940 msgr2=0x7f72b4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:30.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.821+0000 7f72d9200640 1 --2- 172.21.15.90:0/4209517421 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72d4143940 0x7f72b4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.821+0000 7f72d9200640 1 -- 172.21.15.90:0/4209517421 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72b4004d40 msgr2=0x7f72b40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:30.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.821+0000 7f72d9200640 1 --2- 172.21.15.90:0/4209517421 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72b4004d40 0x7f72b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.821+0000 7f72d9200640 1 -- 172.21.15.90:0/4209517421 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72b4003520 con 0x7f72b40039b0 2024-04-25T01:16:30.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.823+0000 7f72d9200640 1 --2- 172.21.15.90:0/4209517421 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72b40039b0 0x7f72b4003dd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f72bc009ac0 tx=0x7f72b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72df02b652dbea54 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:30.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.823+0000 7f72d3e00640 1 -- 172.21.15.90:0/4209517421 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc036070 con 0x7f72b40039b0 2024-04-25T01:16:30.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.823+0000 7f72d3e00640 1 -- 172.21.15.90:0/4209517421 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72bc031400 con 0x7f72b40039b0 2024-04-25T01:16:30.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.823+0000 7f72d3e00640 1 -- 172.21.15.90:0/4209517421 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc009690 con 0x7f72b40039b0 2024-04-25T01:16:30.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.823+0000 7f72dbc00640 1 -- 172.21.15.90:0/4209517421 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72b40039b0 msgr2=0x7f72b4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:30.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.823+0000 7f72dbc00640 1 --2- 172.21.15.90:0/4209517421 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72b40039b0 0x7f72b4003dd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f72bc009ac0 tx=0x7f72b4005830 comp rx=0 tx=0).stop 2024-04-25T01:16:30.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.823+0000 7f72d9c00640 1 -- 172.21.15.90:0/4209517421 reap_dead start 2024-04-25T01:16:30.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.824+0000 7f72dbc00640 1 -- 172.21.15.90:0/4209517421 shutdown_connections 2024-04-25T01:16:30.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.824+0000 7f72dbc00640 1 -- 172.21.15.90:0/4209517421 >> 172.21.15.90:0/4209517421 conn(0x7f72d40b1820 msgr2=0x7f72d40b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:30.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.824+0000 7f72dbc00640 1 -- 172.21.15.90:0/4209517421 shutdown_connections 2024-04-25T01:16:30.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.824+0000 7f72dbc00640 1 -- 172.21.15.90:0/4209517421 wait complete. 2024-04-25T01:16:30.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.825+0000 7f72dbc00640 1 Processor -- start 2024-04-25T01:16:30.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.826+0000 7f72dbc00640 1 -- start start 2024-04-25T01:16:30.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.826+0000 7f72dbc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72b40039b0 0x7f72d40c5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.826+0000 7f72dbc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72d41423c0 0x7f72d40cda90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.826+0000 7f72dbc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72d40c5910 0x7f72d40c5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.826+0000 7f72dbc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f72d4071fb0 con 0x7f72b40039b0 2024-04-25T01:16:30.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.826+0000 7f72dbc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f72d4071e30 con 0x7f72d41423c0 2024-04-25T01:16:30.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.826+0000 7f72dbc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f72d4072130 con 0x7f72d40c5910 2024-04-25T01:16:30.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.827+0000 7f72d9200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72b40039b0 0x7f72d40c5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:30.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.827+0000 7f72d9200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72b40039b0 0x7f72d40c5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55108/0 (socket says 172.21.15.90:55108) 2024-04-25T01:16:30.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.827+0000 7f72d9200640 1 -- 172.21.15.90:0/194616090 learned_addr learned my addr 172.21.15.90:0/194616090 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:30.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.827+0000 7f72cbe00640 1 --2- 172.21.15.90:0/194616090 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72d41423c0 0x7f72d40cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:30.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.827+0000 7f72d9200640 1 -- 172.21.15.90:0/194616090 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72d40c5910 msgr2=0x7f72d40c5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:30.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.827+0000 7f72d9c00640 1 --2- 172.21.15.90:0/194616090 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72d40c5910 0x7f72d40c5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:30.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.827+0000 7f72d9200640 1 --2- 172.21.15.90:0/194616090 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72d40c5910 0x7f72d40c5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.827+0000 7f72d9200640 1 -- 172.21.15.90:0/194616090 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72d41423c0 msgr2=0x7f72d40cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:30.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.827+0000 7f72d9200640 1 --2- 172.21.15.90:0/194616090 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72d41423c0 0x7f72d40cda90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:30.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.827+0000 7f72d9200640 1 -- 172.21.15.90:0/194616090 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d40ce2e0 con 0x7f72b40039b0 2024-04-25T01:16:30.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.827+0000 7f72d9c00640 1 --2- 172.21.15.90:0/194616090 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72d40c5910 0x7f72d40c5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:30.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.827+0000 7f72d9200640 1 --2- 172.21.15.90:0/194616090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72b40039b0 0x7f72d40c5310 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f72bc009ac0 tx=0x7f72bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:30.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.828+0000 7f72d1600640 1 -- 172.21.15.90:0/194616090 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc040020 con 0x7f72b40039b0 2024-04-25T01:16:30.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.828+0000 7f72d1600640 1 -- 172.21.15.90:0/194616090 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72bc003bc0 con 0x7f72b40039b0 2024-04-25T01:16:30.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.828+0000 7f72d1600640 1 -- 172.21.15.90:0/194616090 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc036040 con 0x7f72b40039b0 2024-04-25T01:16:30.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.828+0000 7f72dbc00640 1 -- 172.21.15.90:0/194616090 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72d40ce5c0 con 0x7f72b40039b0 2024-04-25T01:16:30.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.828+0000 7f72dbc00640 1 -- 172.21.15.90:0/194616090 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72d40ceb80 con 0x7f72b40039b0 2024-04-25T01:16:30.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.829+0000 7f72dbc00640 1 -- 172.21.15.90:0/194616090 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72d41427a0 con 0x7f72b40039b0 2024-04-25T01:16:30.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.830+0000 7f72d1600640 1 -- 172.21.15.90:0/194616090 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f72bc043020 con 0x7f72b40039b0 2024-04-25T01:16:30.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.832+0000 7f72d1600640 1 --2- 172.21.15.90:0/194616090 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f72b006be30 0x7f72b006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:30.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.832+0000 7f72d1600640 1 -- 172.21.15.90:0/194616090 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f72bc0ab240 con 0x7f72b40039b0 2024-04-25T01:16:30.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.832+0000 7f72cbe00640 1 --2- 172.21.15.90:0/194616090 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f72b006be30 0x7f72b006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:30.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.833+0000 7f72cbe00640 1 --2- 172.21.15.90:0/194616090 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f72b006be30 0x7f72b006e2f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f72d40c7720 tx=0x7f72c4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:30.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:30.836+0000 7f72d1600640 1 -- 172.21.15.90:0/194616090 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f72bc04a070 con 0x7f72b40039b0 2024-04-25T01:16:31.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:31.075+0000 7f72dbc00640 1 -- 172.21.15.90:0/194616090 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f72d4068fb0 con 0x7f72b40039b0 2024-04-25T01:16:31.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:31.075+0000 7f72d1600640 1 -- 172.21.15.90:0/194616090 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+88 (secure 0 0 0) 0x7f72bc0044e0 con 0x7f72b40039b0 2024-04-25T01:16:31.060 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T01:16:31.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:31.078+0000 7f72dbc00640 1 -- 172.21.15.90:0/194616090 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f72b006be30 msgr2=0x7f72b006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:31.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:31.078+0000 7f72dbc00640 1 --2- 172.21.15.90:0/194616090 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f72b006be30 0x7f72b006e2f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f72d40c7720 tx=0x7f72c4008040 comp rx=0 tx=0).stop 2024-04-25T01:16:31.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:31.078+0000 7f72dbc00640 1 -- 172.21.15.90:0/194616090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72b40039b0 msgr2=0x7f72d40c5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:31.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:31.078+0000 7f72dbc00640 1 --2- 172.21.15.90:0/194616090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72b40039b0 0x7f72d40c5310 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f72bc009ac0 tx=0x7f72bc004730 comp rx=0 tx=0).stop 2024-04-25T01:16:31.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:31.078+0000 7f72dbc00640 1 -- 172.21.15.90:0/194616090 shutdown_connections 2024-04-25T01:16:31.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:31.078+0000 7f72dbc00640 1 --2- 172.21.15.90:0/194616090 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f72b006be30 0x7f72b006e2f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:31.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:31.078+0000 7f72dbc00640 1 --2- 172.21.15.90:0/194616090 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f72d40c5910 0x7f72d40c5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:31.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:31.078+0000 7f72dbc00640 1 --2- 172.21.15.90:0/194616090 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72d41423c0 0x7f72d40cda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:31.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:31.078+0000 7f72dbc00640 1 --2- 172.21.15.90:0/194616090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72b40039b0 0x7f72d40c5310 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:31.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:31.078+0000 7f72dbc00640 1 -- 172.21.15.90:0/194616090 >> 172.21.15.90:0/194616090 conn(0x7f72d40b1820 msgr2=0x7f72d40afa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:31.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:31.078+0000 7f72dbc00640 1 -- 172.21.15.90:0/194616090 shutdown_connections 2024-04-25T01:16:31.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:31.078+0000 7f72dbc00640 1 -- 172.21.15.90:0/194616090 wait complete. 2024-04-25T01:16:31.081 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-25T01:16:31.081 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T01:16:31.081 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2024-04-25T01:16:31.081 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-25T01:16:31.081 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:16:31.081 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-25T01:16:31.081 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T01:16:31.081 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T01:16:31.081 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-25T01:16:31.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:16:31.082 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2024-04-25T01:16:31.132 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T01:16:31.132 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T01:16:31.132 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T01:16:31.132 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:16:31.132 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-04-25T01:16:31.132 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:16:31.133 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:16:31.133 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:16:31.133 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f0 2024-04-25T01:16:31.133 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:16:31.133 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:16:31.133 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f1 2024-04-25T01:16:31.133 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:16:31.133 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:16:31.133 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T01:16:31.133 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:16:31.133 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-25T01:16:31.133 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 42648sec preferred_lft 42648sec 2024-04-25T01:16:31.133 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::ec4:7aff:fe8f:d767/64 scope link 2024-04-25T01:16:31.133 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:16:31.135 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T01:16:31.135 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:16:31.135 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-04-25T01:16:31.135 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add name ceph-brx type bridge 2024-04-25T01:16:31.135 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip addr flush dev ceph-brx 2024-04-25T01:16:31.135 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set ceph-brx up 2024-04-25T01:16:31.135 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T01:16:31.135 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-04-25T01:16:31.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:16:31.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:16:31.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:16:31.647 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T01:16:31.717 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-04-25T01:16:31.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:16:31.720 DEBUG:teuthology.orchestra.run.smithi090:> route 2024-04-25T01:16:31.799 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2024-04-25T01:16:31.800 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T01:16:31.800 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T01:16:31.800 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T01:16:31.800 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T01:16:31.801 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:16:31.801 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-04-25T01:16:31.801 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T01:16:31.801 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T01:16:31.801 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T01:16:31.801 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-04-25T01:16:31.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:16:32.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:16:32.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:16:32.595 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-04-25T01:16:32.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:16:32.651 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list-id 2024-04-25T01:16:32.703 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:16:32.704 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-04-25T01:16:32.704 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T01:16:32.704 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-25T01:16:32.704 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-04-25T01:16:32.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:32 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:16:32.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:16:32.889 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-25T01:16:32.889 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:16:32.889 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-04-25T01:16:32.889 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-25T01:16:32.889 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T01:16:32.889 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-25T01:16:32.889 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-25T01:16:32.889 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-25T01:16:32.889 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-04-25T01:16:32.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:16:33.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:16:33.178 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:16:33.178 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-04-25T01:16:33.178 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.0 up 2024-04-25T01:16:33.179 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T01:16:33.179 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-04-25T01:16:33.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:16:33.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:16:33.377 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-25T01:16:33.377 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:16:33.377 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:16:33.420 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-25T01:16:33.420 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:16:33.420 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:16:33.511 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-04-25T01:16:33.587 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T01:16:33.634 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-04-25T01:16:33.634 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-04-25T01:16:33.634 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-04-25T01:16:33.634 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-04-25T01:16:33.634 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-04-25T01:16:33.634 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-04-25T01:16:33.634 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-04-25T01:16:33.634 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-04-25T01:16:33.634 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-04-25T01:16:33.634 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-04-25T01:16:33.635 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-04-25T01:16:33.635 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/selinux 2024-04-25T01:16:33.635 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T01:16:33.635 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-04-25T01:16:33.635 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-04-25T01:16:33.635 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-04-25T01:16:33.635 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-04-25T01:16:33.635 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-04-25T01:16:33.635 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-04-25T01:16:33.635 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T01:16:33.635 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T01:16:33.635 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T01:16:33.635 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T01:16:33.635 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-04-25T01:16:33.636 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-04-25T01:16:33.636 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T01:16:33.636 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T01:16:33.636 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T01:16:33.636 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T01:16:33.636 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-04-25T01:16:33.636 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T01:16:33.636 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T01:16:33.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:16:33.637 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-04-25T01:16:33.683 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-25T01:16:33.683 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-25T01:16:33.732 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-04-25T01:16:33.805 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T01:16:33.848 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-04-25T01:16:33.849 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-04-25T01:16:33.849 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-04-25T01:16:33.849 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-04-25T01:16:33.849 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-04-25T01:16:33.849 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-04-25T01:16:33.849 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-04-25T01:16:33.849 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-04-25T01:16:33.849 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-04-25T01:16:33.849 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-04-25T01:16:33.849 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-04-25T01:16:33.849 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/selinux 2024-04-25T01:16:33.849 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T01:16:33.850 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-04-25T01:16:33.850 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-04-25T01:16:33.850 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-04-25T01:16:33.850 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-04-25T01:16:33.850 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-04-25T01:16:33.850 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-04-25T01:16:33.850 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T01:16:33.850 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T01:16:33.850 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T01:16:33.850 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T01:16:33.850 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-04-25T01:16:33.850 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-04-25T01:16:33.850 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T01:16:33.851 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T01:16:33.851 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T01:16:33.851 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T01:16:33.851 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-04-25T01:16:33.851 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T01:16:33.851 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T01:16:33.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:16:33.852 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-04-25T01:16:33.921 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi090.stderr:2024-04-25T01:16:33.937+0000 7f0029907480 -1 init, newargv = 0x558642170da0 newargc=15 2024-04-25T01:16:33.922 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi090.stderr:ceph-fuse[39923]: starting ceph client 2024-04-25T01:16:33.936 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi090.stderr:ceph-fuse[39923]: starting fuse 2024-04-25T01:16:35.035 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-04-25T01:16:35.126 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T01:16:35.181 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-04-25T01:16:35.181 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-04-25T01:16:35.181 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-04-25T01:16:35.181 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-04-25T01:16:35.181 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-04-25T01:16:35.181 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-04-25T01:16:35.181 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-04-25T01:16:35.181 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-04-25T01:16:35.181 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-04-25T01:16:35.181 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-04-25T01:16:35.181 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-04-25T01:16:35.182 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/selinux 2024-04-25T01:16:35.182 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T01:16:35.182 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-04-25T01:16:35.182 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-04-25T01:16:35.182 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-04-25T01:16:35.182 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-04-25T01:16:35.182 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-04-25T01:16:35.182 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-04-25T01:16:35.182 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T01:16:35.182 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T01:16:35.182 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T01:16:35.182 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T01:16:35.182 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-04-25T01:16:35.183 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-04-25T01:16:35.183 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T01:16:35.183 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T01:16:35.183 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T01:16:35.183 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T01:16:35.183 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-04-25T01:16:35.183 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T01:16:35.183 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T01:16:35.183 INFO:teuthology.orchestra.run.smithi090.stdout:/home/ubuntu/cephtest/mnt.admin 2024-04-25T01:16:35.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:16:35.184 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-04-25T01:16:35.234 INFO:teuthology.orchestra.run.smithi090.stdout:42 2024-04-25T01:16:35.234 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-04-25T01:16:35.235 DEBUG:teuthology.orchestra.run.smithi090:> sudo stdin-killer -- python3 -c ' 2024-04-25T01:16:35.235 DEBUG:teuthology.orchestra.run.smithi090:> import glob 2024-04-25T01:16:35.235 DEBUG:teuthology.orchestra.run.smithi090:> import re 2024-04-25T01:16:35.235 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-04-25T01:16:35.235 DEBUG:teuthology.orchestra.run.smithi090:> import subprocess 2024-04-25T01:16:35.235 DEBUG:teuthology.orchestra.run.smithi090:> 2024-04-25T01:16:35.235 DEBUG:teuthology.orchestra.run.smithi090:> def _find_admin_socket(client_name): 2024-04-25T01:16:35.235 DEBUG:teuthology.orchestra.run.smithi090:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-25T01:16:35.235 DEBUG:teuthology.orchestra.run.smithi090:> files = glob.glob(asok_path) 2024-04-25T01:16:35.236 DEBUG:teuthology.orchestra.run.smithi090:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-25T01:16:35.236 DEBUG:teuthology.orchestra.run.smithi090:> 2024-04-25T01:16:35.236 DEBUG:teuthology.orchestra.run.smithi090:> # Given a non-glob path, it better be there 2024-04-25T01:16:35.236 DEBUG:teuthology.orchestra.run.smithi090:> if "*" not in asok_path: 2024-04-25T01:16:35.236 DEBUG:teuthology.orchestra.run.smithi090:> assert(len(files) == 1) 2024-04-25T01:16:35.236 DEBUG:teuthology.orchestra.run.smithi090:> return files[0] 2024-04-25T01:16:35.236 DEBUG:teuthology.orchestra.run.smithi090:> 2024-04-25T01:16:35.236 DEBUG:teuthology.orchestra.run.smithi090:> for f in files: 2024-04-25T01:16:35.236 DEBUG:teuthology.orchestra.run.smithi090:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T01:16:35.236 DEBUG:teuthology.orchestra.run.smithi090:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T01:16:35.236 DEBUG:teuthology.orchestra.run.smithi090:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T01:16:35.236 DEBUG:teuthology.orchestra.run.smithi090:> contents = proc_f.read() 2024-04-25T01:16:35.236 DEBUG:teuthology.orchestra.run.smithi090:> if mountpoint in contents: 2024-04-25T01:16:35.236 DEBUG:teuthology.orchestra.run.smithi090:> return f 2024-04-25T01:16:35.236 DEBUG:teuthology.orchestra.run.smithi090:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T01:16:35.236 DEBUG:teuthology.orchestra.run.smithi090:> 2024-04-25T01:16:35.236 DEBUG:teuthology.orchestra.run.smithi090:> print(_find_admin_socket("client.admin")) 2024-04-25T01:16:35.237 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-04-25T01:16:35.401 INFO:teuthology.orchestra.run.smithi090.stdout:/var/run/ceph/ceph-client.admin.39923.asok 2024-04-25T01:16:35.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:16:35.419 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.39923.asok 2024-04-25T01:16:35.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:16:35.420 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.39923.asok status 2024-04-25T01:16:35.641 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T01:16:35.641 INFO:teuthology.orchestra.run.smithi090.stdout: "metadata": { 2024-04-25T01:16:35.641 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_sha1": "d2c45d977142e024a59b68a8f1dce8f0b9840551", 2024-04-25T01:16:35.641 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-25T01:16:35.642 INFO:teuthology.orchestra.run.smithi090.stdout: "entity_id": "admin", 2024-04-25T01:16:35.642 INFO:teuthology.orchestra.run.smithi090.stdout: "hostname": "smithi090", 2024-04-25T01:16:35.642 INFO:teuthology.orchestra.run.smithi090.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-25T01:16:35.642 INFO:teuthology.orchestra.run.smithi090.stdout: "pid": "39923", 2024-04-25T01:16:35.642 INFO:teuthology.orchestra.run.smithi090.stdout: "root": "/" 2024-04-25T01:16:35.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T01:16:35.642 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_count": 0, 2024-04-25T01:16:35.643 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_pinned_count": 0, 2024-04-25T01:16:35.643 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 4419, 2024-04-25T01:16:35.643 INFO:teuthology.orchestra.run.smithi090.stdout: "inst": { 2024-04-25T01:16:35.643 INFO:teuthology.orchestra.run.smithi090.stdout: "name": { 2024-04-25T01:16:35.643 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "client", 2024-04-25T01:16:35.643 INFO:teuthology.orchestra.run.smithi090.stdout: "num": 4419 2024-04-25T01:16:35.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T01:16:35.643 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-04-25T01:16:35.643 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-25T01:16:35.643 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-04-25T01:16:35.643 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1459284142 2024-04-25T01:16:35.643 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:16:35.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T01:16:35.644 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-04-25T01:16:35.644 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-25T01:16:35.644 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-04-25T01:16:35.644 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1459284142 2024-04-25T01:16:35.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T01:16:35.644 INFO:teuthology.orchestra.run.smithi090.stdout: "inst_str": "client.4419 192.168.0.1:0/1459284142", 2024-04-25T01:16:35.644 INFO:teuthology.orchestra.run.smithi090.stdout: "addr_str": "192.168.0.1:0/1459284142", 2024-04-25T01:16:35.644 INFO:teuthology.orchestra.run.smithi090.stdout: "inode_count": 1, 2024-04-25T01:16:35.644 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_epoch": 7, 2024-04-25T01:16:35.645 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch": 26, 2024-04-25T01:16:35.645 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch_barrier": 0, 2024-04-25T01:16:35.645 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklisted": false, 2024-04-25T01:16:35.645 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs" 2024-04-25T01:16:35.645 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T01:16:35.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:16:35.656 DEBUG:teuthology.orchestra.run.smithi090:> stat --file-system '--printf=%T 2024-04-25T01:16:35.656 DEBUG:teuthology.orchestra.run.smithi090:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:16:35.706 INFO:teuthology.orchestra.run.smithi090.stdout:fuseblk 2024-04-25T01:16:35.706 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:16:35.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:16:35.707 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:16:35.793 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-25T01:16:35.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:35 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:16:35.948 INFO:teuthology.orchestra.run.smithi090.stdout:# file: . 2024-04-25T01:16:35.948 INFO:teuthology.orchestra.run.smithi090.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-25T01:16:35.948 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:16:35.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:16:35.958 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:16:35.959 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:16:36.005 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:16:36.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.264+0000 7f75af200640 1 Processor -- start 2024-04-25T01:16:36.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.264+0000 7f75af200640 1 -- start start 2024-04-25T01:16:36.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.266+0000 7f75af200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f75b0161680 0x7f75b00d1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:36.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.266+0000 7f75af200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75b00b2bf0 0x7f75b00d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:36.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.266+0000 7f75af200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75b00d0e90 con 0x7f75b00d1ad0 2024-04-25T01:16:36.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.266+0000 7f75af200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f75b00d4c20 con 0x7f75b0161680 2024-04-25T01:16:36.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.266+0000 7f75af200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f75b00d4940 con 0x7f75b00b2bf0 2024-04-25T01:16:36.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.266+0000 7f75ade00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f75b0161680 0x7f75b00d1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:36.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.266+0000 7f75ade00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f75b0161680 0x7f75b00d1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51666/0 (socket says 172.21.15.90:51666) 2024-04-25T01:16:36.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.266+0000 7f75ade00640 1 -- 172.21.15.90:0/1220313103 learned_addr learned my addr 172.21.15.90:0/1220313103 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:36.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.267+0000 7f75a7e00640 1 --2- 172.21.15.90:0/1220313103 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75b00b2bf0 0x7f75b00d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:36.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.267+0000 7f75ade00640 1 -- 172.21.15.90:0/1220313103 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75b00b2bf0 msgr2=0x7f75b00d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:36.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.267+0000 7f75ade00640 1 --2- 172.21.15.90:0/1220313103 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75b00b2bf0 0x7f75b00d24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:36.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.267+0000 7f75ade00640 1 -- 172.21.15.90:0/1220313103 >> v1:172.21.15.103:6789/0 conn(0x7f75b00d1ad0 legacy=0x7f75b00ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:36.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.267+0000 7f75ade00640 1 -- 172.21.15.90:0/1220313103 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75b00b0ae0 con 0x7f75b0161680 2024-04-25T01:16:36.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.268+0000 7f75ade00640 1 --2- 172.21.15.90:0/1220313103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75b0161680 0x7f75b00d1fb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f75a00098a0 tx=0x7f75a002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de9454edbe59cb99 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:36.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.268+0000 7f75ad400640 1 -- 172.21.15.90:0/1220313103 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0037070 con 0x7f75b0161680 2024-04-25T01:16:36.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.268+0000 7f75ad400640 1 -- 172.21.15.90:0/1220313103 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75a0032440 con 0x7f75b0161680 2024-04-25T01:16:36.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.268+0000 7f75ad400640 1 -- 172.21.15.90:0/1220313103 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0032810 con 0x7f75b0161680 2024-04-25T01:16:36.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.269+0000 7f75af200640 1 -- 172.21.15.90:0/1220313103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75b0161680 msgr2=0x7f75b00d1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:36.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.269+0000 7f75af200640 1 --2- 172.21.15.90:0/1220313103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75b0161680 0x7f75b00d1fb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f75a00098a0 tx=0x7f75a002a1a0 comp rx=0 tx=0).stop 2024-04-25T01:16:36.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.269+0000 7f75af200640 1 -- 172.21.15.90:0/1220313103 shutdown_connections 2024-04-25T01:16:36.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.269+0000 7f75af200640 1 --2- 172.21.15.90:0/1220313103 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75b00b2bf0 0x7f75b00d24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:36.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.269+0000 7f75af200640 1 --2- 172.21.15.90:0/1220313103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75b0161680 0x7f75b00d1fb0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:36.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.269+0000 7f75af200640 1 -- 172.21.15.90:0/1220313103 >> 172.21.15.90:0/1220313103 conn(0x7f75b00d3cf0 msgr2=0x7f75b00d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:36.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.269+0000 7f75af200640 1 -- 172.21.15.90:0/1220313103 shutdown_connections 2024-04-25T01:16:36.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.269+0000 7f75af200640 1 -- 172.21.15.90:0/1220313103 wait complete. 2024-04-25T01:16:36.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.271+0000 7f75af200640 1 Processor -- start 2024-04-25T01:16:36.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.272+0000 7f75af200640 1 -- start start 2024-04-25T01:16:36.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.273+0000 7f75af200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75b00b2bf0 0x7f75b0154b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:36.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.273+0000 7f75af200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75b00d1ad0 0x7f75b00cc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:36.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.273+0000 7f75af200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75b0161680 0x7f75b00cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:36.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.273+0000 7f75af200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f75b00b2010 con 0x7f75b00b2bf0 2024-04-25T01:16:36.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.273+0000 7f75af200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f75b00b2190 con 0x7f75b0161680 2024-04-25T01:16:36.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.273+0000 7f75af200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f75b00b2310 con 0x7f75b00d1ad0 2024-04-25T01:16:36.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.273+0000 7f75ade00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75b00b2bf0 0x7f75b0154b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:36.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.273+0000 7f75ade00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75b00b2bf0 0x7f75b0154b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51680/0 (socket says 172.21.15.90:51680) 2024-04-25T01:16:36.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.273+0000 7f75ade00640 1 -- 172.21.15.90:0/3897860098 learned_addr learned my addr 172.21.15.90:0/3897860098 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:36.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.274+0000 7f75a7e00640 1 --2- 172.21.15.90:0/3897860098 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75b00d1ad0 0x7f75b00cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:36.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.274+0000 7f75ae800640 1 --2- 172.21.15.90:0/3897860098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75b0161680 0x7f75b00cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:36.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.274+0000 7f75ade00640 1 -- 172.21.15.90:0/3897860098 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75b00d1ad0 msgr2=0x7f75b00cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:36.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.274+0000 7f75ade00640 1 --2- 172.21.15.90:0/3897860098 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75b00d1ad0 0x7f75b00cc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:36.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.274+0000 7f75ade00640 1 -- 172.21.15.90:0/3897860098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75b0161680 msgr2=0x7f75b00cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:36.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.274+0000 7f75ade00640 1 --2- 172.21.15.90:0/3897860098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75b0161680 0x7f75b00cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:36.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.274+0000 7f75ade00640 1 -- 172.21.15.90:0/3897860098 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75b00a0aa0 con 0x7f75b00b2bf0 2024-04-25T01:16:36.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.274+0000 7f75ade00640 1 --2- 172.21.15.90:0/3897860098 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75b00b2bf0 0x7f75b0154b20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f75a00098a0 tx=0x7f75a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:36.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.275+0000 7f75a6000640 1 -- 172.21.15.90:0/3897860098 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0037040 con 0x7f75b00b2bf0 2024-04-25T01:16:36.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.275+0000 7f75a6000640 1 -- 172.21.15.90:0/3897860098 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75a0004230 con 0x7f75b00b2bf0 2024-04-25T01:16:36.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.275+0000 7f75a6000640 1 -- 172.21.15.90:0/3897860098 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0030620 con 0x7f75b00b2bf0 2024-04-25T01:16:36.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.275+0000 7f75af200640 1 -- 172.21.15.90:0/3897860098 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75b00a0d80 con 0x7f75b00b2bf0 2024-04-25T01:16:36.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.275+0000 7f75af200640 1 -- 172.21.15.90:0/3897860098 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75b00a1340 con 0x7f75b00b2bf0 2024-04-25T01:16:36.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.276+0000 7f75af200640 1 -- 172.21.15.90:0/3897860098 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75b0075350 con 0x7f75b00b2bf0 2024-04-25T01:16:36.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.277+0000 7f75a6000640 1 -- 172.21.15.90:0/3897860098 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f75a00307c0 con 0x7f75b00b2bf0 2024-04-25T01:16:36.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.278+0000 7f75a6000640 1 --2- 172.21.15.90:0/3897860098 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f758c06be60 0x7f758c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:36.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.278+0000 7f75a6000640 1 -- 172.21.15.90:0/3897860098 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f75a00ac120 con 0x7f75b00b2bf0 2024-04-25T01:16:36.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.278+0000 7f75a7e00640 1 --2- 172.21.15.90:0/3897860098 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f758c06be60 0x7f758c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:36.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.279+0000 7f75a7e00640 1 --2- 172.21.15.90:0/3897860098 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f758c06be60 0x7f758c06e320 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7598006fd0 tx=0x7f7598008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:36.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.282+0000 7f75a6000640 1 -- 172.21.15.90:0/3897860098 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f75a007a6f0 con 0x7f75b00b2bf0 2024-04-25T01:16:36.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.480+0000 7f75af200640 1 -- 172.21.15.90:0/3897860098 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f75b0074540 con 0x7f75b00b2bf0 2024-04-25T01:16:36.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.483+0000 7f75a6000640 1 -- 172.21.15.90:0/3897860098 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18482 (secure 0 0 0) 0x7f75a0079ff0 con 0x7f75b00b2bf0 2024-04-25T01:16:36.468 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:16:36.468 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":26,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:16:27.092855+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:16:20.094512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:16:21.078268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:16:24.670642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:16:36.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.486+0000 7f75af200640 1 -- 172.21.15.90:0/3897860098 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f758c06be60 msgr2=0x7f758c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:36.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.486+0000 7f75af200640 1 --2- 172.21.15.90:0/3897860098 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f758c06be60 0x7f758c06e320 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7598006fd0 tx=0x7f7598008040 comp rx=0 tx=0).stop 2024-04-25T01:16:36.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.486+0000 7f75af200640 1 -- 172.21.15.90:0/3897860098 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75b00b2bf0 msgr2=0x7f75b0154b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:36.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.486+0000 7f75af200640 1 --2- 172.21.15.90:0/3897860098 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75b00b2bf0 0x7f75b0154b20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f75a00098a0 tx=0x7f75a00046c0 comp rx=0 tx=0).stop 2024-04-25T01:16:36.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.486+0000 7f75af200640 1 -- 172.21.15.90:0/3897860098 shutdown_connections 2024-04-25T01:16:36.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.486+0000 7f75af200640 1 --2- 172.21.15.90:0/3897860098 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f758c06be60 0x7f758c06e320 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:36.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.486+0000 7f75af200640 1 --2- 172.21.15.90:0/3897860098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f75b0161680 0x7f75b00cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:36.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.486+0000 7f75af200640 1 --2- 172.21.15.90:0/3897860098 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f75b00d1ad0 0x7f75b00cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:36.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.486+0000 7f75af200640 1 --2- 172.21.15.90:0/3897860098 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75b00b2bf0 0x7f75b0154b20 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:36.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.486+0000 7f75af200640 1 -- 172.21.15.90:0/3897860098 >> 172.21.15.90:0/3897860098 conn(0x7f75b00d3cf0 msgr2=0x7f75b00d40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:36.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.486+0000 7f75af200640 1 -- 172.21.15.90:0/3897860098 shutdown_connections 2024-04-25T01:16:36.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.486+0000 7f75af200640 1 -- 172.21.15.90:0/3897860098 wait complete. 2024-04-25T01:16:36.488 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:16:15.894126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:16:20.094512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:16:21.078268+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:16:24.670642+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:16:36.489 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:16:36.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.707+0000 7f56b3200640 1 Processor -- start 2024-04-25T01:16:36.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.707+0000 7f56b3200640 1 -- start start 2024-04-25T01:16:36.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.708+0000 7f56b3200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f56b4155130 0x7f56b406db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:36.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.709+0000 7f56b3200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56b4155510 con 0x7f56b406e0b0 2024-04-25T01:16:36.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.709+0000 7f56b3200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56b415d1d0 con 0x7f56b415b950 2024-04-25T01:16:36.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.709+0000 7f56b3200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f56b415bfe0 con 0x7f56b4155130 2024-04-25T01:16:36.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.709+0000 7f56b1400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f56b406e0b0 0x7f56b415b240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56448/0 (socket says 172.21.15.90:56448) 2024-04-25T01:16:36.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.709+0000 7f56b1400640 1 -- 172.21.15.90:0/2787940268 learned_addr learned my addr 172.21.15.90:0/2787940268 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:36.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.709+0000 7f56b1e00640 1 --2- 172.21.15.90:0/2787940268 >> v2:172.21.15.90:3300/0 conn(0x7f56b4155130 0x7f56b406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:36.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.710+0000 7f56b0a00640 1 -- 172.21.15.90:0/2787940268 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2609336765 0 0) 0x7f56b4155510 con 0x7f56b406e0b0 2024-04-25T01:16:36.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.710+0000 7f56b1e00640 1 -- 172.21.15.90:0/2787940268 >> v1:172.21.15.103:6789/0 conn(0x7f56b415b950 legacy=0x7f56b40ae620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:36.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.710+0000 7f56b1e00640 1 -- 172.21.15.90:0/2787940268 >> v1:172.21.15.90:6789/0 conn(0x7f56b406e0b0 legacy=0x7f56b415b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:36.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.710+0000 7f56b1e00640 1 -- 172.21.15.90:0/2787940268 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56b415bd30 con 0x7f56b4155130 2024-04-25T01:16:36.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.711+0000 7f56b1e00640 1 --2- 172.21.15.90:0/2787940268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56b4155130 0x7f56b406db70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f56b415c160 tx=0x7f56a402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24e838cf803343a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:36.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.711+0000 7f56b0a00640 1 -- 172.21.15.90:0/2787940268 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a4037070 con 0x7f56b4155130 2024-04-25T01:16:36.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.711+0000 7f56b0a00640 1 -- 172.21.15.90:0/2787940268 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56a4032440 con 0x7f56b4155130 2024-04-25T01:16:36.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.711+0000 7f56b0a00640 1 -- 172.21.15.90:0/2787940268 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a4032810 con 0x7f56b4155130 2024-04-25T01:16:36.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.712+0000 7f56b3200640 1 -- 172.21.15.90:0/2787940268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56b4155130 msgr2=0x7f56b406db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:36.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.712+0000 7f56b3200640 1 --2- 172.21.15.90:0/2787940268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56b4155130 0x7f56b406db70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f56b415c160 tx=0x7f56a402a180 comp rx=0 tx=0).stop 2024-04-25T01:16:36.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.712+0000 7f56b3200640 1 -- 172.21.15.90:0/2787940268 shutdown_connections 2024-04-25T01:16:36.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.712+0000 7f56b3200640 1 --2- 172.21.15.90:0/2787940268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56b4155130 0x7f56b406db70 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:36.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.712+0000 7f56b3200640 1 -- 172.21.15.90:0/2787940268 >> 172.21.15.90:0/2787940268 conn(0x7f56b406b120 msgr2=0x7f56b406b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:36.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.712+0000 7f56b3200640 1 -- 172.21.15.90:0/2787940268 shutdown_connections 2024-04-25T01:16:36.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.712+0000 7f56b3200640 1 -- 172.21.15.90:0/2787940268 wait complete. 2024-04-25T01:16:36.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.714+0000 7f56b3200640 1 Processor -- start 2024-04-25T01:16:36.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.714+0000 7f56b3200640 1 -- start start 2024-04-25T01:16:36.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.715+0000 7f56b3200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56b406e0b0 0x7f56b41f3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:36.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.715+0000 7f56b3200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56b4155130 0x7f56b414c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:36.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.715+0000 7f56b3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56b415b950 0x7f56b40cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.715+0000 7f56b3200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f56b40720b0 con 0x7f56b406e0b0 2024-04-25T01:16:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.715+0000 7f56b3200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f56b4071dd0 con 0x7f56b4155130 2024-04-25T01:16:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.715+0000 7f56b3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f56b40728b0 con 0x7f56b415b950 2024-04-25T01:16:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.715+0000 7f56b1e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56b406e0b0 0x7f56b41f3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.715+0000 7f56b1e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56b406e0b0 0x7f56b41f3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51704/0 (socket says 172.21.15.90:51704) 2024-04-25T01:16:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.715+0000 7f56b1e00640 1 -- 172.21.15.90:0/4190678279 learned_addr learned my addr 172.21.15.90:0/4190678279 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.715+0000 7f56b1e00640 1 -- 172.21.15.90:0/4190678279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56b415b950 msgr2=0x7f56b40cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.715+0000 7f56b1e00640 1 --2- 172.21.15.90:0/4190678279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56b415b950 0x7f56b40cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.715+0000 7f56b1e00640 1 -- 172.21.15.90:0/4190678279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56b4155130 msgr2=0x7f56b414c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.715+0000 7f56b1e00640 1 --2- 172.21.15.90:0/4190678279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56b4155130 0x7f56b414c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.715+0000 7f56b1e00640 1 -- 172.21.15.90:0/4190678279 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56b41f44c0 con 0x7f56b406e0b0 2024-04-25T01:16:36.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.716+0000 7f56b1e00640 1 --2- 172.21.15.90:0/4190678279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56b406e0b0 0x7f56b41f3f80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f56a4000c00 tx=0x7f56a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:36.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.716+0000 7f56aa000640 1 -- 172.21.15.90:0/4190678279 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a4037050 con 0x7f56b406e0b0 2024-04-25T01:16:36.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.716+0000 7f56aa000640 1 -- 172.21.15.90:0/4190678279 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56a4038040 con 0x7f56b406e0b0 2024-04-25T01:16:36.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.716+0000 7f56b3200640 1 -- 172.21.15.90:0/4190678279 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56b40ccb10 con 0x7f56b406e0b0 2024-04-25T01:16:36.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.716+0000 7f56aa000640 1 -- 172.21.15.90:0/4190678279 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a4004450 con 0x7f56b406e0b0 2024-04-25T01:16:36.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.716+0000 7f56b3200640 1 -- 172.21.15.90:0/4190678279 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56b40cd0d0 con 0x7f56b406e0b0 2024-04-25T01:16:36.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.717+0000 7f56b3200640 1 -- 172.21.15.90:0/4190678279 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56b4075350 con 0x7f56b406e0b0 2024-04-25T01:16:36.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.718+0000 7f56aa000640 1 -- 172.21.15.90:0/4190678279 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f56a4030900 con 0x7f56b406e0b0 2024-04-25T01:16:36.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.719+0000 7f56aa000640 1 --2- 172.21.15.90:0/4190678279 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f569406be30 0x7f569406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:36.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.719+0000 7f56aa000640 1 -- 172.21.15.90:0/4190678279 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f56a40abe30 con 0x7f56b406e0b0 2024-04-25T01:16:36.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.719+0000 7f56b1400640 1 --2- 172.21.15.90:0/4190678279 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f569406be30 0x7f569406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:36.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.720+0000 7f56b1400640 1 --2- 172.21.15.90:0/4190678279 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f569406be30 0x7f569406e2f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f56a0000c00 tx=0x7f56a00079c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:36.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.724+0000 7f56aa000640 1 -- 172.21.15.90:0/4190678279 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f56b41f44c0 con 0x7f56b406e0b0 2024-04-25T01:16:36.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.922+0000 7f56b3200640 1 -- 172.21.15.90:0/4190678279 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f56b414c650 con 0x7f56b406e0b0 2024-04-25T01:16:36.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.923+0000 7f56aa000640 1 -- 172.21.15.90:0/4190678279 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f56a407a520 con 0x7f56b406e0b0 2024-04-25T01:16:36.907 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T01:16:36.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.925+0000 7f56b3200640 1 -- 172.21.15.90:0/4190678279 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f569406be30 msgr2=0x7f569406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:36.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.925+0000 7f56b3200640 1 --2- 172.21.15.90:0/4190678279 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f569406be30 0x7f569406e2f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f56a0000c00 tx=0x7f56a00079c0 comp rx=0 tx=0).stop 2024-04-25T01:16:36.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.925+0000 7f56b3200640 1 -- 172.21.15.90:0/4190678279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56b406e0b0 msgr2=0x7f56b41f3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:36.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.925+0000 7f56b3200640 1 --2- 172.21.15.90:0/4190678279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56b406e0b0 0x7f56b41f3f80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f56a4000c00 tx=0x7f56a40046c0 comp rx=0 tx=0).stop 2024-04-25T01:16:36.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.926+0000 7f56b3200640 1 -- 172.21.15.90:0/4190678279 shutdown_connections 2024-04-25T01:16:36.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.926+0000 7f56b3200640 1 --2- 172.21.15.90:0/4190678279 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f569406be30 0x7f569406e2f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:36.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.926+0000 7f56b3200640 1 --2- 172.21.15.90:0/4190678279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f56b415b950 0x7f56b40cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:36.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.926+0000 7f56b3200640 1 --2- 172.21.15.90:0/4190678279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56b4155130 0x7f56b414c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:36.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.926+0000 7f56b3200640 1 --2- 172.21.15.90:0/4190678279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56b406e0b0 0x7f56b41f3f80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:36.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.926+0000 7f56b3200640 1 -- 172.21.15.90:0/4190678279 >> 172.21.15.90:0/4190678279 conn(0x7f56b406b120 msgr2=0x7f56b4159210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:36.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.926+0000 7f56b3200640 1 -- 172.21.15.90:0/4190678279 shutdown_connections 2024-04-25T01:16:36.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:36.926+0000 7f56b3200640 1 -- 172.21.15.90:0/4190678279 wait complete. 2024-04-25T01:16:36.930 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:16:37.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.147+0000 7f8389600640 1 Processor -- start 2024-04-25T01:16:37.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.148+0000 7f8389600640 1 -- start start 2024-04-25T01:16:37.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.149+0000 7f8389600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f838415e140 0x7f83840d4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:37.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.149+0000 7f8389600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f83840d1ba0 0x7f83840d06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:37.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.149+0000 7f8389600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83840d1020 con 0x7f83840d0c10 2024-04-25T01:16:37.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.149+0000 7f8389600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f83840d2060 con 0x7f83840d1ba0 2024-04-25T01:16:37.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.149+0000 7f8389600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f83840d23b0 con 0x7f838415e140 2024-04-25T01:16:37.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.150+0000 7f8383400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f83840d1ba0 0x7f83840d06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:37.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.150+0000 7f8388c00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f83840d0c10 0x7f83840ceb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56460/0 (socket says 172.21.15.90:56460) 2024-04-25T01:16:37.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.150+0000 7f8388c00640 1 -- 172.21.15.90:0/2734150293 learned_addr learned my addr 172.21.15.90:0/2734150293 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:37.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.150+0000 7f8383e00640 1 --2- 172.21.15.90:0/2734150293 >> v2:172.21.15.103:3300/0 conn(0x7f838415e140 0x7f83840d4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:37.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.150+0000 7f8383400640 1 -- 172.21.15.90:0/2734150293 >> v2:172.21.15.103:3300/0 conn(0x7f838415e140 msgr2=0x7f83840d4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:37.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.150+0000 7f8383400640 1 --2- 172.21.15.90:0/2734150293 >> v2:172.21.15.103:3300/0 conn(0x7f838415e140 0x7f83840d4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.150+0000 7f8383400640 1 -- 172.21.15.90:0/2734150293 >> v1:172.21.15.90:6789/0 conn(0x7f83840d0c10 legacy=0x7f83840ceb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:37.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.150+0000 7f8383400640 1 -- 172.21.15.90:0/2734150293 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83840d2690 con 0x7f83840d1ba0 2024-04-25T01:16:37.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.150+0000 7f8382a00640 1 -- 172.21.15.90:0/2734150293 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 569481170 0 0) 0x7f837c001110 con 0x7f83840d0c10 2024-04-25T01:16:37.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.151+0000 7f8383400640 1 --2- 172.21.15.90:0/2734150293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83840d1ba0 0x7f83840d06d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f838408d490 tx=0x7f836c02a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b70b228d9c041b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:37.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.151+0000 7f8382a00640 1 -- 172.21.15.90:0/2734150293 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f836c007740 con 0x7f83840d1ba0 2024-04-25T01:16:37.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.151+0000 7f8382a00640 1 -- 172.21.15.90:0/2734150293 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f836c0078e0 con 0x7f83840d1ba0 2024-04-25T01:16:37.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.152+0000 7f8382a00640 1 -- 172.21.15.90:0/2734150293 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f836c007cb0 con 0x7f83840d1ba0 2024-04-25T01:16:37.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.152+0000 7f8389600640 1 -- 172.21.15.90:0/2734150293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83840d1ba0 msgr2=0x7f83840d06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:37.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.152+0000 7f8389600640 1 --2- 172.21.15.90:0/2734150293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83840d1ba0 0x7f83840d06d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f838408d490 tx=0x7f836c02a5f0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.152+0000 7f8389600640 1 -- 172.21.15.90:0/2734150293 shutdown_connections 2024-04-25T01:16:37.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.152+0000 7f8389600640 1 --2- 172.21.15.90:0/2734150293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83840d1ba0 0x7f83840d06d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.152+0000 7f8389600640 1 --2- 172.21.15.90:0/2734150293 >> v2:172.21.15.103:3300/0 conn(0x7f838415e140 0x7f83840d4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.152+0000 7f8389600640 1 -- 172.21.15.90:0/2734150293 >> 172.21.15.90:0/2734150293 conn(0x7f83840d3d30 msgr2=0x7f83840d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:37.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.152+0000 7f8389600640 1 -- 172.21.15.90:0/2734150293 shutdown_connections 2024-04-25T01:16:37.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.152+0000 7f8389600640 1 -- 172.21.15.90:0/2734150293 wait complete. 2024-04-25T01:16:37.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.154+0000 7f8389600640 1 Processor -- start 2024-04-25T01:16:37.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.154+0000 7f8389600640 1 -- start start 2024-04-25T01:16:37.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.155+0000 7f8389600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83840d0c10 0x7f838415c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:37.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.155+0000 7f8389600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83840d1ba0 0x7f83841604b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:37.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.155+0000 7f8389600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f838415e140 0x7f838409da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:37.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.155+0000 7f8389600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f83840b2fa0 con 0x7f83840d1ba0 2024-04-25T01:16:37.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.155+0000 7f8389600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f83840d2c50 con 0x7f83840d0c10 2024-04-25T01:16:37.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.155+0000 7f8389600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f83840b3120 con 0x7f838415e140 2024-04-25T01:16:37.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.155+0000 7f8383400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83840d1ba0 0x7f83841604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:37.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.156+0000 7f8383400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83840d1ba0 0x7f83841604b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51710/0 (socket says 172.21.15.90:51710) 2024-04-25T01:16:37.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.156+0000 7f8383400640 1 -- 172.21.15.90:0/3828259499 learned_addr learned my addr 172.21.15.90:0/3828259499 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:37.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.156+0000 7f8383e00640 1 --2- 172.21.15.90:0/3828259499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83840d0c10 0x7f838415c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:37.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.156+0000 7f8388c00640 1 --2- 172.21.15.90:0/3828259499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f838415e140 0x7f838409da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:37.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.156+0000 7f8383400640 1 -- 172.21.15.90:0/3828259499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f838415e140 msgr2=0x7f838409da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:37.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.156+0000 7f8383400640 1 --2- 172.21.15.90:0/3828259499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f838415e140 0x7f838409da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.156+0000 7f8383400640 1 -- 172.21.15.90:0/3828259499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83840d0c10 msgr2=0x7f838415c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:37.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.156+0000 7f8383400640 1 --2- 172.21.15.90:0/3828259499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83840d0c10 0x7f838415c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.156+0000 7f8383400640 1 -- 172.21.15.90:0/3828259499 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8384160e60 con 0x7f83840d1ba0 2024-04-25T01:16:37.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.156+0000 7f8388c00640 1 --2- 172.21.15.90:0/3828259499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f838415e140 0x7f838409da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:37.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.156+0000 7f8383400640 1 --2- 172.21.15.90:0/3828259499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83840d1ba0 0x7f83841604b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f836c009e80 tx=0x7f836c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:37.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.157+0000 7f837be00640 1 -- 172.21.15.90:0/3828259499 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f836c040070 con 0x7f83840d1ba0 2024-04-25T01:16:37.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.157+0000 7f837be00640 1 -- 172.21.15.90:0/3828259499 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f836c031c40 con 0x7f83840d1ba0 2024-04-25T01:16:37.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.157+0000 7f8389600640 1 -- 172.21.15.90:0/3828259499 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83840cba40 con 0x7f83840d1ba0 2024-04-25T01:16:37.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.157+0000 7f837be00640 1 -- 172.21.15.90:0/3828259499 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f836c03c420 con 0x7f83840d1ba0 2024-04-25T01:16:37.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.157+0000 7f8389600640 1 -- 172.21.15.90:0/3828259499 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83840cc000 con 0x7f83840d1ba0 2024-04-25T01:16:37.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.158+0000 7f8389600640 1 -- 172.21.15.90:0/3828259499 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8384075350 con 0x7f83840d1ba0 2024-04-25T01:16:37.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.159+0000 7f837be00640 1 -- 172.21.15.90:0/3828259499 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f836c039040 con 0x7f83840d1ba0 2024-04-25T01:16:37.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.160+0000 7f837be00640 1 --2- 172.21.15.90:0/3828259499 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f836006be30 0x7f836006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:37.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.160+0000 7f837be00640 1 -- 172.21.15.90:0/3828259499 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f836c0ac790 con 0x7f83840d1ba0 2024-04-25T01:16:37.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.160+0000 7f8383e00640 1 --2- 172.21.15.90:0/3828259499 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f836006be30 0x7f836006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:37.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.161+0000 7f8383e00640 1 --2- 172.21.15.90:0/3828259499 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f836006be30 0x7f836006e2f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8374006fd0 tx=0x7f8374008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:37.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.165+0000 7f837be00640 1 -- 172.21.15.90:0/3828259499 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f836c07adf0 con 0x7f83840d1ba0 2024-04-25T01:16:37.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.359+0000 7f8389600640 1 -- 172.21.15.90:0/3828259499 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8384073540 con 0x7f83840d1ba0 2024-04-25T01:16:37.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.360+0000 7f837be00640 1 -- 172.21.15.90:0/3828259499 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f836c07a6f0 con 0x7f83840d1ba0 2024-04-25T01:16:37.344 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:16:37.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.362+0000 7f8389600640 1 -- 172.21.15.90:0/3828259499 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f836006be30 msgr2=0x7f836006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:37.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.362+0000 7f8389600640 1 --2- 172.21.15.90:0/3828259499 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f836006be30 0x7f836006e2f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8374006fd0 tx=0x7f8374008040 comp rx=0 tx=0).stop 2024-04-25T01:16:37.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.362+0000 7f8389600640 1 -- 172.21.15.90:0/3828259499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83840d1ba0 msgr2=0x7f83841604b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:37.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.362+0000 7f8389600640 1 --2- 172.21.15.90:0/3828259499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83840d1ba0 0x7f83841604b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f836c009e80 tx=0x7f836c004a40 comp rx=0 tx=0).stop 2024-04-25T01:16:37.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.363+0000 7f8389600640 1 -- 172.21.15.90:0/3828259499 shutdown_connections 2024-04-25T01:16:37.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.363+0000 7f8389600640 1 --2- 172.21.15.90:0/3828259499 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f836006be30 0x7f836006e2f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.363+0000 7f8389600640 1 --2- 172.21.15.90:0/3828259499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f838415e140 0x7f838409da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.363+0000 7f8389600640 1 --2- 172.21.15.90:0/3828259499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83840d1ba0 0x7f83841604b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.363+0000 7f8389600640 1 --2- 172.21.15.90:0/3828259499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83840d0c10 0x7f838415c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.363+0000 7f8389600640 1 -- 172.21.15.90:0/3828259499 >> 172.21.15.90:0/3828259499 conn(0x7f83840d3d30 msgr2=0x7f83840ca1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:37.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.363+0000 7f8389600640 1 -- 172.21.15.90:0/3828259499 shutdown_connections 2024-04-25T01:16:37.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.363+0000 7f8389600640 1 -- 172.21.15.90:0/3828259499 wait complete. 2024-04-25T01:16:37.366 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:16:37.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.567+0000 7f394f200640 1 Processor -- start 2024-04-25T01:16:37.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.567+0000 7f394f200640 1 -- start start 2024-04-25T01:16:37.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.568+0000 7f394f200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f394815e0e0 0x7f39480d0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:37.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.569+0000 7f394f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f39480d49b0 0x7f39480d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:37.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.569+0000 7f394f200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f39480cf980 0x7f39480d27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:37.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.569+0000 7f394f200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f39480cf4f0 con 0x7f39480cf980 2024-04-25T01:16:37.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.569+0000 7f394f200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f394815e4c0 con 0x7f394815e0e0 2024-04-25T01:16:37.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.569+0000 7f394f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f39480cebc0 con 0x7f39480d49b0 2024-04-25T01:16:37.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.569+0000 7f394e800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f39480cf980 0x7f39480d27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:37.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.569+0000 7f394e800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f39480cf980 0x7f39480d27f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51714/0 (socket says 172.21.15.90:51714) 2024-04-25T01:16:37.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.569+0000 7f394e800640 1 -- 172.21.15.90:0/1898346271 learned_addr learned my addr 172.21.15.90:0/1898346271 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:37.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.569+0000 7f394d400640 1 --2- 172.21.15.90:0/1898346271 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f39480d49b0 0x7f39480d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:37.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.569+0000 7f394de00640 1 --2- 172.21.15.90:0/1898346271 >> v2:172.21.15.103:3300/0 conn(0x7f394815e0e0 0x7f39480d0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:37.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.570+0000 7f394e800640 1 -- 172.21.15.90:0/1898346271 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f39480d49b0 msgr2=0x7f39480d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:37.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.570+0000 7f394e800640 1 --2- 172.21.15.90:0/1898346271 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f39480d49b0 0x7f39480d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.570+0000 7f394e800640 1 -- 172.21.15.90:0/1898346271 >> v2:172.21.15.103:3300/0 conn(0x7f394815e0e0 msgr2=0x7f39480d0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:37.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.570+0000 7f394e800640 1 --2- 172.21.15.90:0/1898346271 >> v2:172.21.15.103:3300/0 conn(0x7f394815e0e0 0x7f39480d0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.570+0000 7f394e800640 1 -- 172.21.15.90:0/1898346271 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39480ceea0 con 0x7f39480cf980 2024-04-25T01:16:37.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.571+0000 7f394e800640 1 --2- 172.21.15.90:0/1898346271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39480cf980 0x7f39480d27f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f394808d620 tx=0x7f394402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a58a009b784200a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:37.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.571+0000 7f394ca00640 1 -- 172.21.15.90:0/1898346271 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3944036070 con 0x7f39480cf980 2024-04-25T01:16:37.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.571+0000 7f394ca00640 1 -- 172.21.15.90:0/1898346271 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3944031600 con 0x7f39480cf980 2024-04-25T01:16:37.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.571+0000 7f394ca00640 1 -- 172.21.15.90:0/1898346271 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39440319d0 con 0x7f39480cf980 2024-04-25T01:16:37.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.572+0000 7f394f200640 1 -- 172.21.15.90:0/1898346271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39480cf980 msgr2=0x7f39480d27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:37.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.572+0000 7f394f200640 1 --2- 172.21.15.90:0/1898346271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39480cf980 0x7f39480d27f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f394808d620 tx=0x7f394402a6a0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.572+0000 7f394f200640 1 -- 172.21.15.90:0/1898346271 shutdown_connections 2024-04-25T01:16:37.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.572+0000 7f394f200640 1 --2- 172.21.15.90:0/1898346271 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f39480d49b0 0x7f39480d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.572+0000 7f394f200640 1 --2- 172.21.15.90:0/1898346271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39480cf980 0x7f39480d27f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.572+0000 7f394f200640 1 --2- 172.21.15.90:0/1898346271 >> v2:172.21.15.103:3300/0 conn(0x7f394815e0e0 0x7f39480d0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.572+0000 7f394f200640 1 -- 172.21.15.90:0/1898346271 >> 172.21.15.90:0/1898346271 conn(0x7f39480d3cd0 msgr2=0x7f39480d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:37.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.572+0000 7f394f200640 1 -- 172.21.15.90:0/1898346271 shutdown_connections 2024-04-25T01:16:37.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.572+0000 7f394f200640 1 -- 172.21.15.90:0/1898346271 wait complete. 2024-04-25T01:16:37.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.574+0000 7f394f200640 1 Processor -- start 2024-04-25T01:16:37.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.574+0000 7f394f200640 1 -- start start 2024-04-25T01:16:37.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.575+0000 7f394f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f39480cf980 0x7f39481f4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:37.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.575+0000 7f394f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39480d49b0 0x7f39480ca030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:37.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.575+0000 7f394f200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f394815e0e0 0x7f39480b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:37.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.575+0000 7f394f200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f39480ce960 con 0x7f394815e0e0 2024-04-25T01:16:37.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.575+0000 7f394f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f39480ce7e0 con 0x7f39480d49b0 2024-04-25T01:16:37.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.575+0000 7f394f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f39480d1250 con 0x7f39480cf980 2024-04-25T01:16:37.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.575+0000 7f394e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f394815e0e0 0x7f39480b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:37.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.575+0000 7f394e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f394815e0e0 0x7f39480b4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51716/0 (socket says 172.21.15.90:51716) 2024-04-25T01:16:37.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.575+0000 7f394e800640 1 -- 172.21.15.90:0/3156540900 learned_addr learned my addr 172.21.15.90:0/3156540900 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:37.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.575+0000 7f394e800640 1 -- 172.21.15.90:0/3156540900 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f39480cf980 msgr2=0x7f39481f4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:37.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.575+0000 7f394e800640 1 --2- 172.21.15.90:0/3156540900 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f39480cf980 0x7f39481f4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.575+0000 7f394e800640 1 -- 172.21.15.90:0/3156540900 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39480d49b0 msgr2=0x7f39480ca030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:37.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.575+0000 7f394e800640 1 --2- 172.21.15.90:0/3156540900 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39480d49b0 0x7f39480ca030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.575+0000 7f394e800640 1 -- 172.21.15.90:0/3156540900 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39480b7c20 con 0x7f394815e0e0 2024-04-25T01:16:37.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.576+0000 7f394e800640 1 --2- 172.21.15.90:0/3156540900 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f394815e0e0 0x7f39480b4a00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f394402ac90 tx=0x7f3944004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.576+0000 7f3942000640 1 -- 172.21.15.90:0/3156540900 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394403f020 con 0x7f394815e0e0 2024-04-25T01:16:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.576+0000 7f3942000640 1 -- 172.21.15.90:0/3156540900 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3944037040 con 0x7f394815e0e0 2024-04-25T01:16:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.576+0000 7f3942000640 1 -- 172.21.15.90:0/3156540900 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3944036070 con 0x7f394815e0e0 2024-04-25T01:16:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.576+0000 7f394f200640 1 -- 172.21.15.90:0/3156540900 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39480b7f00 con 0x7f394815e0e0 2024-04-25T01:16:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.576+0000 7f394f200640 1 -- 172.21.15.90:0/3156540900 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39480b84c0 con 0x7f394815e0e0 2024-04-25T01:16:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.577+0000 7f394f200640 1 -- 172.21.15.90:0/3156540900 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3948075350 con 0x7f394815e0e0 2024-04-25T01:16:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.578+0000 7f3942000640 1 -- 172.21.15.90:0/3156540900 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3944042020 con 0x7f394815e0e0 2024-04-25T01:16:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.579+0000 7f3942000640 1 --2- 172.21.15.90:0/3156540900 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f392806be30 0x7f392806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.579+0000 7f3942000640 1 -- 172.21.15.90:0/3156540900 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f39440ab6b0 con 0x7f394815e0e0 2024-04-25T01:16:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.579+0000 7f394de00640 1 --2- 172.21.15.90:0/3156540900 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f392806be30 0x7f392806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:37.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.580+0000 7f394de00640 1 --2- 172.21.15.90:0/3156540900 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f392806be30 0x7f392806e2f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f393c006fd0 tx=0x7f393c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:37.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.583+0000 7f3942000640 1 -- 172.21.15.90:0/3156540900 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f3944049030 con 0x7f394815e0e0 2024-04-25T01:16:37.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.786+0000 7f394f200640 1 -- 172.21.15.90:0/3156540900 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f39480b3330 con 0x7f394815e0e0 2024-04-25T01:16:37.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.787+0000 7f3942000640 1 -- 172.21.15.90:0/3156540900 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f3944079d10 con 0x7f394815e0e0 2024-04-25T01:16:37.771 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-25T01:16:37.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.790+0000 7f394f200640 1 -- 172.21.15.90:0/3156540900 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f392806be30 msgr2=0x7f392806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:37.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.790+0000 7f394f200640 1 --2- 172.21.15.90:0/3156540900 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f392806be30 0x7f392806e2f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f393c006fd0 tx=0x7f393c008040 comp rx=0 tx=0).stop 2024-04-25T01:16:37.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.790+0000 7f394f200640 1 -- 172.21.15.90:0/3156540900 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f394815e0e0 msgr2=0x7f39480b4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:37.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.790+0000 7f394f200640 1 --2- 172.21.15.90:0/3156540900 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f394815e0e0 0x7f39480b4a00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f394402ac90 tx=0x7f3944004a40 comp rx=0 tx=0).stop 2024-04-25T01:16:37.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.790+0000 7f394f200640 1 -- 172.21.15.90:0/3156540900 shutdown_connections 2024-04-25T01:16:37.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.790+0000 7f394f200640 1 --2- 172.21.15.90:0/3156540900 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f392806be30 0x7f392806e2f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.790+0000 7f394f200640 1 --2- 172.21.15.90:0/3156540900 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f394815e0e0 0x7f39480b4a00 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.790+0000 7f394f200640 1 --2- 172.21.15.90:0/3156540900 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39480d49b0 0x7f39480ca030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.790+0000 7f394f200640 1 --2- 172.21.15.90:0/3156540900 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f39480cf980 0x7f39481f4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.790+0000 7f394f200640 1 -- 172.21.15.90:0/3156540900 >> 172.21.15.90:0/3156540900 conn(0x7f39480d3cd0 msgr2=0x7f39480b28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:37.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.790+0000 7f394f200640 1 -- 172.21.15.90:0/3156540900 shutdown_connections 2024-04-25T01:16:37.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.790+0000 7f394f200640 1 -- 172.21.15.90:0/3156540900 wait complete. 2024-04-25T01:16:37.793 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:16:37.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.995+0000 7fc8d5a00640 1 Processor -- start 2024-04-25T01:16:37.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.995+0000 7fc8d5a00640 1 -- start start 2024-04-25T01:16:37.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.997+0000 7fc8d5a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8d00722e0 0x7fc8d006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:37.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.997+0000 7fc8d5a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8d006e410 con 0x7fc8d00cf4d0 2024-04-25T01:16:37.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.997+0000 7fc8d5a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8d006d110 con 0x7fc8d015fc40 2024-04-25T01:16:37.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.997+0000 7fc8d5a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc8d0160aa0 con 0x7fc8d00722e0 2024-04-25T01:16:37.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.997+0000 7fc8cf400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc8d00cf4d0 0x7fc8d0070b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56468/0 (socket says 172.21.15.90:56468) 2024-04-25T01:16:37.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.997+0000 7fc8cf400640 1 -- 172.21.15.90:0/3364307566 learned_addr learned my addr 172.21.15.90:0/3364307566 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:37.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.998+0000 7fc8cea00640 1 --2- 172.21.15.90:0/3364307566 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8d00722e0 0x7fc8d006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:37.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.998+0000 7fc8cd600640 1 -- 172.21.15.90:0/3364307566 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 444498718 0 0) 0x7fc8d006e410 con 0x7fc8d00cf4d0 2024-04-25T01:16:37.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.998+0000 7fc8cd600640 1 -- 172.21.15.90:0/3364307566 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8b4003520 con 0x7fc8d00cf4d0 2024-04-25T01:16:37.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.998+0000 7fc8cd600640 1 -- 172.21.15.90:0/3364307566 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 185556720 0 0) 0x7fc8b4003520 con 0x7fc8d00cf4d0 2024-04-25T01:16:37.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.998+0000 7fc8cd600640 1 -- 172.21.15.90:0/3364307566 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8d00722e0 msgr2=0x7fc8d006ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:37.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.998+0000 7fc8cd600640 1 --2- 172.21.15.90:0/3364307566 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8d00722e0 0x7fc8d006ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.998+0000 7fc8cd600640 1 -- 172.21.15.90:0/3364307566 >> v1:172.21.15.103:6789/0 conn(0x7fc8d015fc40 legacy=0x7fc8d00cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:37.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.998+0000 7fc8cd600640 1 -- 172.21.15.90:0/3364307566 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d006d3f0 con 0x7fc8d00cf4d0 2024-04-25T01:16:37.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.998+0000 7fc8cd600640 1 -- 172.21.15.90:0/3364307566 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7fc8c80019f0 con 0x7fc8d00cf4d0 2024-04-25T01:16:37.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.998+0000 7fc8cd600640 1 -- 172.21.15.90:0/3364307566 >> v1:172.21.15.90:6789/0 conn(0x7fc8d00cf4d0 legacy=0x7fc8d0070b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:37.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.998+0000 7fc8cd600640 1 --2- 172.21.15.90:0/3364307566 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8d00722e0 0x7fc8b40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:37.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.998+0000 7fc8cd600640 1 --2- 172.21.15.90:0/3364307566 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8b4003f60 0x7fc8b4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:37.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.998+0000 7fc8cd600640 1 --2- 172.21.15.90:0/3364307566 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8b4004d40 0x7fc8b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:37.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.998+0000 7fc8cd600640 1 -- 172.21.15.90:0/3364307566 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d0160aa0 con 0x7fc8b4003f60 2024-04-25T01:16:37.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.998+0000 7fc8cd600640 1 -- 172.21.15.90:0/3364307566 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b400bb50 con 0x7fc8b4004d40 2024-04-25T01:16:37.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.998+0000 7fc8cd600640 1 -- 172.21.15.90:0/3364307566 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc8b400bd50 con 0x7fc8d00722e0 2024-04-25T01:16:37.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.999+0000 7fc8ce000640 1 --2- 172.21.15.90:0/3364307566 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8b4003f60 0x7fc8b4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:37.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.999+0000 7fc8cf400640 1 --2- 172.21.15.90:0/3364307566 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8b4004d40 0x7fc8b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:37.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.999+0000 7fc8ce000640 1 -- 172.21.15.90:0/3364307566 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8d00722e0 msgr2=0x7fc8b40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:37.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.999+0000 7fc8ce000640 1 --2- 172.21.15.90:0/3364307566 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8d00722e0 0x7fc8b40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.999+0000 7fc8ce000640 1 -- 172.21.15.90:0/3364307566 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8b4004d40 msgr2=0x7fc8b40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:37.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.999+0000 7fc8ce000640 1 --2- 172.21.15.90:0/3364307566 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8b4004d40 0x7fc8b40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:37.999+0000 7fc8ce000640 1 -- 172.21.15.90:0/3364307566 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8b4003520 con 0x7fc8b4003f60 2024-04-25T01:16:37.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.000+0000 7fc8ce000640 1 --2- 172.21.15.90:0/3364307566 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8b4003f60 0x7fc8b4004800 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc001750 tx=0x7fc8b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47e0b9ecc61b1430 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:37.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.000+0000 7fc8cd600640 1 -- 172.21.15.90:0/3364307566 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc001370 con 0x7fc8b4003f60 2024-04-25T01:16:37.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.000+0000 7fc8cd600640 1 -- 172.21.15.90:0/3364307566 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8bc0052d0 con 0x7fc8b4003f60 2024-04-25T01:16:37.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.000+0000 7fc8cd600640 1 -- 172.21.15.90:0/3364307566 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc02abe0 con 0x7fc8b4003f60 2024-04-25T01:16:37.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.001+0000 7fc8d5a00640 1 -- 172.21.15.90:0/3364307566 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8b4003f60 msgr2=0x7fc8b4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:37.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.001+0000 7fc8d5a00640 1 --2- 172.21.15.90:0/3364307566 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8b4003f60 0x7fc8b4004800 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc001750 tx=0x7fc8b4005830 comp rx=0 tx=0).stop 2024-04-25T01:16:37.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.001+0000 7fc8cf400640 1 -- 172.21.15.90:0/3364307566 reap_dead start 2024-04-25T01:16:37.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.001+0000 7fc8d5a00640 1 -- 172.21.15.90:0/3364307566 shutdown_connections 2024-04-25T01:16:37.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.001+0000 7fc8d5a00640 1 -- 172.21.15.90:0/3364307566 >> 172.21.15.90:0/3364307566 conn(0x7fc8d006b720 msgr2=0x7fc8d00b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:37.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.001+0000 7fc8d5a00640 1 -- 172.21.15.90:0/3364307566 shutdown_connections 2024-04-25T01:16:37.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.001+0000 7fc8d5a00640 1 -- 172.21.15.90:0/3364307566 wait complete. 2024-04-25T01:16:37.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.002+0000 7fc8d5a00640 1 Processor -- start 2024-04-25T01:16:37.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.003+0000 7fc8d5a00640 1 -- start start 2024-04-25T01:16:37.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.003+0000 7fc8d5a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8b4003f60 0x7fc8d014c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:37.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.004+0000 7fc8d5a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8d01551d0 0x7fc8d0155650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:37.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.004+0000 7fc8d5a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8d0158ab0 0x7fc8d0158f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:37.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.004+0000 7fc8d5a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d006cec0 con 0x7fc8d0158ab0 2024-04-25T01:16:37.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.004+0000 7fc8d5a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d006ded0 con 0x7fc8d01551d0 2024-04-25T01:16:37.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.004+0000 7fc8d5a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc8d006e050 con 0x7fc8b4003f60 2024-04-25T01:16:37.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.004+0000 7fc8cf400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8d0158ab0 0x7fc8d0158f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:37.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.004+0000 7fc8cf400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8d0158ab0 0x7fc8d0158f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51726/0 (socket says 172.21.15.90:51726) 2024-04-25T01:16:37.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.004+0000 7fc8cf400640 1 -- 172.21.15.90:0/174763857 learned_addr learned my addr 172.21.15.90:0/174763857 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:37.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.004+0000 7fc8ce000640 1 --2- 172.21.15.90:0/174763857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8d01551d0 0x7fc8d0155650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:37.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.004+0000 7fc8cea00640 1 --2- 172.21.15.90:0/174763857 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8b4003f60 0x7fc8d014c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:37.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.004+0000 7fc8cf400640 1 -- 172.21.15.90:0/174763857 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8b4003f60 msgr2=0x7fc8d014c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:37.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.004+0000 7fc8cf400640 1 --2- 172.21.15.90:0/174763857 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8b4003f60 0x7fc8d014c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.004+0000 7fc8cf400640 1 -- 172.21.15.90:0/174763857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8d01551d0 msgr2=0x7fc8d0155650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:37.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.004+0000 7fc8cf400640 1 --2- 172.21.15.90:0/174763857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8d01551d0 0x7fc8d0155650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:37.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.004+0000 7fc8cf400640 1 -- 172.21.15.90:0/174763857 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d01f65c0 con 0x7fc8d0158ab0 2024-04-25T01:16:37.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.005+0000 7fc8cf400640 1 --2- 172.21.15.90:0/174763857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8d0158ab0 0x7fc8d0158f10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0072920 tx=0x7fc8c800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:37.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.005+0000 7fc8c6a00640 1 -- 172.21.15.90:0/174763857 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c800ee20 con 0x7fc8d0158ab0 2024-04-25T01:16:37.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.005+0000 7fc8c6a00640 1 -- 172.21.15.90:0/174763857 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8c8014430 con 0x7fc8d0158ab0 2024-04-25T01:16:37.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.005+0000 7fc8c6a00640 1 -- 172.21.15.90:0/174763857 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c8014820 con 0x7fc8d0158ab0 2024-04-25T01:16:37.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.005+0000 7fc8d5a00640 1 -- 172.21.15.90:0/174763857 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8d01f5800 con 0x7fc8d0158ab0 2024-04-25T01:16:37.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.005+0000 7fc8d5a00640 1 -- 172.21.15.90:0/174763857 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8d01f5dc0 con 0x7fc8d0158ab0 2024-04-25T01:16:37.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.006+0000 7fc8d5a00640 1 -- 172.21.15.90:0/174763857 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8d0068fb0 con 0x7fc8d0158ab0 2024-04-25T01:16:37.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.007+0000 7fc8c6a00640 1 -- 172.21.15.90:0/174763857 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc8c80149c0 con 0x7fc8d0158ab0 2024-04-25T01:16:37.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.008+0000 7fc8c6a00640 1 --2- 172.21.15.90:0/174763857 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc8b006be30 0x7fc8b006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:37.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.008+0000 7fc8c6a00640 1 -- 172.21.15.90:0/174763857 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fc8c808b170 con 0x7fc8d0158ab0 2024-04-25T01:16:37.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.008+0000 7fc8cea00640 1 --2- 172.21.15.90:0/174763857 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc8b006be30 0x7fc8b006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:37.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.009+0000 7fc8cea00640 1 --2- 172.21.15.90:0/174763857 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc8b006be30 0x7fc8b006e2f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc8c00047b0 tx=0x7fc8c0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:37.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.012+0000 7fc8c6a00640 1 -- 172.21.15.90:0/174763857 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc8c80596b0 con 0x7fc8d0158ab0 2024-04-25T01:16:38.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.206+0000 7fc8d5a00640 1 -- 172.21.15.90:0/174763857 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc8d0072c90 con 0x7fc8d0158ab0 2024-04-25T01:16:38.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.207+0000 7fc8c6a00640 1 -- 172.21.15.90:0/174763857 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7fc8c8058fb0 con 0x7fc8d0158ab0 2024-04-25T01:16:38.191 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:16:38.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.209+0000 7fc8d5a00640 1 -- 172.21.15.90:0/174763857 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc8b006be30 msgr2=0x7fc8b006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:38.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.209+0000 7fc8d5a00640 1 --2- 172.21.15.90:0/174763857 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc8b006be30 0x7fc8b006e2f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc8c00047b0 tx=0x7fc8c0008040 comp rx=0 tx=0).stop 2024-04-25T01:16:38.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.209+0000 7fc8d5a00640 1 -- 172.21.15.90:0/174763857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8d0158ab0 msgr2=0x7fc8d0158f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:38.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.210+0000 7fc8d5a00640 1 --2- 172.21.15.90:0/174763857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8d0158ab0 0x7fc8d0158f10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0072920 tx=0x7fc8c800e530 comp rx=0 tx=0).stop 2024-04-25T01:16:38.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.210+0000 7fc8d5a00640 1 -- 172.21.15.90:0/174763857 shutdown_connections 2024-04-25T01:16:38.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.210+0000 7fc8d5a00640 1 --2- 172.21.15.90:0/174763857 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc8b006be30 0x7fc8b006e2f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:38.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.210+0000 7fc8d5a00640 1 --2- 172.21.15.90:0/174763857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8d0158ab0 0x7fc8d0158f10 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:38.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.210+0000 7fc8d5a00640 1 --2- 172.21.15.90:0/174763857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8d01551d0 0x7fc8d0155650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:38.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.210+0000 7fc8d5a00640 1 --2- 172.21.15.90:0/174763857 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc8b4003f60 0x7fc8d014c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:38.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.210+0000 7fc8d5a00640 1 -- 172.21.15.90:0/174763857 >> 172.21.15.90:0/174763857 conn(0x7fc8d006b720 msgr2=0x7fc8d015c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:38.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.210+0000 7fc8d5a00640 1 -- 172.21.15.90:0/174763857 shutdown_connections 2024-04-25T01:16:38.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.210+0000 7fc8d5a00640 1 -- 172.21.15.90:0/174763857 wait complete. 2024-04-25T01:16:38.213 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T01:16:38.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.429+0000 7fb6a4a00640 1 Processor -- start 2024-04-25T01:16:38.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.429+0000 7fb6a4a00640 1 -- start start 2024-04-25T01:16:38.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.430+0000 7fb6a4a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6a015c8a0 0x7fb6a01633d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:38.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.430+0000 7fb6a4a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6a00af400 con 0x7fb6a00ae730 2024-04-25T01:16:38.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.430+0000 7fb6a4a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6a00aeb90 con 0x7fb6a006b120 2024-04-25T01:16:38.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.430+0000 7fb6a4a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb6a00cc800 con 0x7fb6a015c8a0 2024-04-25T01:16:38.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.430+0000 7fb69ea00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fb6a00ae730 0x7fb6a00ccc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56474/0 (socket says 172.21.15.90:56474) 2024-04-25T01:16:38.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.430+0000 7fb69ea00640 1 -- 172.21.15.90:0/2877406089 learned_addr learned my addr 172.21.15.90:0/2877406089 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:38.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.431+0000 7fb69f400640 1 --2- 172.21.15.90:0/2877406089 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6a015c8a0 0x7fb6a01633d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:38.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.431+0000 7fb69e000640 1 -- 172.21.15.90:0/2877406089 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1883532049 0 0) 0x7fb6a00af400 con 0x7fb6a00ae730 2024-04-25T01:16:38.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.431+0000 7fb69e000640 1 -- 172.21.15.90:0/2877406089 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb684003520 con 0x7fb6a00ae730 2024-04-25T01:16:38.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.431+0000 7fb69e000640 1 -- 172.21.15.90:0/2877406089 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3403308211 0 0) 0x7fb684003520 con 0x7fb6a00ae730 2024-04-25T01:16:38.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.431+0000 7fb69e000640 1 -- 172.21.15.90:0/2877406089 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6a015c8a0 msgr2=0x7fb6a01633d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:38.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.431+0000 7fb69e000640 1 --2- 172.21.15.90:0/2877406089 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6a015c8a0 0x7fb6a01633d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:38.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.431+0000 7fb69e000640 1 -- 172.21.15.90:0/2877406089 >> v1:172.21.15.103:6789/0 conn(0x7fb6a006b120 legacy=0x7fb6a006b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:38.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.431+0000 7fb69e000640 1 -- 172.21.15.90:0/2877406089 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6a00aee40 con 0x7fb6a00ae730 2024-04-25T01:16:38.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.431+0000 7fb69e000640 1 -- 172.21.15.90:0/2877406089 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7fb688001670 con 0x7fb6a00ae730 2024-04-25T01:16:38.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.431+0000 7fb69e000640 1 -- 172.21.15.90:0/2877406089 >> v1:172.21.15.90:6789/0 conn(0x7fb6a00ae730 legacy=0x7fb6a00ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:38.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.432+0000 7fb69e000640 1 --2- 172.21.15.90:0/2877406089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6840039e0 0x7fb684003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:38.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.432+0000 7fb69e000640 1 --2- 172.21.15.90:0/2877406089 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6a015c8a0 0x7fb684004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:38.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.432+0000 7fb69e000640 1 --2- 172.21.15.90:0/2877406089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb684004d40 0x7fb6840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:38.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.432+0000 7fb69e000640 1 -- 172.21.15.90:0/2877406089 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a00cc800 con 0x7fb6840039e0 2024-04-25T01:16:38.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.432+0000 7fb69e000640 1 -- 172.21.15.90:0/2877406089 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb68400ba70 con 0x7fb684004d40 2024-04-25T01:16:38.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.432+0000 7fb69e000640 1 -- 172.21.15.90:0/2877406089 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb68400bc70 con 0x7fb6a015c8a0 2024-04-25T01:16:38.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.432+0000 7fb69f400640 1 --2- 172.21.15.90:0/2877406089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6840039e0 0x7fb684003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:38.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.432+0000 7fb69ea00640 1 --2- 172.21.15.90:0/2877406089 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6a015c8a0 0x7fb684004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:38.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.432+0000 7fb69fe00640 1 --2- 172.21.15.90:0/2877406089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb684004d40 0x7fb6840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:38.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.432+0000 7fb69f400640 1 -- 172.21.15.90:0/2877406089 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6a015c8a0 msgr2=0x7fb684004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:38.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.432+0000 7fb69f400640 1 --2- 172.21.15.90:0/2877406089 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6a015c8a0 0x7fb684004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:38.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.432+0000 7fb69f400640 1 -- 172.21.15.90:0/2877406089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb684004d40 msgr2=0x7fb6840051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:38.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.432+0000 7fb69f400640 1 --2- 172.21.15.90:0/2877406089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb684004d40 0x7fb6840051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:38.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.432+0000 7fb69f400640 1 -- 172.21.15.90:0/2877406089 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb684003520 con 0x7fb6840039e0 2024-04-25T01:16:38.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.432+0000 7fb69fe00640 1 --2- 172.21.15.90:0/2877406089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb684004d40 0x7fb6840051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:38.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.433+0000 7fb69f400640 1 --2- 172.21.15.90:0/2877406089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6840039e0 0x7fb684003e00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb690009f90 tx=0x7fb684005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbfae239d84c747d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:38.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.434+0000 7fb69e000640 1 -- 172.21.15.90:0/2877406089 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb690009c10 con 0x7fb6840039e0 2024-04-25T01:16:38.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.434+0000 7fb69e000640 1 -- 172.21.15.90:0/2877406089 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb690036070 con 0x7fb6840039e0 2024-04-25T01:16:38.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.434+0000 7fb69e000640 1 -- 172.21.15.90:0/2877406089 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb690031850 con 0x7fb6840039e0 2024-04-25T01:16:38.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.434+0000 7fb6a4a00640 1 -- 172.21.15.90:0/2877406089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6840039e0 msgr2=0x7fb684003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:38.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.434+0000 7fb6a4a00640 1 --2- 172.21.15.90:0/2877406089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6840039e0 0x7fb684003e00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb690009f90 tx=0x7fb684005830 comp rx=0 tx=0).stop 2024-04-25T01:16:38.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.434+0000 7fb69fe00640 1 -- 172.21.15.90:0/2877406089 reap_dead start 2024-04-25T01:16:38.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.434+0000 7fb6a4a00640 1 -- 172.21.15.90:0/2877406089 shutdown_connections 2024-04-25T01:16:38.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.434+0000 7fb6a4a00640 1 -- 172.21.15.90:0/2877406089 >> 172.21.15.90:0/2877406089 conn(0x7fb6a00b1660 msgr2=0x7fb6a00b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:38.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.434+0000 7fb6a4a00640 1 -- 172.21.15.90:0/2877406089 shutdown_connections 2024-04-25T01:16:38.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.434+0000 7fb6a4a00640 1 -- 172.21.15.90:0/2877406089 wait complete. 2024-04-25T01:16:38.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.436+0000 7fb6a4a00640 1 Processor -- start 2024-04-25T01:16:38.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.436+0000 7fb6a4a00640 1 -- start start 2024-04-25T01:16:38.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.437+0000 7fb6a4a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6840039e0 0x7fb6a01535e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:38.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.437+0000 7fb6a4a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6a01eb4b0 0x7fb6a01e94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:38.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.437+0000 7fb6a4a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6a01e9a30 0x7fb6a00ce8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:38.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.437+0000 7fb6a4a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a015cc80 con 0x7fb6a01eb4b0 2024-04-25T01:16:38.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.437+0000 7fb6a4a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a00cc490 con 0x7fb6840039e0 2024-04-25T01:16:38.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.437+0000 7fb6a4a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb6a01633d0 con 0x7fb6a01e9a30 2024-04-25T01:16:38.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.437+0000 7fb69ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6a01eb4b0 0x7fb6a01e94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:38.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.437+0000 7fb69ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6a01eb4b0 0x7fb6a01e94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51742/0 (socket says 172.21.15.90:51742) 2024-04-25T01:16:38.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.437+0000 7fb69ea00640 1 -- 172.21.15.90:0/3060611999 learned_addr learned my addr 172.21.15.90:0/3060611999 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:38.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.437+0000 7fb69f400640 1 --2- 172.21.15.90:0/3060611999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6840039e0 0x7fb6a01535e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:38.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.438+0000 7fb69fe00640 1 --2- 172.21.15.90:0/3060611999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6a01e9a30 0x7fb6a00ce8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:38.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.438+0000 7fb69ea00640 1 -- 172.21.15.90:0/3060611999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6a01e9a30 msgr2=0x7fb6a00ce8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:38.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.438+0000 7fb69ea00640 1 --2- 172.21.15.90:0/3060611999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6a01e9a30 0x7fb6a00ce8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:38.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.438+0000 7fb69ea00640 1 -- 172.21.15.90:0/3060611999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6840039e0 msgr2=0x7fb6a01535e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:38.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.438+0000 7fb69ea00640 1 --2- 172.21.15.90:0/3060611999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6840039e0 0x7fb6a01535e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:38.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.438+0000 7fb69ea00640 1 -- 172.21.15.90:0/3060611999 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6a01ea6f0 con 0x7fb6a01eb4b0 2024-04-25T01:16:38.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.438+0000 7fb69fe00640 1 --2- 172.21.15.90:0/3060611999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6a01e9a30 0x7fb6a00ce8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:38.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.438+0000 7fb69ea00640 1 --2- 172.21.15.90:0/3060611999 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6a01eb4b0 0x7fb6a01e94f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb68800e7b0 tx=0x7fb68800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:38.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.438+0000 7fb697400640 1 -- 172.21.15.90:0/3060611999 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb68800ee60 con 0x7fb6a01eb4b0 2024-04-25T01:16:38.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.438+0000 7fb697400640 1 -- 172.21.15.90:0/3060611999 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb688018070 con 0x7fb6a01eb4b0 2024-04-25T01:16:38.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.438+0000 7fb6a4a00640 1 -- 172.21.15.90:0/3060611999 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6a01ea9d0 con 0x7fb6a01eb4b0 2024-04-25T01:16:38.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.438+0000 7fb697400640 1 -- 172.21.15.90:0/3060611999 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb68800cab0 con 0x7fb6a01eb4b0 2024-04-25T01:16:38.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.438+0000 7fb6a4a00640 1 -- 172.21.15.90:0/3060611999 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6a01eb930 con 0x7fb6a01eb4b0 2024-04-25T01:16:38.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.440+0000 7fb6a4a00640 1 -- 172.21.15.90:0/3060611999 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6a0075350 con 0x7fb6a01eb4b0 2024-04-25T01:16:38.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.440+0000 7fb697400640 1 -- 172.21.15.90:0/3060611999 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb68800cc50 con 0x7fb6a01eb4b0 2024-04-25T01:16:38.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.442+0000 7fb697400640 1 --2- 172.21.15.90:0/3060611999 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb67c06be30 0x7fb67c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:38.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.442+0000 7fb697400640 1 -- 172.21.15.90:0/3060611999 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fb68808afe0 con 0x7fb6a01eb4b0 2024-04-25T01:16:38.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.442+0000 7fb69f400640 1 --2- 172.21.15.90:0/3060611999 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb67c06be30 0x7fb67c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:38.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.443+0000 7fb69f400640 1 --2- 172.21.15.90:0/3060611999 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb67c06be30 0x7fb67c06e2f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb690035000 tx=0x7fb69002f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:38.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.446+0000 7fb697400640 1 -- 172.21.15.90:0/3060611999 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb6880595b0 con 0x7fb6a01eb4b0 2024-04-25T01:16:38.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.647+0000 7fb6a4a00640 1 -- 172.21.15.90:0/3060611999 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb6a00b29f0 con 0x7fb6a01eb4b0 2024-04-25T01:16:38.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.647+0000 7fb697400640 1 -- 172.21.15.90:0/3060611999 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7fb688058eb0 con 0x7fb6a01eb4b0 2024-04-25T01:16:38.632 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-04-25T01:16:38.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.650+0000 7fb6a4a00640 1 -- 172.21.15.90:0/3060611999 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb67c06be30 msgr2=0x7fb67c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:38.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.650+0000 7fb6a4a00640 1 --2- 172.21.15.90:0/3060611999 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb67c06be30 0x7fb67c06e2f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb690035000 tx=0x7fb69002f000 comp rx=0 tx=0).stop 2024-04-25T01:16:38.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.650+0000 7fb6a4a00640 1 -- 172.21.15.90:0/3060611999 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6a01eb4b0 msgr2=0x7fb6a01e94f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:38.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.650+0000 7fb6a4a00640 1 --2- 172.21.15.90:0/3060611999 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6a01eb4b0 0x7fb6a01e94f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb68800e7b0 tx=0x7fb68800ec70 comp rx=0 tx=0).stop 2024-04-25T01:16:38.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.650+0000 7fb6a4a00640 1 -- 172.21.15.90:0/3060611999 shutdown_connections 2024-04-25T01:16:38.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.651+0000 7fb6a4a00640 1 --2- 172.21.15.90:0/3060611999 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb67c06be30 0x7fb67c06e2f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:38.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.651+0000 7fb6a4a00640 1 --2- 172.21.15.90:0/3060611999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6a01e9a30 0x7fb6a00ce8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:38.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.651+0000 7fb6a4a00640 1 --2- 172.21.15.90:0/3060611999 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6a01eb4b0 0x7fb6a01e94f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:38.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.651+0000 7fb6a4a00640 1 --2- 172.21.15.90:0/3060611999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6840039e0 0x7fb6a01535e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:38.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.651+0000 7fb6a4a00640 1 -- 172.21.15.90:0/3060611999 >> 172.21.15.90:0/3060611999 conn(0x7fb6a00b1660 msgr2=0x7fb6a0070b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:38.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.651+0000 7fb6a4a00640 1 -- 172.21.15.90:0/3060611999 shutdown_connections 2024-04-25T01:16:38.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.651+0000 7fb6a4a00640 1 -- 172.21.15.90:0/3060611999 wait complete. 2024-04-25T01:16:38.652 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:16:38.652 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:16:38.701 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:16:38.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.961+0000 7f4172600640 1 Processor -- start 2024-04-25T01:16:38.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.961+0000 7f4172600640 1 -- start start 2024-04-25T01:16:38.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.963+0000 7f4172600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f416c15ca10 0x7f416c15cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:38.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.963+0000 7f4172600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f416c15c220 con 0x7f416c0ae730 2024-04-25T01:16:38.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.963+0000 7f4172600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f416c0aec80 con 0x7f416c06b120 2024-04-25T01:16:38.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.963+0000 7f4172600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f416c0cef40 con 0x7f416c15ca10 2024-04-25T01:16:38.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.963+0000 7f416be00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f416c0ae730 0x7f416c0cd900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56480/0 (socket says 172.21.15.90:56480) 2024-04-25T01:16:38.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.963+0000 7f416be00640 1 -- 172.21.15.90:0/3912697325 learned_addr learned my addr 172.21.15.90:0/3912697325 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:38.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.964+0000 7f4171c00640 1 --2- 172.21.15.90:0/3912697325 >> v2:172.21.15.90:3300/0 conn(0x7f416c15ca10 0x7f416c15cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:38.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.964+0000 7f416b400640 1 -- 172.21.15.90:0/3912697325 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1551905723 0 0) 0x7f416c15c220 con 0x7f416c0ae730 2024-04-25T01:16:38.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.964+0000 7f416b400640 1 -- 172.21.15.90:0/3912697325 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4154003520 con 0x7f416c0ae730 2024-04-25T01:16:38.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.964+0000 7f4171c00640 1 -- 172.21.15.90:0/3912697325 >> v1:172.21.15.103:6789/0 conn(0x7f416c06b120 legacy=0x7f416c06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:38.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.964+0000 7f4171c00640 1 -- 172.21.15.90:0/3912697325 >> v1:172.21.15.90:6789/0 conn(0x7f416c0ae730 legacy=0x7f416c0cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:38.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.964+0000 7f4171c00640 1 -- 172.21.15.90:0/3912697325 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f416c0aef60 con 0x7f416c15ca10 2024-04-25T01:16:38.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.965+0000 7f4171c00640 1 --2- 172.21.15.90:0/3912697325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f416c15ca10 0x7f416c15cea0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4164009d90 tx=0x7f416402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e74ce57d35b70cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:38.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.965+0000 7f416b400640 1 -- 172.21.15.90:0/3912697325 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4164007740 con 0x7f416c15ca10 2024-04-25T01:16:38.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.965+0000 7f416b400640 1 -- 172.21.15.90:0/3912697325 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41640078e0 con 0x7f416c15ca10 2024-04-25T01:16:38.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.965+0000 7f416b400640 1 -- 172.21.15.90:0/3912697325 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4164007cb0 con 0x7f416c15ca10 2024-04-25T01:16:38.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.966+0000 7f4172600640 1 -- 172.21.15.90:0/3912697325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f416c15ca10 msgr2=0x7f416c15cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:38.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.966+0000 7f4172600640 1 --2- 172.21.15.90:0/3912697325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f416c15ca10 0x7f416c15cea0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4164009d90 tx=0x7f416402a6a0 comp rx=0 tx=0).stop 2024-04-25T01:16:38.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.966+0000 7f4172600640 1 -- 172.21.15.90:0/3912697325 shutdown_connections 2024-04-25T01:16:38.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.966+0000 7f4172600640 1 --2- 172.21.15.90:0/3912697325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f416c15ca10 0x7f416c15cea0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:38.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.966+0000 7f4172600640 1 -- 172.21.15.90:0/3912697325 >> 172.21.15.90:0/3912697325 conn(0x7f416c0b1660 msgr2=0x7f416c0b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:38.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.966+0000 7f4172600640 1 -- 172.21.15.90:0/3912697325 shutdown_connections 2024-04-25T01:16:38.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.966+0000 7f4172600640 1 -- 172.21.15.90:0/3912697325 wait complete. 2024-04-25T01:16:38.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.968+0000 7f4172600640 1 Processor -- start 2024-04-25T01:16:38.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.968+0000 7f4172600640 1 -- start start 2024-04-25T01:16:38.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.969+0000 7f4172600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f416c06b120 0x7f416c156930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:38.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.969+0000 7f4172600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f416c0ae730 0x7f416c1443e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:38.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.969+0000 7f4172600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f416c15ca10 0x7f416c1503d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:38.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.969+0000 7f4172600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f416c153080 con 0x7f416c0ae730 2024-04-25T01:16:38.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.969+0000 7f4172600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f416c15c580 con 0x7f416c06b120 2024-04-25T01:16:38.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.969+0000 7f4172600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f416c153390 con 0x7f416c15ca10 2024-04-25T01:16:38.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.969+0000 7f416be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f416c0ae730 0x7f416c1443e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:38.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.969+0000 7f416be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f416c0ae730 0x7f416c1443e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51774/0 (socket says 172.21.15.90:51774) 2024-04-25T01:16:38.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.969+0000 7f416be00640 1 -- 172.21.15.90:0/3380546872 learned_addr learned my addr 172.21.15.90:0/3380546872 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.969+0000 7f4171200640 1 --2- 172.21.15.90:0/3380546872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f416c06b120 0x7f416c156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.969+0000 7f4171c00640 1 --2- 172.21.15.90:0/3380546872 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f416c15ca10 0x7f416c1503d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.969+0000 7f416be00640 1 -- 172.21.15.90:0/3380546872 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f416c15ca10 msgr2=0x7f416c1503d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.969+0000 7f416be00640 1 --2- 172.21.15.90:0/3380546872 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f416c15ca10 0x7f416c1503d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.969+0000 7f416be00640 1 -- 172.21.15.90:0/3380546872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f416c06b120 msgr2=0x7f416c156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.969+0000 7f416be00640 1 --2- 172.21.15.90:0/3380546872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f416c06b120 0x7f416c156930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.969+0000 7f416be00640 1 -- 172.21.15.90:0/3380546872 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f416c1535e0 con 0x7f416c0ae730 2024-04-25T01:16:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.969+0000 7f4171200640 1 --2- 172.21.15.90:0/3380546872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f416c06b120 0x7f416c156930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.970+0000 7f416be00640 1 --2- 172.21.15.90:0/3380546872 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f416c0ae730 0x7f416c1443e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f415800c430 tx=0x7f415800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.970+0000 7f4168c00640 1 -- 172.21.15.90:0/3380546872 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4158017070 con 0x7f416c0ae730 2024-04-25T01:16:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.970+0000 7f4168c00640 1 -- 172.21.15.90:0/3380546872 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4158005ce0 con 0x7f416c0ae730 2024-04-25T01:16:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.970+0000 7f4168c00640 1 -- 172.21.15.90:0/3380546872 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41580072f0 con 0x7f416c0ae730 2024-04-25T01:16:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.970+0000 7f4172600640 1 -- 172.21.15.90:0/3380546872 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f416c1538c0 con 0x7f416c0ae730 2024-04-25T01:16:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.970+0000 7f4172600640 1 -- 172.21.15.90:0/3380546872 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f416c144920 con 0x7f416c0ae730 2024-04-25T01:16:38.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.971+0000 7f4172600640 1 -- 172.21.15.90:0/3380546872 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f416c075350 con 0x7f416c0ae730 2024-04-25T01:16:38.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.972+0000 7f4168c00640 1 -- 172.21.15.90:0/3380546872 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f415801a070 con 0x7f416c0ae730 2024-04-25T01:16:38.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.973+0000 7f4168c00640 1 --2- 172.21.15.90:0/3380546872 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f414c06be30 0x7f414c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:38.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.973+0000 7f4168c00640 1 -- 172.21.15.90:0/3380546872 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f415808a610 con 0x7f416c0ae730 2024-04-25T01:16:38.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.973+0000 7f4171200640 1 --2- 172.21.15.90:0/3380546872 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f414c06be30 0x7f414c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:38.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.974+0000 7f4171200640 1 --2- 172.21.15.90:0/3380546872 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f414c06be30 0x7f414c06e2f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f416c159da0 tx=0x7f4160009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:38.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:38.977+0000 7f4168c00640 1 -- 172.21.15.90:0/3380546872 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4158058d10 con 0x7f416c0ae730 2024-04-25T01:16:39.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.177+0000 7f4172600640 1 -- 172.21.15.90:0/3380546872 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f416c06ccd0 con 0x7f416c0ae730 2024-04-25T01:16:39.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.180+0000 7f4168c00640 1 -- 172.21.15.90:0/3380546872 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18482 (secure 0 0 0) 0x7f4158058610 con 0x7f416c0ae730 2024-04-25T01:16:39.164 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:16:39.164 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":26,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:16:27.092855+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:16:20.094512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:16:21.078268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:16:24.670642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:16:39.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.182+0000 7f4172600640 1 -- 172.21.15.90:0/3380546872 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f414c06be30 msgr2=0x7f414c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:39.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.182+0000 7f4172600640 1 --2- 172.21.15.90:0/3380546872 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f414c06be30 0x7f414c06e2f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f416c159da0 tx=0x7f4160009040 comp rx=0 tx=0).stop 2024-04-25T01:16:39.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.182+0000 7f4172600640 1 -- 172.21.15.90:0/3380546872 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f416c0ae730 msgr2=0x7f416c1443e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:39.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.182+0000 7f4172600640 1 --2- 172.21.15.90:0/3380546872 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f416c0ae730 0x7f416c1443e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f415800c430 tx=0x7f415800c8f0 comp rx=0 tx=0).stop 2024-04-25T01:16:39.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.183+0000 7f4172600640 1 -- 172.21.15.90:0/3380546872 shutdown_connections 2024-04-25T01:16:39.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.183+0000 7f4172600640 1 --2- 172.21.15.90:0/3380546872 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f414c06be30 0x7f414c06e2f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:39.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.183+0000 7f4172600640 1 --2- 172.21.15.90:0/3380546872 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f416c15ca10 0x7f416c1503d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:39.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.183+0000 7f4172600640 1 --2- 172.21.15.90:0/3380546872 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f416c0ae730 0x7f416c1443e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:39.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.183+0000 7f4172600640 1 --2- 172.21.15.90:0/3380546872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f416c06b120 0x7f416c156930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:39.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.183+0000 7f4172600640 1 -- 172.21.15.90:0/3380546872 >> 172.21.15.90:0/3380546872 conn(0x7f416c0b1660 msgr2=0x7f416c06b500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:39.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.183+0000 7f4172600640 1 -- 172.21.15.90:0/3380546872 shutdown_connections 2024-04-25T01:16:39.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.183+0000 7f4172600640 1 -- 172.21.15.90:0/3380546872 wait complete. 2024-04-25T01:16:39.185 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:16:15.894126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:16:20.094512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:16:21.078268+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:16:24.670642+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:16:39.185 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:16:39.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.407+0000 7f12d5600640 1 Processor -- start 2024-04-25T01:16:39.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.408+0000 7f12d5600640 1 -- start start 2024-04-25T01:16:39.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.409+0000 7f12d5600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f12d01478a0 0x7f12d0072b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:39.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.409+0000 7f12d5600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f12d006ec40 0x7f12d006b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:39.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.409+0000 7f12d5600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12d006bbc0 con 0x7f12d006b120 2024-04-25T01:16:39.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.409+0000 7f12d5600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f12d0071da0 con 0x7f12d01478a0 2024-04-25T01:16:39.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.409+0000 7f12d5600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f12d0072080 con 0x7f12d006ec40 2024-04-25T01:16:39.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.410+0000 7f12cea00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f12d01478a0 0x7f12d0072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:39.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.410+0000 7f12c6000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f12d006b120 0x7f12d006e530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56486/0 (socket says 172.21.15.90:56486) 2024-04-25T01:16:39.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.410+0000 7f12c6000640 1 -- 172.21.15.90:0/1776049807 learned_addr learned my addr 172.21.15.90:0/1776049807 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:39.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.410+0000 7f12cf400640 1 --2- 172.21.15.90:0/1776049807 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f12d006ec40 0x7f12d006b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:39.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.410+0000 7f12cea00640 1 -- 172.21.15.90:0/1776049807 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f12d006ec40 msgr2=0x7f12d006b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:39.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.410+0000 7f12cea00640 1 --2- 172.21.15.90:0/1776049807 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f12d006ec40 0x7f12d006b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:39.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.410+0000 7f12cea00640 1 -- 172.21.15.90:0/1776049807 >> v1:172.21.15.90:6789/0 conn(0x7f12d006b120 legacy=0x7f12d006e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:39.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.410+0000 7f12cea00640 1 -- 172.21.15.90:0/1776049807 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12d0072360 con 0x7f12d01478a0 2024-04-25T01:16:39.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.410+0000 7f12ce000640 1 -- 172.21.15.90:0/1776049807 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2822093804 0 0) 0x7f12d006bbc0 con 0x7f12d006b120 2024-04-25T01:16:39.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.411+0000 7f12cea00640 1 --2- 172.21.15.90:0/1776049807 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12d01478a0 0x7f12d0072b60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f12c0009d90 tx=0x7f12c002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4b62f3f16da3f7a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:39.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.412+0000 7f12ce000640 1 -- 172.21.15.90:0/1776049807 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c0007740 con 0x7f12d01478a0 2024-04-25T01:16:39.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.412+0000 7f12ce000640 1 -- 172.21.15.90:0/1776049807 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12c00078e0 con 0x7f12d01478a0 2024-04-25T01:16:39.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.412+0000 7f12ce000640 1 -- 172.21.15.90:0/1776049807 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c0007cb0 con 0x7f12d01478a0 2024-04-25T01:16:39.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.412+0000 7f12d5600640 1 -- 172.21.15.90:0/1776049807 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12d01478a0 msgr2=0x7f12d0072b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:39.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.412+0000 7f12d5600640 1 --2- 172.21.15.90:0/1776049807 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12d01478a0 0x7f12d0072b60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f12c0009d90 tx=0x7f12c002a660 comp rx=0 tx=0).stop 2024-04-25T01:16:39.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.412+0000 7f12d5600640 1 -- 172.21.15.90:0/1776049807 shutdown_connections 2024-04-25T01:16:39.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.412+0000 7f12d5600640 1 --2- 172.21.15.90:0/1776049807 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f12d006ec40 0x7f12d006b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:39.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.412+0000 7f12d5600640 1 --2- 172.21.15.90:0/1776049807 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12d01478a0 0x7f12d0072b60 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:39.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.412+0000 7f12d5600640 1 -- 172.21.15.90:0/1776049807 >> 172.21.15.90:0/1776049807 conn(0x7f12d00d0bf0 msgr2=0x7f12d00d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:39.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.412+0000 7f12d5600640 1 -- 172.21.15.90:0/1776049807 shutdown_connections 2024-04-25T01:16:39.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.412+0000 7f12d5600640 1 -- 172.21.15.90:0/1776049807 wait complete. 2024-04-25T01:16:39.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.414+0000 7f12d5600640 1 Processor -- start 2024-04-25T01:16:39.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.414+0000 7f12d5600640 1 -- start start 2024-04-25T01:16:39.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.415+0000 7f12d5600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12d006b120 0x7f12d013f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:39.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.415+0000 7f12d5600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f12d006ec40 0x7f12d00b5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:39.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.415+0000 7f12d5600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12d01478a0 0x7f12d00b5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:39.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.415+0000 7f12d5600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f12d0162900 con 0x7f12d006b120 2024-04-25T01:16:39.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.415+0000 7f12d5600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f12d00cef60 con 0x7f12d01478a0 2024-04-25T01:16:39.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.415+0000 7f12d5600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f12d01405b0 con 0x7f12d006ec40 2024-04-25T01:16:39.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.416+0000 7f12cea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12d006b120 0x7f12d013f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:39.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.416+0000 7f12cea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12d006b120 0x7f12d013f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51784/0 (socket says 172.21.15.90:51784) 2024-04-25T01:16:39.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.416+0000 7f12cea00640 1 -- 172.21.15.90:0/2927931448 learned_addr learned my addr 172.21.15.90:0/2927931448 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:39.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.416+0000 7f12c6000640 1 --2- 172.21.15.90:0/2927931448 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f12d006ec40 0x7f12d00b5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:39.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.416+0000 7f12cf400640 1 --2- 172.21.15.90:0/2927931448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12d01478a0 0x7f12d00b5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:39.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.416+0000 7f12cea00640 1 -- 172.21.15.90:0/2927931448 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f12d006ec40 msgr2=0x7f12d00b5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:39.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.416+0000 7f12cea00640 1 --2- 172.21.15.90:0/2927931448 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f12d006ec40 0x7f12d00b5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:39.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.416+0000 7f12cea00640 1 -- 172.21.15.90:0/2927931448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12d01478a0 msgr2=0x7f12d00b5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:39.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.416+0000 7f12cea00640 1 --2- 172.21.15.90:0/2927931448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12d01478a0 0x7f12d00b5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:39.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.416+0000 7f12cea00640 1 -- 172.21.15.90:0/2927931448 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12d00b5f00 con 0x7f12d006b120 2024-04-25T01:16:39.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.417+0000 7f12cea00640 1 --2- 172.21.15.90:0/2927931448 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12d006b120 0x7f12d013f720 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f12c0009eb0 tx=0x7f12c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:39.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.417+0000 7f12c7400640 1 -- 172.21.15.90:0/2927931448 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c003f070 con 0x7f12d006b120 2024-04-25T01:16:39.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.417+0000 7f12c7400640 1 -- 172.21.15.90:0/2927931448 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12c0031c40 con 0x7f12d006b120 2024-04-25T01:16:39.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.417+0000 7f12d5600640 1 -- 172.21.15.90:0/2927931448 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12d00b61e0 con 0x7f12d006b120 2024-04-25T01:16:39.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.417+0000 7f12c7400640 1 -- 172.21.15.90:0/2927931448 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c0043410 con 0x7f12d006b120 2024-04-25T01:16:39.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.417+0000 7f12d5600640 1 -- 172.21.15.90:0/2927931448 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12d0140240 con 0x7f12d006b120 2024-04-25T01:16:39.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.418+0000 7f12d5600640 1 -- 172.21.15.90:0/2927931448 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12d0068180 con 0x7f12d006b120 2024-04-25T01:16:39.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.419+0000 7f12c7400640 1 -- 172.21.15.90:0/2927931448 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f12c00435b0 con 0x7f12d006b120 2024-04-25T01:16:39.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.420+0000 7f12c7400640 1 --2- 172.21.15.90:0/2927931448 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f12ac06be30 0x7f12ac06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:39.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.420+0000 7f12c7400640 1 -- 172.21.15.90:0/2927931448 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f12c00ac710 con 0x7f12d006b120 2024-04-25T01:16:39.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.420+0000 7f12c6000640 1 --2- 172.21.15.90:0/2927931448 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f12ac06be30 0x7f12ac06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:39.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.421+0000 7f12c6000640 1 --2- 172.21.15.90:0/2927931448 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f12ac06be30 0x7f12ac06e2f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f12d01613a0 tx=0x7f12b8007b70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:39.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.425+0000 7f12c7400640 1 -- 172.21.15.90:0/2927931448 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f12c007ace0 con 0x7f12d006b120 2024-04-25T01:16:39.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.623+0000 7f12d5600640 1 -- 172.21.15.90:0/2927931448 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f12d006be00 con 0x7f12d006b120 2024-04-25T01:16:39.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.623+0000 7f12c7400640 1 -- 172.21.15.90:0/2927931448 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f12c007a5e0 con 0x7f12d006b120 2024-04-25T01:16:39.607 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T01:16:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.626+0000 7f12d5600640 1 -- 172.21.15.90:0/2927931448 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f12ac06be30 msgr2=0x7f12ac06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:39.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.626+0000 7f12d5600640 1 --2- 172.21.15.90:0/2927931448 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f12ac06be30 0x7f12ac06e2f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f12d01613a0 tx=0x7f12b8007b70 comp rx=0 tx=0).stop 2024-04-25T01:16:39.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.626+0000 7f12d5600640 1 -- 172.21.15.90:0/2927931448 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12d006b120 msgr2=0x7f12d013f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:39.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.626+0000 7f12d5600640 1 --2- 172.21.15.90:0/2927931448 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12d006b120 0x7f12d013f720 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f12c0009eb0 tx=0x7f12c0004a40 comp rx=0 tx=0).stop 2024-04-25T01:16:39.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.626+0000 7f12d5600640 1 -- 172.21.15.90:0/2927931448 shutdown_connections 2024-04-25T01:16:39.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.626+0000 7f12d5600640 1 --2- 172.21.15.90:0/2927931448 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f12ac06be30 0x7f12ac06e2f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:39.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.626+0000 7f12d5600640 1 --2- 172.21.15.90:0/2927931448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12d01478a0 0x7f12d00b5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:39.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.626+0000 7f12d5600640 1 --2- 172.21.15.90:0/2927931448 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f12d006ec40 0x7f12d00b5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:39.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.626+0000 7f12d5600640 1 --2- 172.21.15.90:0/2927931448 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12d006b120 0x7f12d013f720 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:39.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.626+0000 7f12d5600640 1 -- 172.21.15.90:0/2927931448 >> 172.21.15.90:0/2927931448 conn(0x7f12d00d0bf0 msgr2=0x7f12d0074750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:39.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.626+0000 7f12d5600640 1 -- 172.21.15.90:0/2927931448 shutdown_connections 2024-04-25T01:16:39.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.626+0000 7f12d5600640 1 -- 172.21.15.90:0/2927931448 wait complete. 2024-04-25T01:16:39.629 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:16:39.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.851+0000 7fa76dc00640 1 Processor -- start 2024-04-25T01:16:39.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.851+0000 7fa76dc00640 1 -- start start 2024-04-25T01:16:39.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.853+0000 7fa76dc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa7680722e0 0x7fa76806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:39.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.853+0000 7fa76dc00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa76815fc40 0x7fa7681600c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:39.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.853+0000 7fa76dc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7680cedc0 0x7fa7680cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:39.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.853+0000 7fa76dc00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fa768160aa0 con 0x7fa7680722e0 2024-04-25T01:16:39.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.853+0000 7fa76dc00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa76806da40 con 0x7fa76815fc40 2024-04-25T01:16:39.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.853+0000 7fa76dc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa76806cec0 con 0x7fa7680cedc0 2024-04-25T01:16:39.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.853+0000 7fa766a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa7680722e0 0x7fa76806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:39.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.853+0000 7fa766a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa7680722e0 0x7fa76806ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51790/0 (socket says 172.21.15.90:51790) 2024-04-25T01:16:39.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.853+0000 7fa766a00640 1 -- 172.21.15.90:0/3501602841 learned_addr learned my addr 172.21.15.90:0/3501602841 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:39.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.854+0000 7fa75e000640 1 --2- 172.21.15.90:0/3501602841 >> v2:172.21.15.103:3300/0 conn(0x7fa76815fc40 0x7fa7681600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:39.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.854+0000 7fa767400640 1 --2- 172.21.15.90:0/3501602841 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7680cedc0 0x7fa7680cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:39.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.854+0000 7fa766a00640 1 -- 172.21.15.90:0/3501602841 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7680cedc0 msgr2=0x7fa7680cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:39.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.854+0000 7fa766a00640 1 --2- 172.21.15.90:0/3501602841 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7680cedc0 0x7fa7680cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:39.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.854+0000 7fa766a00640 1 -- 172.21.15.90:0/3501602841 >> v2:172.21.15.103:3300/0 conn(0x7fa76815fc40 msgr2=0x7fa7681600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:39.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.854+0000 7fa766a00640 1 --2- 172.21.15.90:0/3501602841 >> v2:172.21.15.103:3300/0 conn(0x7fa76815fc40 0x7fa7681600c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:39.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.854+0000 7fa766a00640 1 -- 172.21.15.90:0/3501602841 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa76806e410 con 0x7fa7680722e0 2024-04-25T01:16:39.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.855+0000 7fa766a00640 1 --2- 172.21.15.90:0/3501602841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7680722e0 0x7fa76806ded0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa7680ce0a0 tx=0x7fa75802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecb58a0c5222120e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:39.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.855+0000 7fa766000640 1 -- 172.21.15.90:0/3501602841 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa758037070 con 0x7fa7680722e0 2024-04-25T01:16:39.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.855+0000 7fa766000640 1 -- 172.21.15.90:0/3501602841 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa758032440 con 0x7fa7680722e0 2024-04-25T01:16:39.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.855+0000 7fa766000640 1 -- 172.21.15.90:0/3501602841 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa758032810 con 0x7fa7680722e0 2024-04-25T01:16:39.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.856+0000 7fa76dc00640 1 -- 172.21.15.90:0/3501602841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7680722e0 msgr2=0x7fa76806ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:39.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.856+0000 7fa76dc00640 1 --2- 172.21.15.90:0/3501602841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7680722e0 0x7fa76806ded0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa7680ce0a0 tx=0x7fa75802a140 comp rx=0 tx=0).stop 2024-04-25T01:16:39.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.856+0000 7fa76dc00640 1 -- 172.21.15.90:0/3501602841 shutdown_connections 2024-04-25T01:16:39.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.856+0000 7fa76dc00640 1 --2- 172.21.15.90:0/3501602841 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7680cedc0 0x7fa7680cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:39.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.856+0000 7fa76dc00640 1 --2- 172.21.15.90:0/3501602841 >> v2:172.21.15.103:3300/0 conn(0x7fa76815fc40 0x7fa7681600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:39.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.856+0000 7fa76dc00640 1 --2- 172.21.15.90:0/3501602841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7680722e0 0x7fa76806ded0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:39.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.856+0000 7fa76dc00640 1 -- 172.21.15.90:0/3501602841 >> 172.21.15.90:0/3501602841 conn(0x7fa76806b720 msgr2=0x7fa7680b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:39.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.856+0000 7fa76dc00640 1 -- 172.21.15.90:0/3501602841 shutdown_connections 2024-04-25T01:16:39.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.856+0000 7fa76dc00640 1 -- 172.21.15.90:0/3501602841 wait complete. 2024-04-25T01:16:39.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.858+0000 7fa76dc00640 1 Processor -- start 2024-04-25T01:16:39.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.858+0000 7fa76dc00640 1 -- start start 2024-04-25T01:16:39.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.859+0000 7fa76dc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7680722e0 0x7fa7681f85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:39.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.859+0000 7fa76dc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7680cedc0 0x7fa7680b2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:39.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.859+0000 7fa76dc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa76815fc40 0x7fa7681f4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:39.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.859+0000 7fa76dc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa76806d3f0 con 0x7fa7680722e0 2024-04-25T01:16:39.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.859+0000 7fa76dc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa7680cf770 con 0x7fa76815fc40 2024-04-25T01:16:39.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.859+0000 7fa76dc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa76806d6d0 con 0x7fa7680cedc0 2024-04-25T01:16:39.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.859+0000 7fa766a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7680722e0 0x7fa7681f85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:39.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.859+0000 7fa766a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7680722e0 0x7fa7681f85b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51800/0 (socket says 172.21.15.90:51800) 2024-04-25T01:16:39.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.859+0000 7fa766a00640 1 -- 172.21.15.90:0/3414278748 learned_addr learned my addr 172.21.15.90:0/3414278748 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:39.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.860+0000 7fa766a00640 1 -- 172.21.15.90:0/3414278748 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7680cedc0 msgr2=0x7fa7680b2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:39.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.860+0000 7fa766a00640 1 --2- 172.21.15.90:0/3414278748 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7680cedc0 0x7fa7680b2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:39.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.860+0000 7fa766a00640 1 -- 172.21.15.90:0/3414278748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa76815fc40 msgr2=0x7fa7681f4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:39.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.860+0000 7fa766a00640 1 --2- 172.21.15.90:0/3414278748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa76815fc40 0x7fa7681f4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:39.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.860+0000 7fa766a00640 1 -- 172.21.15.90:0/3414278748 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7680b2e70 con 0x7fa7680722e0 2024-04-25T01:16:39.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.860+0000 7fa766a00640 1 --2- 172.21.15.90:0/3414278748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7680722e0 0x7fa7681f85b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa75802a180 tx=0x7fa7580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:39.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.860+0000 7fa75f400640 1 -- 172.21.15.90:0/3414278748 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa758041020 con 0x7fa7680722e0 2024-04-25T01:16:39.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.860+0000 7fa75f400640 1 -- 172.21.15.90:0/3414278748 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7580306c0 con 0x7fa7680722e0 2024-04-25T01:16:39.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.860+0000 7fa76dc00640 1 -- 172.21.15.90:0/3414278748 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7680b3110 con 0x7fa7680722e0 2024-04-25T01:16:39.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.860+0000 7fa75f400640 1 -- 172.21.15.90:0/3414278748 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa758037040 con 0x7fa7680722e0 2024-04-25T01:16:39.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.860+0000 7fa76dc00640 1 -- 172.21.15.90:0/3414278748 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7680b36d0 con 0x7fa7680722e0 2024-04-25T01:16:39.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.862+0000 7fa76dc00640 1 -- 172.21.15.90:0/3414278748 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa768068fb0 con 0x7fa7680722e0 2024-04-25T01:16:39.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.863+0000 7fa75f400640 1 -- 172.21.15.90:0/3414278748 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa758044020 con 0x7fa7680722e0 2024-04-25T01:16:39.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.863+0000 7fa75f400640 1 --2- 172.21.15.90:0/3414278748 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa74406be30 0x7fa74406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:39.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.864+0000 7fa75f400640 1 -- 172.21.15.90:0/3414278748 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fa7580aba80 con 0x7fa7680722e0 2024-04-25T01:16:39.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.864+0000 7fa75e000640 1 --2- 172.21.15.90:0/3414278748 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa74406be30 0x7fa74406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:39.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.864+0000 7fa75e000640 1 --2- 172.21.15.90:0/3414278748 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa74406be30 0x7fa74406e2f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa7681e6f00 tx=0x7fa750008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:39.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:39.868+0000 7fa75f400640 1 -- 172.21.15.90:0/3414278748 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa7580ade20 con 0x7fa7680722e0 2024-04-25T01:16:40.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.063+0000 7fa76dc00640 1 -- 172.21.15.90:0/3414278748 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa768072c90 con 0x7fa7680722e0 2024-04-25T01:16:40.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.063+0000 7fa75f400640 1 -- 172.21.15.90:0/3414278748 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7fa75804b030 con 0x7fa7680722e0 2024-04-25T01:16:40.048 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:16:40.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.066+0000 7fa76dc00640 1 -- 172.21.15.90:0/3414278748 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa74406be30 msgr2=0x7fa74406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:40.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.066+0000 7fa76dc00640 1 --2- 172.21.15.90:0/3414278748 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa74406be30 0x7fa74406e2f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa7681e6f00 tx=0x7fa750008040 comp rx=0 tx=0).stop 2024-04-25T01:16:40.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.066+0000 7fa76dc00640 1 -- 172.21.15.90:0/3414278748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7680722e0 msgr2=0x7fa7681f85b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:40.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.066+0000 7fa76dc00640 1 --2- 172.21.15.90:0/3414278748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7680722e0 0x7fa7681f85b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa75802a180 tx=0x7fa7580046c0 comp rx=0 tx=0).stop 2024-04-25T01:16:40.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.066+0000 7fa76dc00640 1 -- 172.21.15.90:0/3414278748 shutdown_connections 2024-04-25T01:16:40.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.066+0000 7fa76dc00640 1 --2- 172.21.15.90:0/3414278748 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa74406be30 0x7fa74406e2f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:40.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.066+0000 7fa76dc00640 1 --2- 172.21.15.90:0/3414278748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa76815fc40 0x7fa7681f4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:40.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.066+0000 7fa76dc00640 1 --2- 172.21.15.90:0/3414278748 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa7680cedc0 0x7fa7680b2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:40.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.066+0000 7fa76dc00640 1 --2- 172.21.15.90:0/3414278748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7680722e0 0x7fa7681f85b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:40.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.066+0000 7fa76dc00640 1 -- 172.21.15.90:0/3414278748 >> 172.21.15.90:0/3414278748 conn(0x7fa76806b720 msgr2=0x7fa768070b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:40.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.066+0000 7fa76dc00640 1 -- 172.21.15.90:0/3414278748 shutdown_connections 2024-04-25T01:16:40.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.067+0000 7fa76dc00640 1 -- 172.21.15.90:0/3414278748 wait complete. 2024-04-25T01:16:40.068 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:16:40.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.282+0000 7f2361200640 1 Processor -- start 2024-04-25T01:16:40.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.282+0000 7f2361200640 1 -- start start 2024-04-25T01:16:40.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.284+0000 7f2361200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f235c0b3c60 0x7f235c06df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:40.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.284+0000 7f2361200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f235c06d140 con 0x7f235c158a60 2024-04-25T01:16:40.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.284+0000 7f2361200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f235c06d420 con 0x7f235c158680 2024-04-25T01:16:40.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.284+0000 7f2361200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f235c06d700 con 0x7f235c0b3c60 2024-04-25T01:16:40.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.284+0000 7f2359400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f235c158a60 0x7f235c0b3550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56496/0 (socket says 172.21.15.90:56496) 2024-04-25T01:16:40.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.284+0000 7f2359400640 1 -- 172.21.15.90:0/699737310 learned_addr learned my addr 172.21.15.90:0/699737310 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:40.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.284+0000 7f235a800640 1 --2- 172.21.15.90:0/699737310 >> v2:172.21.15.103:3300/0 conn(0x7f235c0b3c60 0x7f235c06df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:40.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.284+0000 7f2358a00640 1 -- 172.21.15.90:0/699737310 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1213590247 0 0) 0x7f235c06d140 con 0x7f235c158a60 2024-04-25T01:16:40.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.285+0000 7f2358a00640 1 -- 172.21.15.90:0/699737310 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2340003520 con 0x7f235c158a60 2024-04-25T01:16:40.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.285+0000 7f2358a00640 1 -- 172.21.15.90:0/699737310 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2937753638 0 0) 0x7f2340003520 con 0x7f235c158a60 2024-04-25T01:16:40.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.285+0000 7f2358a00640 1 -- 172.21.15.90:0/699737310 >> v2:172.21.15.103:3300/0 conn(0x7f235c0b3c60 msgr2=0x7f235c06df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:40.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.285+0000 7f2358a00640 1 --2- 172.21.15.90:0/699737310 >> v2:172.21.15.103:3300/0 conn(0x7f235c0b3c60 0x7f235c06df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:40.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.285+0000 7f2358a00640 1 -- 172.21.15.90:0/699737310 >> v1:172.21.15.103:6789/0 conn(0x7f235c158680 legacy=0x7f235c06e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:40.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.285+0000 7f2358a00640 1 -- 172.21.15.90:0/699737310 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f235c06d9e0 con 0x7f235c158a60 2024-04-25T01:16:40.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.285+0000 7f2358a00640 1 -- 172.21.15.90:0/699737310 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f2348001670 con 0x7f235c158a60 2024-04-25T01:16:40.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.285+0000 7f2358a00640 1 -- 172.21.15.90:0/699737310 >> v1:172.21.15.90:6789/0 conn(0x7f235c158a60 legacy=0x7f235c0b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:40.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.285+0000 7f2358a00640 1 --2- 172.21.15.90:0/699737310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23400039b0 0x7f2340003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:40.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.285+0000 7f2358a00640 1 --2- 172.21.15.90:0/699737310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2340004800 0x7f2340004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:40.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.285+0000 7f2358a00640 1 --2- 172.21.15.90:0/699737310 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23400051c0 0x7f234000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:40.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.285+0000 7f2358a00640 1 -- 172.21.15.90:0/699737310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f235c06d700 con 0x7f2340004800 2024-04-25T01:16:40.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.285+0000 7f2358a00640 1 -- 172.21.15.90:0/699737310 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f234000c160 con 0x7f23400039b0 2024-04-25T01:16:40.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.285+0000 7f2358a00640 1 -- 172.21.15.90:0/699737310 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f234000c360 con 0x7f23400051c0 2024-04-25T01:16:40.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.285+0000 7f2359400640 1 --2- 172.21.15.90:0/699737310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2340004800 0x7f2340004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:40.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.286+0000 7f2359e00640 1 --2- 172.21.15.90:0/699737310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23400039b0 0x7f2340003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:40.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.286+0000 7f235a800640 1 --2- 172.21.15.90:0/699737310 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23400051c0 0x7f234000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:40.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.286+0000 7f2359400640 1 -- 172.21.15.90:0/699737310 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23400051c0 msgr2=0x7f234000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:40.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.286+0000 7f2359400640 1 --2- 172.21.15.90:0/699737310 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f23400051c0 0x7f234000b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:40.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.286+0000 7f2359400640 1 -- 172.21.15.90:0/699737310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23400039b0 msgr2=0x7f2340003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:40.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.286+0000 7f2359400640 1 --2- 172.21.15.90:0/699737310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23400039b0 0x7f2340003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:40.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.286+0000 7f2359400640 1 -- 172.21.15.90:0/699737310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2340003520 con 0x7f2340004800 2024-04-25T01:16:40.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.286+0000 7f235a800640 1 -- 172.21.15.90:0/699737310 reap_dead start 2024-04-25T01:16:40.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.287+0000 7f2359400640 1 --2- 172.21.15.90:0/699737310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2340004800 0x7f2340004c80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f234800c210 tx=0x7f2340005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f203197d8b17c4b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:40.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.287+0000 7f2358a00640 1 -- 172.21.15.90:0/699737310 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2348007cf0 con 0x7f2340004800 2024-04-25T01:16:40.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.287+0000 7f2358a00640 1 -- 172.21.15.90:0/699737310 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2348005af0 con 0x7f2340004800 2024-04-25T01:16:40.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.287+0000 7f2358a00640 1 -- 172.21.15.90:0/699737310 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2348007070 con 0x7f2340004800 2024-04-25T01:16:40.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.288+0000 7f2361200640 1 -- 172.21.15.90:0/699737310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2340004800 msgr2=0x7f2340004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:40.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.288+0000 7f2361200640 1 --2- 172.21.15.90:0/699737310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2340004800 0x7f2340004c80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f234800c210 tx=0x7f2340005830 comp rx=0 tx=0).stop 2024-04-25T01:16:40.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.288+0000 7f2361200640 1 -- 172.21.15.90:0/699737310 shutdown_connections 2024-04-25T01:16:40.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.288+0000 7f2361200640 1 --2- 172.21.15.90:0/699737310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2340004800 0x7f2340004c80 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:40.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.288+0000 7f2361200640 1 -- 172.21.15.90:0/699737310 >> 172.21.15.90:0/699737310 conn(0x7f235c06b720 msgr2=0x7f235c0b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:40.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.288+0000 7f2361200640 1 -- 172.21.15.90:0/699737310 shutdown_connections 2024-04-25T01:16:40.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.288+0000 7f2361200640 1 -- 172.21.15.90:0/699737310 wait complete. 2024-04-25T01:16:40.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.290+0000 7f2361200640 1 Processor -- start 2024-04-25T01:16:40.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.290+0000 7f2361200640 1 -- start start 2024-04-25T01:16:40.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.291+0000 7f2361200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2340004800 0x7f235c1f4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:40.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.291+0000 7f2361200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f235c1493f0 0x7f235c15e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:40.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.291+0000 7f2361200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f235c149870 0x7f235c0714c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:40.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.291+0000 7f2361200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f235c15f640 con 0x7f235c1493f0 2024-04-25T01:16:40.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.291+0000 7f2361200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f235c15f4c0 con 0x7f2340004800 2024-04-25T01:16:40.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.291+0000 7f2361200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f235c1551f0 con 0x7f235c149870 2024-04-25T01:16:40.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.291+0000 7f2359400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f235c1493f0 0x7f235c15e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:40.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.291+0000 7f2359400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f235c1493f0 0x7f235c15e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51824/0 (socket says 172.21.15.90:51824) 2024-04-25T01:16:40.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.292+0000 7f2359400640 1 -- 172.21.15.90:0/2808092318 learned_addr learned my addr 172.21.15.90:0/2808092318 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:40.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.292+0000 7f2359e00640 1 --2- 172.21.15.90:0/2808092318 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2340004800 0x7f235c1f4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:40.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.292+0000 7f235a800640 1 --2- 172.21.15.90:0/2808092318 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f235c149870 0x7f235c0714c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:40.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.292+0000 7f2359400640 1 -- 172.21.15.90:0/2808092318 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f235c149870 msgr2=0x7f235c0714c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:40.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.292+0000 7f2359400640 1 --2- 172.21.15.90:0/2808092318 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f235c149870 0x7f235c0714c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:40.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.292+0000 7f2359400640 1 -- 172.21.15.90:0/2808092318 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2340004800 msgr2=0x7f235c1f4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:40.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.292+0000 7f2359400640 1 --2- 172.21.15.90:0/2808092318 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2340004800 0x7f235c1f4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:40.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.292+0000 7f2359400640 1 -- 172.21.15.90:0/2808092318 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f235c155fc0 con 0x7f235c1493f0 2024-04-25T01:16:40.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.292+0000 7f2359400640 1 --2- 172.21.15.90:0/2808092318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f235c1493f0 0x7f235c15e990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2348000f60 tx=0x7f235c06d5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:40.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.293+0000 7f2352000640 1 -- 172.21.15.90:0/2808092318 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2348041020 con 0x7f235c1493f0 2024-04-25T01:16:40.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.293+0000 7f2352000640 1 -- 172.21.15.90:0/2808092318 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2348038040 con 0x7f235c1493f0 2024-04-25T01:16:40.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.293+0000 7f2352000640 1 -- 172.21.15.90:0/2808092318 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2348007270 con 0x7f235c1493f0 2024-04-25T01:16:40.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.293+0000 7f2361200640 1 -- 172.21.15.90:0/2808092318 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f235c155440 con 0x7f235c1493f0 2024-04-25T01:16:40.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.293+0000 7f2361200640 1 -- 172.21.15.90:0/2808092318 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f235c155a00 con 0x7f235c1493f0 2024-04-25T01:16:40.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.294+0000 7f2361200640 1 -- 172.21.15.90:0/2808092318 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f235c068fb0 con 0x7f235c1493f0 2024-04-25T01:16:40.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.295+0000 7f2352000640 1 -- 172.21.15.90:0/2808092318 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2348044020 con 0x7f235c1493f0 2024-04-25T01:16:40.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.296+0000 7f2352000640 1 --2- 172.21.15.90:0/2808092318 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f233c06be30 0x7f233c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:40.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.296+0000 7f2352000640 1 -- 172.21.15.90:0/2808092318 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f23480ab5c0 con 0x7f235c1493f0 2024-04-25T01:16:40.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.296+0000 7f2359e00640 1 --2- 172.21.15.90:0/2808092318 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f233c06be30 0x7f233c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:40.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.297+0000 7f2359e00640 1 --2- 172.21.15.90:0/2808092318 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f233c06be30 0x7f233c06e2f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f235c153580 tx=0x7f234c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:40.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.300+0000 7f2352000640 1 -- 172.21.15.90:0/2808092318 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2348079b90 con 0x7f235c1493f0 2024-04-25T01:16:40.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.497+0000 7f2361200640 1 -- 172.21.15.90:0/2808092318 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f235c0ae0d0 con 0x7f235c1493f0 2024-04-25T01:16:40.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.498+0000 7f2352000640 1 -- 172.21.15.90:0/2808092318 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f2348079490 con 0x7f235c1493f0 2024-04-25T01:16:40.482 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-25T01:16:40.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.500+0000 7f2361200640 1 -- 172.21.15.90:0/2808092318 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f233c06be30 msgr2=0x7f233c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:40.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.500+0000 7f2361200640 1 --2- 172.21.15.90:0/2808092318 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f233c06be30 0x7f233c06e2f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f235c153580 tx=0x7f234c009040 comp rx=0 tx=0).stop 2024-04-25T01:16:40.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.500+0000 7f2361200640 1 -- 172.21.15.90:0/2808092318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f235c1493f0 msgr2=0x7f235c15e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:40.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.500+0000 7f2361200640 1 --2- 172.21.15.90:0/2808092318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f235c1493f0 0x7f235c15e990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2348000f60 tx=0x7f235c06d5f0 comp rx=0 tx=0).stop 2024-04-25T01:16:40.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.500+0000 7f2361200640 1 -- 172.21.15.90:0/2808092318 shutdown_connections 2024-04-25T01:16:40.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.500+0000 7f2361200640 1 --2- 172.21.15.90:0/2808092318 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f233c06be30 0x7f233c06e2f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:40.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.500+0000 7f2361200640 1 --2- 172.21.15.90:0/2808092318 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f235c149870 0x7f235c0714c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:40.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.500+0000 7f2361200640 1 --2- 172.21.15.90:0/2808092318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f235c1493f0 0x7f235c15e990 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:40.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.500+0000 7f2361200640 1 --2- 172.21.15.90:0/2808092318 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2340004800 0x7f235c1f4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:40.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.500+0000 7f2361200640 1 -- 172.21.15.90:0/2808092318 >> 172.21.15.90:0/2808092318 conn(0x7f235c06b720 msgr2=0x7f235c06e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:40.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.501+0000 7f2361200640 1 -- 172.21.15.90:0/2808092318 shutdown_connections 2024-04-25T01:16:40.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.501+0000 7f2361200640 1 -- 172.21.15.90:0/2808092318 wait complete. 2024-04-25T01:16:40.503 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:16:40.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.724+0000 7f1a7fe00640 1 Processor -- start 2024-04-25T01:16:40.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.724+0000 7f1a7fe00640 1 -- start start 2024-04-25T01:16:40.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.726+0000 7f1a7fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1a8006e5a0 0x7f1a800cf7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:40.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.726+0000 7f1a7fe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1a800b2880 0x7f1a800cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:40.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.726+0000 7f1a7fe00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a8015c200 con 0x7f1a801596c0 2024-04-25T01:16:40.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.726+0000 7f1a7fe00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f1a80159aa0 con 0x7f1a8006e5a0 2024-04-25T01:16:40.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.726+0000 7f1a7fe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1a8015c470 con 0x7f1a800b2880 2024-04-25T01:16:40.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.726+0000 7f1a76000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1a8006e5a0 0x7f1a800cf7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:40.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.726+0000 7f1a7ea00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f1a801596c0 0x7f1a8006de90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56508/0 (socket says 172.21.15.90:56508) 2024-04-25T01:16:40.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.726+0000 7f1a7ea00640 1 -- 172.21.15.90:0/4050457071 learned_addr learned my addr 172.21.15.90:0/4050457071 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:40.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.727+0000 7f1a7f400640 1 --2- 172.21.15.90:0/4050457071 >> v2:172.21.15.103:3300/0 conn(0x7f1a800b2880 0x7f1a800cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:40.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.727+0000 7f1a76000640 1 -- 172.21.15.90:0/4050457071 >> v2:172.21.15.103:3300/0 conn(0x7f1a800b2880 msgr2=0x7f1a800cfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:40.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.727+0000 7f1a76000640 1 --2- 172.21.15.90:0/4050457071 >> v2:172.21.15.103:3300/0 conn(0x7f1a800b2880 0x7f1a800cfd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:40.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.727+0000 7f1a76000640 1 -- 172.21.15.90:0/4050457071 >> v1:172.21.15.90:6789/0 conn(0x7f1a801596c0 legacy=0x7f1a8006de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:40.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.727+0000 7f1a76000640 1 -- 172.21.15.90:0/4050457071 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a8015c750 con 0x7f1a8006e5a0 2024-04-25T01:16:40.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.727+0000 7f1a7f400640 1 --2- 172.21.15.90:0/4050457071 >> v2:172.21.15.103:3300/0 conn(0x7f1a800b2880 0x7f1a800cfd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:16:40.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.728+0000 7f1a76000640 1 --2- 172.21.15.90:0/4050457071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a8006e5a0 0x7f1a800cf7f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1a80072ef0 tx=0x7f1a6802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2d9b243f92907e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:40.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.728+0000 7f1a7e000640 1 -- 172.21.15.90:0/4050457071 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a68037070 con 0x7f1a8006e5a0 2024-04-25T01:16:40.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.728+0000 7f1a7e000640 1 -- 172.21.15.90:0/4050457071 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a68032440 con 0x7f1a8006e5a0 2024-04-25T01:16:40.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.728+0000 7f1a7e000640 1 -- 172.21.15.90:0/4050457071 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a68032810 con 0x7f1a8006e5a0 2024-04-25T01:16:40.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.729+0000 7f1a7fe00640 1 -- 172.21.15.90:0/4050457071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a8006e5a0 msgr2=0x7f1a800cf7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:40.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.729+0000 7f1a7fe00640 1 --2- 172.21.15.90:0/4050457071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a8006e5a0 0x7f1a800cf7f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1a80072ef0 tx=0x7f1a6802a1a0 comp rx=0 tx=0).stop 2024-04-25T01:16:40.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.729+0000 7f1a7fe00640 1 -- 172.21.15.90:0/4050457071 shutdown_connections 2024-04-25T01:16:40.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.729+0000 7f1a7fe00640 1 --2- 172.21.15.90:0/4050457071 >> v2:172.21.15.103:3300/0 conn(0x7f1a800b2880 0x7f1a800cfd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:40.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.729+0000 7f1a7fe00640 1 --2- 172.21.15.90:0/4050457071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a8006e5a0 0x7f1a800cf7f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:40.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.729+0000 7f1a7fe00640 1 -- 172.21.15.90:0/4050457071 >> 172.21.15.90:0/4050457071 conn(0x7f1a8006b120 msgr2=0x7f1a8006b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:40.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.729+0000 7f1a7fe00640 1 -- 172.21.15.90:0/4050457071 shutdown_connections 2024-04-25T01:16:40.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.729+0000 7f1a7fe00640 1 -- 172.21.15.90:0/4050457071 wait complete. 2024-04-25T01:16:40.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.731+0000 7f1a7fe00640 1 Processor -- start 2024-04-25T01:16:40.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.731+0000 7f1a7fe00640 1 -- start start 2024-04-25T01:16:40.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.732+0000 7f1a7fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a8006e5a0 0x7f1a8015ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:40.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.732+0000 7f1a7fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a800b2880 0x7f1a801461b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:40.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.732+0000 7f1a7fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a801596c0 0x7f1a80157d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:40.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.732+0000 7f1a7fe00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1a800d0270 con 0x7f1a8006e5a0 2024-04-25T01:16:40.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.732+0000 7f1a7fe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8015cd10 con 0x7f1a800b2880 2024-04-25T01:16:40.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.732+0000 7f1a7fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1a8015f770 con 0x7f1a801596c0 2024-04-25T01:16:40.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.732+0000 7f1a7ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a8006e5a0 0x7f1a8015ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:40.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.732+0000 7f1a7ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a8006e5a0 0x7f1a8015ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51852/0 (socket says 172.21.15.90:51852) 2024-04-25T01:16:40.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.732+0000 7f1a7ea00640 1 -- 172.21.15.90:0/3990984624 learned_addr learned my addr 172.21.15.90:0/3990984624 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:40.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.732+0000 7f1a7ea00640 1 -- 172.21.15.90:0/3990984624 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a801596c0 msgr2=0x7f1a80157d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:40.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.732+0000 7f1a7ea00640 1 --2- 172.21.15.90:0/3990984624 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a801596c0 0x7f1a80157d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:40.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.732+0000 7f1a7ea00640 1 -- 172.21.15.90:0/3990984624 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a800b2880 msgr2=0x7f1a801461b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:40.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.732+0000 7f1a76000640 1 --2- 172.21.15.90:0/3990984624 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a800b2880 0x7f1a801461b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:40.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.732+0000 7f1a7ea00640 1 --2- 172.21.15.90:0/3990984624 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a800b2880 0x7f1a801461b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:40.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.732+0000 7f1a7ea00640 1 -- 172.21.15.90:0/3990984624 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a801f5130 con 0x7f1a8006e5a0 2024-04-25T01:16:40.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.733+0000 7f1a7ea00640 1 --2- 172.21.15.90:0/3990984624 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a8006e5a0 0x7f1a8015ef60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1a7000caa0 tx=0x7f1a7000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:40.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.733+0000 7f1a77400640 1 -- 172.21.15.90:0/3990984624 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a70017070 con 0x7f1a8006e5a0 2024-04-25T01:16:40.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.733+0000 7f1a77400640 1 -- 172.21.15.90:0/3990984624 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a7000ac00 con 0x7f1a8006e5a0 2024-04-25T01:16:40.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.733+0000 7f1a77400640 1 -- 172.21.15.90:0/3990984624 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a70012400 con 0x7f1a8006e5a0 2024-04-25T01:16:40.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.733+0000 7f1a7fe00640 1 -- 172.21.15.90:0/3990984624 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a801f53d0 con 0x7f1a8006e5a0 2024-04-25T01:16:40.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.733+0000 7f1a7fe00640 1 -- 172.21.15.90:0/3990984624 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a801f5990 con 0x7f1a8006e5a0 2024-04-25T01:16:40.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.734+0000 7f1a7fe00640 1 -- 172.21.15.90:0/3990984624 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a80075350 con 0x7f1a8006e5a0 2024-04-25T01:16:40.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.735+0000 7f1a77400640 1 -- 172.21.15.90:0/3990984624 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1a7001a070 con 0x7f1a8006e5a0 2024-04-25T01:16:40.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.736+0000 7f1a77400640 1 --2- 172.21.15.90:0/3990984624 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1a5c06be30 0x7f1a5c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:40.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.736+0000 7f1a77400640 1 -- 172.21.15.90:0/3990984624 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f1a7008aef0 con 0x7f1a8006e5a0 2024-04-25T01:16:40.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.736+0000 7f1a76000640 1 --2- 172.21.15.90:0/3990984624 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1a5c06be30 0x7f1a5c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:40.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.737+0000 7f1a76000640 1 --2- 172.21.15.90:0/3990984624 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1a5c06be30 0x7f1a5c06e2f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1a68034000 tx=0x7f1a6802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:40.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.740+0000 7f1a77400640 1 -- 172.21.15.90:0/3990984624 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1a70059550 con 0x7f1a8006e5a0 2024-04-25T01:16:40.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.938+0000 7f1a7fe00640 1 -- 172.21.15.90:0/3990984624 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f1a8015ed40 con 0x7f1a8006e5a0 2024-04-25T01:16:40.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.939+0000 7f1a77400640 1 -- 172.21.15.90:0/3990984624 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f1a70058e50 con 0x7f1a8006e5a0 2024-04-25T01:16:40.923 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:16:40.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.941+0000 7f1a7fe00640 1 -- 172.21.15.90:0/3990984624 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1a5c06be30 msgr2=0x7f1a5c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:40.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.941+0000 7f1a7fe00640 1 --2- 172.21.15.90:0/3990984624 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1a5c06be30 0x7f1a5c06e2f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1a68034000 tx=0x7f1a6802e000 comp rx=0 tx=0).stop 2024-04-25T01:16:40.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.941+0000 7f1a7fe00640 1 -- 172.21.15.90:0/3990984624 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a8006e5a0 msgr2=0x7f1a8015ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:40.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.941+0000 7f1a7fe00640 1 --2- 172.21.15.90:0/3990984624 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a8006e5a0 0x7f1a8015ef60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1a7000caa0 tx=0x7f1a7000cf60 comp rx=0 tx=0).stop 2024-04-25T01:16:40.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.941+0000 7f1a7fe00640 1 -- 172.21.15.90:0/3990984624 shutdown_connections 2024-04-25T01:16:40.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.941+0000 7f1a7fe00640 1 --2- 172.21.15.90:0/3990984624 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1a5c06be30 0x7f1a5c06e2f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:40.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.941+0000 7f1a7fe00640 1 --2- 172.21.15.90:0/3990984624 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1a801596c0 0x7f1a80157d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:40.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.941+0000 7f1a7fe00640 1 --2- 172.21.15.90:0/3990984624 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a800b2880 0x7f1a801461b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:40.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.941+0000 7f1a7fe00640 1 --2- 172.21.15.90:0/3990984624 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a8006e5a0 0x7f1a8015ef60 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:40.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.941+0000 7f1a7fe00640 1 -- 172.21.15.90:0/3990984624 >> 172.21.15.90:0/3990984624 conn(0x7f1a8006b120 msgr2=0x7f1a8006b500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:40.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.942+0000 7f1a7fe00640 1 -- 172.21.15.90:0/3990984624 shutdown_connections 2024-04-25T01:16:40.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:40.942+0000 7f1a7fe00640 1 -- 172.21.15.90:0/3990984624 wait complete. 2024-04-25T01:16:40.945 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T01:16:41.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.164+0000 7f4ef4800640 1 Processor -- start 2024-04-25T01:16:41.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.165+0000 7f4ef4800640 1 -- start start 2024-04-25T01:16:41.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.166+0000 7f4ef4800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4eec15fe10 0x7f4eec06d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:41.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.166+0000 7f4ef4800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4eec0b4240 con 0x7f4eec15f9b0 2024-04-25T01:16:41.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.166+0000 7f4ef4800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4eec0b39a0 con 0x7f4eec145590 2024-04-25T01:16:41.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.166+0000 7f4ef4800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4eec160830 con 0x7f4eec15fe10 2024-04-25T01:16:41.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.166+0000 7f4ef1400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4eec15f9b0 0x7f4eec06d270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56512/0 (socket says 172.21.15.90:56512) 2024-04-25T01:16:41.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.166+0000 7f4ef1400640 1 -- 172.21.15.90:0/1753731930 learned_addr learned my addr 172.21.15.90:0/1753731930 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:41.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.167+0000 7f4ef2800640 1 --2- 172.21.15.90:0/1753731930 >> v2:172.21.15.103:3300/0 conn(0x7f4eec15fe10 0x7f4eec06d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:41.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.167+0000 7f4ef0a00640 1 -- 172.21.15.90:0/1753731930 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4131524234 0 0) 0x7f4eec0b4240 con 0x7f4eec15f9b0 2024-04-25T01:16:41.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.167+0000 7f4ef0a00640 1 -- 172.21.15.90:0/1753731930 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ed4003520 con 0x7f4eec15f9b0 2024-04-25T01:16:41.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.167+0000 7f4ef0a00640 1 -- 172.21.15.90:0/1753731930 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1038494348 0 0) 0x7f4ed4003520 con 0x7f4eec15f9b0 2024-04-25T01:16:41.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.167+0000 7f4ef0a00640 1 -- 172.21.15.90:0/1753731930 >> v2:172.21.15.103:3300/0 conn(0x7f4eec15fe10 msgr2=0x7f4eec06d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:41.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.167+0000 7f4ef0a00640 1 --2- 172.21.15.90:0/1753731930 >> v2:172.21.15.103:3300/0 conn(0x7f4eec15fe10 0x7f4eec06d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:41.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.167+0000 7f4ef0a00640 1 -- 172.21.15.90:0/1753731930 >> v1:172.21.15.103:6789/0 conn(0x7f4eec145590 legacy=0x7f4eec0b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:41.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.167+0000 7f4ef0a00640 1 -- 172.21.15.90:0/1753731930 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4eec0b3c80 con 0x7f4eec15f9b0 2024-04-25T01:16:41.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.167+0000 7f4ef0a00640 1 -- 172.21.15.90:0/1753731930 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f4edc001670 con 0x7f4eec15f9b0 2024-04-25T01:16:41.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.167+0000 7f4ef0a00640 1 -- 172.21.15.90:0/1753731930 >> v1:172.21.15.90:6789/0 conn(0x7f4eec15f9b0 legacy=0x7f4eec06d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:41.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.167+0000 7f4ef0a00640 1 --2- 172.21.15.90:0/1753731930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ed4003a00 0x7f4ed4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:41.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.167+0000 7f4ef0a00640 1 --2- 172.21.15.90:0/1753731930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ed4004800 0x7f4ed4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:41.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.168+0000 7f4ef0a00640 1 --2- 172.21.15.90:0/1753731930 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ed40051c0 0x7f4ed400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:41.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.168+0000 7f4ef0a00640 1 -- 172.21.15.90:0/1753731930 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed400c090 con 0x7f4ed4004800 2024-04-25T01:16:41.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.168+0000 7f4ef0a00640 1 -- 172.21.15.90:0/1753731930 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed400c240 con 0x7f4ed4003a00 2024-04-25T01:16:41.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.168+0000 7f4ef0a00640 1 -- 172.21.15.90:0/1753731930 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4ed400c440 con 0x7f4ed40051c0 2024-04-25T01:16:41.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.168+0000 7f4ef1400640 1 --2- 172.21.15.90:0/1753731930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ed4004800 0x7f4ed4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:41.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.168+0000 7f4ef1e00640 1 --2- 172.21.15.90:0/1753731930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ed4003a00 0x7f4ed4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:41.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.168+0000 7f4ef1400640 1 -- 172.21.15.90:0/1753731930 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ed40051c0 msgr2=0x7f4ed400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:41.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.168+0000 7f4ef1400640 1 --2- 172.21.15.90:0/1753731930 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ed40051c0 0x7f4ed400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:41.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.168+0000 7f4ef1400640 1 -- 172.21.15.90:0/1753731930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ed4003a00 msgr2=0x7f4ed4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:41.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.168+0000 7f4ef1400640 1 --2- 172.21.15.90:0/1753731930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ed4003a00 0x7f4ed4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:41.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.168+0000 7f4ef1400640 1 -- 172.21.15.90:0/1753731930 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ed4003520 con 0x7f4ed4004800 2024-04-25T01:16:41.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.168+0000 7f4ef2800640 1 -- 172.21.15.90:0/1753731930 reap_dead start 2024-04-25T01:16:41.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.169+0000 7f4ef1400640 1 --2- 172.21.15.90:0/1753731930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ed4004800 0x7f4ed4004c80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4edc00c0e0 tx=0x7f4ed4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef9bd67c01d13bc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:41.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.170+0000 7f4ef0a00640 1 -- 172.21.15.90:0/1753731930 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4edc007c20 con 0x7f4ed4004800 2024-04-25T01:16:41.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.170+0000 7f4ef0a00640 1 -- 172.21.15.90:0/1753731930 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4edc005af0 con 0x7f4ed4004800 2024-04-25T01:16:41.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.170+0000 7f4ef0a00640 1 -- 172.21.15.90:0/1753731930 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4edc007070 con 0x7f4ed4004800 2024-04-25T01:16:41.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.170+0000 7f4ef4800640 1 -- 172.21.15.90:0/1753731930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ed4004800 msgr2=0x7f4ed4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:41.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.170+0000 7f4ef4800640 1 --2- 172.21.15.90:0/1753731930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ed4004800 0x7f4ed4004c80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4edc00c0e0 tx=0x7f4ed4005830 comp rx=0 tx=0).stop 2024-04-25T01:16:41.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.170+0000 7f4ef4800640 1 -- 172.21.15.90:0/1753731930 shutdown_connections 2024-04-25T01:16:41.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.170+0000 7f4ef4800640 1 --2- 172.21.15.90:0/1753731930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ed4004800 0x7f4ed4004c80 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:41.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.170+0000 7f4ef4800640 1 -- 172.21.15.90:0/1753731930 >> 172.21.15.90:0/1753731930 conn(0x7f4eec0b1820 msgr2=0x7f4eec0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:41.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.170+0000 7f4ef4800640 1 -- 172.21.15.90:0/1753731930 shutdown_connections 2024-04-25T01:16:41.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.170+0000 7f4ef4800640 1 -- 172.21.15.90:0/1753731930 wait complete. 2024-04-25T01:16:41.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.172+0000 7f4ef4800640 1 Processor -- start 2024-04-25T01:16:41.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.172+0000 7f4ef4800640 1 -- start start 2024-04-25T01:16:41.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.173+0000 7f4ef4800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ed4004800 0x7f4eec1f4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:41.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.173+0000 7f4ef4800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4eec13cbc0 0x7f4eec13fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:41.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.173+0000 7f4ef4800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4eec15e820 0x7f4eec15eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:41.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.173+0000 7f4ef4800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4eec160280 con 0x7f4eec15e820 2024-04-25T01:16:41.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.173+0000 7f4ef4800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4eec06cde0 con 0x7f4eec13cbc0 2024-04-25T01:16:41.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.173+0000 7f4ef4800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4eec145970 con 0x7f4ed4004800 2024-04-25T01:16:41.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.174+0000 7f4ef2800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4eec15e820 0x7f4eec15eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:41.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.174+0000 7f4ef2800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4eec15e820 0x7f4eec15eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51876/0 (socket says 172.21.15.90:51876) 2024-04-25T01:16:41.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.174+0000 7f4ef2800640 1 -- 172.21.15.90:0/929630539 learned_addr learned my addr 172.21.15.90:0/929630539 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:41.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.174+0000 7f4ef1e00640 1 --2- 172.21.15.90:0/929630539 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ed4004800 0x7f4eec1f4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:41.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.174+0000 7f4ef2800640 1 -- 172.21.15.90:0/929630539 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ed4004800 msgr2=0x7f4eec1f4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:41.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.174+0000 7f4ef2800640 1 --2- 172.21.15.90:0/929630539 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ed4004800 0x7f4eec1f4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:41.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.174+0000 7f4ef1400640 1 --2- 172.21.15.90:0/929630539 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4eec13cbc0 0x7f4eec13fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:41.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.174+0000 7f4ef2800640 1 -- 172.21.15.90:0/929630539 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4eec13cbc0 msgr2=0x7f4eec13fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:41.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.174+0000 7f4ef2800640 1 --2- 172.21.15.90:0/929630539 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4eec13cbc0 0x7f4eec13fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:41.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.174+0000 7f4ef2800640 1 -- 172.21.15.90:0/929630539 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4eec13d9a0 con 0x7f4eec15e820 2024-04-25T01:16:41.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.174+0000 7f4ef2800640 1 --2- 172.21.15.90:0/929630539 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4eec15e820 0x7f4eec15eca0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4eec0b5160 tx=0x7f4ee800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:41.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.174+0000 7f4ee6000640 1 -- 172.21.15.90:0/929630539 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee800bc80 con 0x7f4eec15e820 2024-04-25T01:16:41.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.175+0000 7f4ee6000640 1 -- 172.21.15.90:0/929630539 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ee801a070 con 0x7f4eec15e820 2024-04-25T01:16:41.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.175+0000 7f4ee6000640 1 -- 172.21.15.90:0/929630539 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee8014640 con 0x7f4eec15e820 2024-04-25T01:16:41.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.175+0000 7f4ef4800640 1 -- 172.21.15.90:0/929630539 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4eec1f5280 con 0x7f4eec15e820 2024-04-25T01:16:41.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.175+0000 7f4ef4800640 1 -- 172.21.15.90:0/929630539 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4eec13d6c0 con 0x7f4eec15e820 2024-04-25T01:16:41.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.176+0000 7f4ef4800640 1 -- 172.21.15.90:0/929630539 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4eec068fb0 con 0x7f4eec15e820 2024-04-25T01:16:41.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.177+0000 7f4ee6000640 1 -- 172.21.15.90:0/929630539 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4ee8013070 con 0x7f4eec15e820 2024-04-25T01:16:41.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.178+0000 7f4ee6000640 1 --2- 172.21.15.90:0/929630539 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4ed006be30 0x7f4ed006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:41.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.178+0000 7f4ee6000640 1 -- 172.21.15.90:0/929630539 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f4ee808b530 con 0x7f4eec15e820 2024-04-25T01:16:41.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.178+0000 7f4ef1e00640 1 --2- 172.21.15.90:0/929630539 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4ed006be30 0x7f4ed006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:41.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.178+0000 7f4ef1e00640 1 --2- 172.21.15.90:0/929630539 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4ed006be30 0x7f4ed006e2f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0007fd0 tx=0x7f4ee0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:41.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.182+0000 7f4ee6000640 1 -- 172.21.15.90:0/929630539 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4eec1f5280 con 0x7f4eec15e820 2024-04-25T01:16:41.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.394+0000 7f4ef4800640 1 -- 172.21.15.90:0/929630539 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4eec06ea50 con 0x7f4eec15e820 2024-04-25T01:16:41.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.395+0000 7f4ee6000640 1 -- 172.21.15.90:0/929630539 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7f4ee800be20 con 0x7f4eec15e820 2024-04-25T01:16:41.379 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-04-25T01:16:41.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.398+0000 7f4ef4800640 1 -- 172.21.15.90:0/929630539 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4ed006be30 msgr2=0x7f4ed006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:41.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.398+0000 7f4ef4800640 1 --2- 172.21.15.90:0/929630539 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4ed006be30 0x7f4ed006e2f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0007fd0 tx=0x7f4ee0009040 comp rx=0 tx=0).stop 2024-04-25T01:16:41.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.398+0000 7f4ef4800640 1 -- 172.21.15.90:0/929630539 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4eec15e820 msgr2=0x7f4eec15eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:41.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.398+0000 7f4ef4800640 1 --2- 172.21.15.90:0/929630539 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4eec15e820 0x7f4eec15eca0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4eec0b5160 tx=0x7f4ee800b480 comp rx=0 tx=0).stop 2024-04-25T01:16:41.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.398+0000 7f4ef4800640 1 -- 172.21.15.90:0/929630539 shutdown_connections 2024-04-25T01:16:41.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.398+0000 7f4ef4800640 1 --2- 172.21.15.90:0/929630539 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4ed006be30 0x7f4ed006e2f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:41.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.398+0000 7f4ef4800640 1 --2- 172.21.15.90:0/929630539 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4eec15e820 0x7f4eec15eca0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:41.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.398+0000 7f4ef4800640 1 --2- 172.21.15.90:0/929630539 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4eec13cbc0 0x7f4eec13fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:41.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.398+0000 7f4ef4800640 1 --2- 172.21.15.90:0/929630539 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4ed4004800 0x7f4eec1f4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:41.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.398+0000 7f4ef4800640 1 -- 172.21.15.90:0/929630539 >> 172.21.15.90:0/929630539 conn(0x7f4eec0b1820 msgr2=0x7f4eec0cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:41.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.398+0000 7f4ef4800640 1 -- 172.21.15.90:0/929630539 shutdown_connections 2024-04-25T01:16:41.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:41.398+0000 7f4ef4800640 1 -- 172.21.15.90:0/929630539 wait complete. 2024-04-25T01:16:41.402 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi090.front.sepia.ceph.com... 2024-04-25T01:16:41.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:16:41.402 DEBUG:teuthology.orchestra.run.smithi090:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:16:41.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi090.stderr:ceph-fuse[39923]: fuse finished with error 0 and tester_r 0 2024-04-25T01:16:41.487 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:16:47.588 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-04-25T01:16:47.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:16:47.589 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-25T01:16:47.686 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:16:47.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.912+0000 7f1829a00640 1 Processor -- start 2024-04-25T01:16:47.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.912+0000 7f1829a00640 1 -- start start 2024-04-25T01:16:47.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.913+0000 7f1829a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f182406b120 0x7f182406b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:47.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.913+0000 7f1829a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f18240ae730 0x7f18240cd900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:47.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.913+0000 7f1829a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f182406ba40 con 0x7f182415ca10 2024-04-25T01:16:47.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.913+0000 7f1829a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f18240cef40 con 0x7f18240ae730 2024-04-25T01:16:47.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.913+0000 7f1829a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f182415c580 con 0x7f182406b120 2024-04-25T01:16:47.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.914+0000 7f181b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f18240ae730 0x7f18240cd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:47.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.914+0000 7f181b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f18240ae730 0x7f18240cd900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33390/0 (socket says 172.21.15.90:33390) 2024-04-25T01:16:47.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.914+0000 7f181b400640 1 -- 172.21.15.90:0/863847888 learned_addr learned my addr 172.21.15.90:0/863847888 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:47.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.914+0000 7f1823e00640 1 --2- 172.21.15.90:0/863847888 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f182406b120 0x7f182406b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:47.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.914+0000 7f181b400640 1 -- 172.21.15.90:0/863847888 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f182406b120 msgr2=0x7f182406b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:47.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.914+0000 7f181b400640 1 --2- 172.21.15.90:0/863847888 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f182406b120 0x7f182406b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:47.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.914+0000 7f181b400640 1 -- 172.21.15.90:0/863847888 >> v1:172.21.15.103:6789/0 conn(0x7f182415ca10 legacy=0x7f18240cc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:47.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.914+0000 7f181b400640 1 -- 172.21.15.90:0/863847888 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f182415c220 con 0x7f18240ae730 2024-04-25T01:16:47.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.915+0000 7f181b400640 1 --2- 172.21.15.90:0/863847888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18240ae730 0x7f18240cd900 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f180c009870 tx=0x7f180c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e036218502d07791 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:47.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.915+0000 7f1823400640 1 -- 172.21.15.90:0/863847888 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f180c037070 con 0x7f18240ae730 2024-04-25T01:16:47.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.915+0000 7f1823400640 1 -- 172.21.15.90:0/863847888 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f180c032440 con 0x7f18240ae730 2024-04-25T01:16:47.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.916+0000 7f1823400640 1 -- 172.21.15.90:0/863847888 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f180c032810 con 0x7f18240ae730 2024-04-25T01:16:47.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.916+0000 7f1829a00640 1 -- 172.21.15.90:0/863847888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18240ae730 msgr2=0x7f18240cd900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:47.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.916+0000 7f1829a00640 1 --2- 172.21.15.90:0/863847888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18240ae730 0x7f18240cd900 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f180c009870 tx=0x7f180c02a210 comp rx=0 tx=0).stop 2024-04-25T01:16:47.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.916+0000 7f1829a00640 1 -- 172.21.15.90:0/863847888 shutdown_connections 2024-04-25T01:16:47.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.916+0000 7f1829a00640 1 --2- 172.21.15.90:0/863847888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18240ae730 0x7f18240cd900 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:47.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.916+0000 7f1829a00640 1 --2- 172.21.15.90:0/863847888 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f182406b120 0x7f182406b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:47.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.916+0000 7f1829a00640 1 -- 172.21.15.90:0/863847888 >> 172.21.15.90:0/863847888 conn(0x7f18240b1660 msgr2=0x7f18240b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:47.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.916+0000 7f1829a00640 1 -- 172.21.15.90:0/863847888 shutdown_connections 2024-04-25T01:16:47.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.916+0000 7f1829a00640 1 -- 172.21.15.90:0/863847888 wait complete. 2024-04-25T01:16:47.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.918+0000 7f1829a00640 1 Processor -- start 2024-04-25T01:16:47.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.918+0000 7f1829a00640 1 -- start start 2024-04-25T01:16:47.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.919+0000 7f1829a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f182406b120 0x7f18241f7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:47.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.919+0000 7f1829a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18240ae730 0x7f182415b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:47.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.919+0000 7f1829a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f182415ca10 0x7f182415b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:47.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.919+0000 7f1829a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f18240af240 con 0x7f18240ae730 2024-04-25T01:16:47.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.919+0000 7f1829a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f18240aef60 con 0x7f182406b120 2024-04-25T01:16:47.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.919+0000 7f1829a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1824153080 con 0x7f182415ca10 2024-04-25T01:16:47.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.919+0000 7f181b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18240ae730 0x7f182415b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:47.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.919+0000 7f181b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18240ae730 0x7f182415b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33394/0 (socket says 172.21.15.90:33394) 2024-04-25T01:16:47.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.919+0000 7f181b400640 1 -- 172.21.15.90:0/669176919 learned_addr learned my addr 172.21.15.90:0/669176919 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:47.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.920+0000 7f1829000640 1 --2- 172.21.15.90:0/669176919 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f182415ca10 0x7f182415b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:47.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.920+0000 7f181b400640 1 -- 172.21.15.90:0/669176919 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f182415ca10 msgr2=0x7f182415b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:47.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.920+0000 7f181b400640 1 --2- 172.21.15.90:0/669176919 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f182415ca10 0x7f182415b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:47.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.920+0000 7f181b400640 1 -- 172.21.15.90:0/669176919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f182406b120 msgr2=0x7f18241f7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:47.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.920+0000 7f1823e00640 1 --2- 172.21.15.90:0/669176919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f182406b120 0x7f18241f7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:47.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.920+0000 7f181b400640 1 --2- 172.21.15.90:0/669176919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f182406b120 0x7f18241f7fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:47.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.920+0000 7f181b400640 1 -- 172.21.15.90:0/669176919 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18241f8510 con 0x7f18240ae730 2024-04-25T01:16:47.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.920+0000 7f181b400640 1 --2- 172.21.15.90:0/669176919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18240ae730 0x7f182415b180 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f180c009870 tx=0x7f180c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:47.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.921+0000 7f1820c00640 1 -- 172.21.15.90:0/669176919 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f180c041070 con 0x7f18240ae730 2024-04-25T01:16:47.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.921+0000 7f1820c00640 1 -- 172.21.15.90:0/669176919 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f180c03c890 con 0x7f18240ae730 2024-04-25T01:16:47.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.921+0000 7f1829a00640 1 -- 172.21.15.90:0/669176919 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18241469c0 con 0x7f18240ae730 2024-04-25T01:16:47.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.921+0000 7f1820c00640 1 -- 172.21.15.90:0/669176919 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f180c037040 con 0x7f18240ae730 2024-04-25T01:16:47.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.921+0000 7f1829a00640 1 -- 172.21.15.90:0/669176919 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1824146090 con 0x7f18240ae730 2024-04-25T01:16:47.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.922+0000 7f1829a00640 1 -- 172.21.15.90:0/669176919 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1824075350 con 0x7f18240ae730 2024-04-25T01:16:47.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.923+0000 7f1820c00640 1 -- 172.21.15.90:0/669176919 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f180c044070 con 0x7f18240ae730 2024-04-25T01:16:47.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.924+0000 7f1820c00640 1 --2- 172.21.15.90:0/669176919 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f180006be30 0x7f180006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:47.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.924+0000 7f1820c00640 1 -- 172.21.15.90:0/669176919 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f180c0abcf0 con 0x7f18240ae730 2024-04-25T01:16:47.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.924+0000 7f1823e00640 1 --2- 172.21.15.90:0/669176919 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f180006be30 0x7f180006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:47.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.925+0000 7f1823e00640 1 --2- 172.21.15.90:0/669176919 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f180006be30 0x7f180006e2f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f18240b14d0 tx=0x7f181400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:47.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:47.928+0000 7f1820c00640 1 -- 172.21.15.90:0/669176919 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f180c0b0070 con 0x7f18240ae730 2024-04-25T01:16:48.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.125+0000 7f1829a00640 1 -- 172.21.15.90:0/669176919 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f182406ccd0 con 0x7f18240ae730 2024-04-25T01:16:48.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.129+0000 7f1820c00640 1 -- 172.21.15.90:0/669176919 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18482 (secure 0 0 0) 0x7f18241f8510 con 0x7f18240ae730 2024-04-25T01:16:48.113 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:16:48.114 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":26,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:16:27.092855+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:16:20.094512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:16:21.078268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:16:24.670642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:16:48.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.132+0000 7f1829a00640 1 -- 172.21.15.90:0/669176919 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f180006be30 msgr2=0x7f180006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:48.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.132+0000 7f1829a00640 1 --2- 172.21.15.90:0/669176919 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f180006be30 0x7f180006e2f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f18240b14d0 tx=0x7f181400b040 comp rx=0 tx=0).stop 2024-04-25T01:16:48.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.132+0000 7f1829a00640 1 -- 172.21.15.90:0/669176919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18240ae730 msgr2=0x7f182415b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:48.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.132+0000 7f1829a00640 1 --2- 172.21.15.90:0/669176919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18240ae730 0x7f182415b180 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f180c009870 tx=0x7f180c0046c0 comp rx=0 tx=0).stop 2024-04-25T01:16:48.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.132+0000 7f1829a00640 1 -- 172.21.15.90:0/669176919 shutdown_connections 2024-04-25T01:16:48.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.132+0000 7f1829a00640 1 --2- 172.21.15.90:0/669176919 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f180006be30 0x7f180006e2f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:48.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.132+0000 7f1829a00640 1 --2- 172.21.15.90:0/669176919 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f182415ca10 0x7f182415b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:48.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.132+0000 7f1829a00640 1 --2- 172.21.15.90:0/669176919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18240ae730 0x7f182415b180 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:48.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.132+0000 7f1829a00640 1 --2- 172.21.15.90:0/669176919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f182406b120 0x7f18241f7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:48.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.132+0000 7f1829a00640 1 -- 172.21.15.90:0/669176919 >> 172.21.15.90:0/669176919 conn(0x7f18240b1660 msgr2=0x7f18240ce280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:48.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.132+0000 7f1829a00640 1 -- 172.21.15.90:0/669176919 shutdown_connections 2024-04-25T01:16:48.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.132+0000 7f1829a00640 1 -- 172.21.15.90:0/669176919 wait complete. 2024-04-25T01:16:48.136 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:16:48.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.340+0000 7f9e22400640 1 Processor -- start 2024-04-25T01:16:48.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.341+0000 7f9e22400640 1 -- start start 2024-04-25T01:16:48.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.342+0000 7f9e22400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9e1c06b120 0x7f9e1c06b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:48.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.342+0000 7f9e22400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9e1c0ae730 0x7f9e1c0cd900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:48.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.342+0000 7f9e22400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e1c06ba40 con 0x7f9e1c15ca10 2024-04-25T01:16:48.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.342+0000 7f9e22400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f9e1c0cef40 con 0x7f9e1c06b120 2024-04-25T01:16:48.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.342+0000 7f9e22400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9e1c15c580 con 0x7f9e1c0ae730 2024-04-25T01:16:48.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.343+0000 7f9e21000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9e1c06b120 0x7f9e1c06b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:48.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.343+0000 7f9e21000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9e1c06b120 0x7f9e1c06b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33408/0 (socket says 172.21.15.90:33408) 2024-04-25T01:16:48.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.343+0000 7f9e21000640 1 -- 172.21.15.90:0/36948660 learned_addr learned my addr 172.21.15.90:0/36948660 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:48.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.343+0000 7f9e13e00640 1 --2- 172.21.15.90:0/36948660 >> v2:172.21.15.103:3300/0 conn(0x7f9e1c0ae730 0x7f9e1c0cd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:48.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.343+0000 7f9e21000640 1 -- 172.21.15.90:0/36948660 >> v2:172.21.15.103:3300/0 conn(0x7f9e1c0ae730 msgr2=0x7f9e1c0cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:48.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.343+0000 7f9e21000640 1 --2- 172.21.15.90:0/36948660 >> v2:172.21.15.103:3300/0 conn(0x7f9e1c0ae730 0x7f9e1c0cd900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:48.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.343+0000 7f9e21000640 1 -- 172.21.15.90:0/36948660 >> v1:172.21.15.103:6789/0 conn(0x7f9e1c15ca10 legacy=0x7f9e1c0cc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:48.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.343+0000 7f9e21000640 1 -- 172.21.15.90:0/36948660 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e1c15c220 con 0x7f9e1c06b120 2024-04-25T01:16:48.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.344+0000 7f9e21000640 1 --2- 172.21.15.90:0/36948660 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1c06b120 0x7f9e1c06b500 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c0b2680 tx=0x7f9e0c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=226ecb762624d8ba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:48.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.345+0000 7f9e1be00640 1 -- 172.21.15.90:0/36948660 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c037070 con 0x7f9e1c06b120 2024-04-25T01:16:48.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.345+0000 7f9e1be00640 1 -- 172.21.15.90:0/36948660 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e0c032440 con 0x7f9e1c06b120 2024-04-25T01:16:48.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.345+0000 7f9e1be00640 1 -- 172.21.15.90:0/36948660 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c032810 con 0x7f9e1c06b120 2024-04-25T01:16:48.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.345+0000 7f9e22400640 1 -- 172.21.15.90:0/36948660 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1c06b120 msgr2=0x7f9e1c06b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:48.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.345+0000 7f9e22400640 1 --2- 172.21.15.90:0/36948660 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1c06b120 0x7f9e1c06b500 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c0b2680 tx=0x7f9e0c02a230 comp rx=0 tx=0).stop 2024-04-25T01:16:48.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.345+0000 7f9e22400640 1 -- 172.21.15.90:0/36948660 shutdown_connections 2024-04-25T01:16:48.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.345+0000 7f9e22400640 1 --2- 172.21.15.90:0/36948660 >> v2:172.21.15.103:3300/0 conn(0x7f9e1c0ae730 0x7f9e1c0cd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:48.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.345+0000 7f9e22400640 1 --2- 172.21.15.90:0/36948660 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1c06b120 0x7f9e1c06b500 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:48.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.345+0000 7f9e22400640 1 -- 172.21.15.90:0/36948660 >> 172.21.15.90:0/36948660 conn(0x7f9e1c0b1660 msgr2=0x7f9e1c0b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:48.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.345+0000 7f9e22400640 1 -- 172.21.15.90:0/36948660 shutdown_connections 2024-04-25T01:16:48.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.345+0000 7f9e22400640 1 -- 172.21.15.90:0/36948660 wait complete. 2024-04-25T01:16:48.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.347+0000 7f9e22400640 1 Processor -- start 2024-04-25T01:16:48.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.347+0000 7f9e22400640 1 -- start start 2024-04-25T01:16:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.348+0000 7f9e22400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e1c06b120 0x7f9e1c1e8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.348+0000 7f9e22400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1c0ae730 0x7f9e1c1f4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.348+0000 7f9e22400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e1c15ca10 0x7f9e1c071490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.348+0000 7f9e22400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1c0af240 con 0x7f9e1c0ae730 2024-04-25T01:16:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.348+0000 7f9e22400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1c0aef60 con 0x7f9e1c15ca10 2024-04-25T01:16:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.348+0000 7f9e22400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9e1c153080 con 0x7f9e1c06b120 2024-04-25T01:16:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.348+0000 7f9e13e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1c0ae730 0x7f9e1c1f4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.348+0000 7f9e13e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1c0ae730 0x7f9e1c1f4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33416/0 (socket says 172.21.15.90:33416) 2024-04-25T01:16:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.349+0000 7f9e13e00640 1 -- 172.21.15.90:0/785867622 learned_addr learned my addr 172.21.15.90:0/785867622 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:48.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.349+0000 7f9e21000640 1 --2- 172.21.15.90:0/785867622 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e1c06b120 0x7f9e1c1e8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:48.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.349+0000 7f9e21a00640 1 --2- 172.21.15.90:0/785867622 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e1c15ca10 0x7f9e1c071490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:48.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.349+0000 7f9e13e00640 1 -- 172.21.15.90:0/785867622 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e1c06b120 msgr2=0x7f9e1c1e8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:48.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.349+0000 7f9e13e00640 1 --2- 172.21.15.90:0/785867622 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e1c06b120 0x7f9e1c1e8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:48.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.349+0000 7f9e13e00640 1 -- 172.21.15.90:0/785867622 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e1c15ca10 msgr2=0x7f9e1c071490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:48.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.349+0000 7f9e13e00640 1 --2- 172.21.15.90:0/785867622 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e1c15ca10 0x7f9e1c071490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:48.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.349+0000 7f9e13e00640 1 -- 172.21.15.90:0/785867622 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e1c1e9150 con 0x7f9e1c0ae730 2024-04-25T01:16:48.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.349+0000 7f9e21000640 1 --2- 172.21.15.90:0/785867622 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e1c06b120 0x7f9e1c1e8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:48.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.349+0000 7f9e13e00640 1 --2- 172.21.15.90:0/785867622 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1c0ae730 0x7f9e1c1f4030 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9e0400d800 tx=0x7f9e0400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:48.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.349+0000 7f9e19600640 1 -- 172.21.15.90:0/785867622 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0400b6c0 con 0x7f9e1c0ae730 2024-04-25T01:16:48.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.350+0000 7f9e19600640 1 -- 172.21.15.90:0/785867622 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e04019070 con 0x7f9e1c0ae730 2024-04-25T01:16:48.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.350+0000 7f9e19600640 1 -- 172.21.15.90:0/785867622 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0400ba80 con 0x7f9e1c0ae730 2024-04-25T01:16:48.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.350+0000 7f9e22400640 1 -- 172.21.15.90:0/785867622 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e1c154050 con 0x7f9e1c0ae730 2024-04-25T01:16:48.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.350+0000 7f9e22400640 1 -- 172.21.15.90:0/785867622 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e1c155220 con 0x7f9e1c0ae730 2024-04-25T01:16:48.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.351+0000 7f9e22400640 1 -- 172.21.15.90:0/785867622 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e1c075350 con 0x7f9e1c0ae730 2024-04-25T01:16:48.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.352+0000 7f9e19600640 1 -- 172.21.15.90:0/785867622 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9e0400bc20 con 0x7f9e1c0ae730 2024-04-25T01:16:48.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.353+0000 7f9e19600640 1 --2- 172.21.15.90:0/785867622 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9df806be30 0x7f9df806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:48.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.353+0000 7f9e19600640 1 -- 172.21.15.90:0/785867622 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f9e0408b620 con 0x7f9e1c0ae730 2024-04-25T01:16:48.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.354+0000 7f9e21000640 1 --2- 172.21.15.90:0/785867622 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9df806be30 0x7f9df806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:48.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.354+0000 7f9e21000640 1 --2- 172.21.15.90:0/785867622 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9df806be30 0x7f9df806e2f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c034000 tx=0x7f9e0c02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:48.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.358+0000 7f9e19600640 1 -- 172.21.15.90:0/785867622 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9e1c1e9150 con 0x7f9e1c0ae730 2024-04-25T01:16:48.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.559+0000 7f9e22400640 1 -- 172.21.15.90:0/785867622 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9e1c06ccd0 con 0x7f9e1c0ae730 2024-04-25T01:16:48.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.562+0000 7f9e19600640 1 -- 172.21.15.90:0/785867622 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18482 (secure 0 0 0) 0x7f9e04059c80 con 0x7f9e1c0ae730 2024-04-25T01:16:48.546 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:16:48.546 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":26,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:16:27.092855+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:16:20.094512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:16:21.078268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:16:24.670642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:16:48.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.564+0000 7f9e22400640 1 -- 172.21.15.90:0/785867622 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9df806be30 msgr2=0x7f9df806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:48.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.564+0000 7f9e22400640 1 --2- 172.21.15.90:0/785867622 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9df806be30 0x7f9df806e2f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c034000 tx=0x7f9e0c02e000 comp rx=0 tx=0).stop 2024-04-25T01:16:48.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.564+0000 7f9e22400640 1 -- 172.21.15.90:0/785867622 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1c0ae730 msgr2=0x7f9e1c1f4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:48.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.564+0000 7f9e22400640 1 --2- 172.21.15.90:0/785867622 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1c0ae730 0x7f9e1c1f4030 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9e0400d800 tx=0x7f9e0400dcc0 comp rx=0 tx=0).stop 2024-04-25T01:16:48.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.564+0000 7f9e22400640 1 -- 172.21.15.90:0/785867622 shutdown_connections 2024-04-25T01:16:48.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.564+0000 7f9e22400640 1 --2- 172.21.15.90:0/785867622 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9df806be30 0x7f9df806e2f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:48.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.564+0000 7f9e22400640 1 --2- 172.21.15.90:0/785867622 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e1c15ca10 0x7f9e1c071490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:48.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.564+0000 7f9e22400640 1 --2- 172.21.15.90:0/785867622 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1c0ae730 0x7f9e1c1f4030 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:48.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.564+0000 7f9e22400640 1 --2- 172.21.15.90:0/785867622 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e1c06b120 0x7f9e1c1e8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:48.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.564+0000 7f9e22400640 1 -- 172.21.15.90:0/785867622 >> 172.21.15.90:0/785867622 conn(0x7f9e1c0b1660 msgr2=0x7f9e1c0af5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:48.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.565+0000 7f9e22400640 1 -- 172.21.15.90:0/785867622 shutdown_connections 2024-04-25T01:16:48.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.565+0000 7f9e22400640 1 -- 172.21.15.90:0/785867622 wait complete. 2024-04-25T01:16:48.568 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T01:16:48.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.783+0000 7fdaa1000640 1 Processor -- start 2024-04-25T01:16:48.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.783+0000 7fdaa1000640 1 -- start start 2024-04-25T01:16:48.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.785+0000 7fdaa1000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda9c15fc40 0x7fda9c1600a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:48.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.785+0000 7fdaa1000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fda9c0cedc0 0x7fda9c0cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:48.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.785+0000 7fdaa1000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda9c0cf790 con 0x7fda9c0722e0 2024-04-25T01:16:48.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.785+0000 7fdaa1000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fda9c160aa0 con 0x7fda9c0cedc0 2024-04-25T01:16:48.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.785+0000 7fdaa1000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fda9c06da40 con 0x7fda9c15fc40 2024-04-25T01:16:48.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.785+0000 7fda9a800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fda9c0cedc0 0x7fda9c0cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:48.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.785+0000 7fda99400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda9c15fc40 0x7fda9c1600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:48.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.785+0000 7fda99400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda9c15fc40 0x7fda9c1600a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.90:36630/0 (socket says 172.21.15.90:36630) 2024-04-25T01:16:48.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.785+0000 7fda99400640 1 -- 172.21.15.90:0/3642275508 learned_addr learned my addr 172.21.15.90:0/3642275508 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:48.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.786+0000 7fda99400640 1 -- 172.21.15.90:0/3642275508 >> v2:172.21.15.103:3300/0 conn(0x7fda9c0cedc0 msgr2=0x7fda9c0cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:48.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.786+0000 7fda99400640 1 --2- 172.21.15.90:0/3642275508 >> v2:172.21.15.103:3300/0 conn(0x7fda9c0cedc0 0x7fda9c0cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:48.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.786+0000 7fda99400640 1 -- 172.21.15.90:0/3642275508 >> v1:172.21.15.103:6789/0 conn(0x7fda9c0722e0 legacy=0x7fda9c06ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:48.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.786+0000 7fda99400640 1 -- 172.21.15.90:0/3642275508 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda9c06d110 con 0x7fda9c15fc40 2024-04-25T01:16:48.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.788+0000 7fda99400640 1 --2- 172.21.15.90:0/3642275508 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda9c15fc40 0x7fda9c1600a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fda88009870 tx=0x7fda8802a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ddd51b822cbcb97 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:48.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.788+0000 7fda98a00640 1 -- 172.21.15.90:0/3642275508 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda88037070 con 0x7fda9c15fc40 2024-04-25T01:16:48.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.788+0000 7fda98a00640 1 -- 172.21.15.90:0/3642275508 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda88032440 con 0x7fda9c15fc40 2024-04-25T01:16:48.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.788+0000 7fda98a00640 1 -- 172.21.15.90:0/3642275508 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda88032810 con 0x7fda9c15fc40 2024-04-25T01:16:48.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.789+0000 7fdaa1000640 1 -- 172.21.15.90:0/3642275508 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda9c15fc40 msgr2=0x7fda9c1600a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:48.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.789+0000 7fdaa1000640 1 --2- 172.21.15.90:0/3642275508 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda9c15fc40 0x7fda9c1600a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fda88009870 tx=0x7fda8802a1c0 comp rx=0 tx=0).stop 2024-04-25T01:16:48.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.789+0000 7fdaa1000640 1 -- 172.21.15.90:0/3642275508 shutdown_connections 2024-04-25T01:16:48.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.789+0000 7fdaa1000640 1 --2- 172.21.15.90:0/3642275508 >> v2:172.21.15.103:3300/0 conn(0x7fda9c0cedc0 0x7fda9c0cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:48.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.789+0000 7fdaa1000640 1 --2- 172.21.15.90:0/3642275508 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda9c15fc40 0x7fda9c1600a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:48.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.789+0000 7fdaa1000640 1 -- 172.21.15.90:0/3642275508 >> 172.21.15.90:0/3642275508 conn(0x7fda9c06b720 msgr2=0x7fda9c0b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:48.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.789+0000 7fdaa1000640 1 -- 172.21.15.90:0/3642275508 shutdown_connections 2024-04-25T01:16:48.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.789+0000 7fdaa1000640 1 -- 172.21.15.90:0/3642275508 wait complete. 2024-04-25T01:16:48.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.791+0000 7fdaa1000640 1 Processor -- start 2024-04-25T01:16:48.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.792+0000 7fdaa1000640 1 -- start start 2024-04-25T01:16:48.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.793+0000 7fdaa1000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda9c0722e0 0x7fda9c147990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:48.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.793+0000 7fdaa1000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda9c0cedc0 0x7fda9c15e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:48.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.793+0000 7fdaa1000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda9c15fc40 0x7fda9c157150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:48.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.793+0000 7fdaa1000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fda9c06cec0 con 0x7fda9c15fc40 2024-04-25T01:16:48.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.793+0000 7fdaa1000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fda9c06d6d0 con 0x7fda9c0722e0 2024-04-25T01:16:48.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.793+0000 7fdaa1000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fda9c070b50 con 0x7fda9c0cedc0 2024-04-25T01:16:48.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.793+0000 7fda9a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda9c15fc40 0x7fda9c157150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:48.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.793+0000 7fda9a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda9c15fc40 0x7fda9c157150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33424/0 (socket says 172.21.15.90:33424) 2024-04-25T01:16:48.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.793+0000 7fda9a800640 1 -- 172.21.15.90:0/2554142118 learned_addr learned my addr 172.21.15.90:0/2554142118 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:48.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.793+0000 7fda99400640 1 --2- 172.21.15.90:0/2554142118 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda9c0cedc0 0x7fda9c15e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:48.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.793+0000 7fda99e00640 1 --2- 172.21.15.90:0/2554142118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda9c0722e0 0x7fda9c147990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:48.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.793+0000 7fda9a800640 1 -- 172.21.15.90:0/2554142118 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda9c0cedc0 msgr2=0x7fda9c15e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:48.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.793+0000 7fda9a800640 1 --2- 172.21.15.90:0/2554142118 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda9c0cedc0 0x7fda9c15e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:48.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.793+0000 7fda9a800640 1 -- 172.21.15.90:0/2554142118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda9c0722e0 msgr2=0x7fda9c147990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:48.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.793+0000 7fda9a800640 1 --2- 172.21.15.90:0/2554142118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda9c0722e0 0x7fda9c147990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:48.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.794+0000 7fda9a800640 1 -- 172.21.15.90:0/2554142118 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda9c147ed0 con 0x7fda9c15fc40 2024-04-25T01:16:48.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.794+0000 7fda9a800640 1 --2- 172.21.15.90:0/2554142118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda9c15fc40 0x7fda9c157150 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fda9400efc0 tx=0x7fda9400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:48.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.794+0000 7fda92000640 1 -- 172.21.15.90:0/2554142118 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda94018020 con 0x7fda9c15fc40 2024-04-25T01:16:48.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.794+0000 7fda92000640 1 -- 172.21.15.90:0/2554142118 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda94014480 con 0x7fda9c15fc40 2024-04-25T01:16:48.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.794+0000 7fda92000640 1 -- 172.21.15.90:0/2554142118 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda94014870 con 0x7fda9c15fc40 2024-04-25T01:16:48.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.794+0000 7fdaa1000640 1 -- 172.21.15.90:0/2554142118 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda9c1452e0 con 0x7fda9c15fc40 2024-04-25T01:16:48.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.794+0000 7fdaa1000640 1 -- 172.21.15.90:0/2554142118 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda9c1458a0 con 0x7fda9c15fc40 2024-04-25T01:16:48.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.795+0000 7fdaa1000640 1 -- 172.21.15.90:0/2554142118 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda9c15b990 con 0x7fda9c15fc40 2024-04-25T01:16:48.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.797+0000 7fda92000640 1 -- 172.21.15.90:0/2554142118 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fda94005d10 con 0x7fda9c15fc40 2024-04-25T01:16:48.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.797+0000 7fda92000640 1 --2- 172.21.15.90:0/2554142118 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fda7c06be30 0x7fda7c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:48.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.798+0000 7fda92000640 1 -- 172.21.15.90:0/2554142118 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fda9408c140 con 0x7fda9c15fc40 2024-04-25T01:16:48.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.798+0000 7fda99e00640 1 --2- 172.21.15.90:0/2554142118 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fda7c06be30 0x7fda7c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:48.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.798+0000 7fda99e00640 1 --2- 172.21.15.90:0/2554142118 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fda7c06be30 0x7fda7c06e2f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fda8c001640 tx=0x7fda8c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:48.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:48.802+0000 7fda92000640 1 -- 172.21.15.90:0/2554142118 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fda9c15b990 con 0x7fda9c15fc40 2024-04-25T01:16:49.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:49.021+0000 7fdaa1000640 1 -- 172.21.15.90:0/2554142118 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fda9c068fb0 con 0x7fda9c15fc40 2024-04-25T01:16:49.536 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:16:49.544 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:16:49.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:49.920+0000 7fda92000640 1 -- 172.21.15.90:0/2554142118 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) ==== 118+0+0 (secure 0 0 0) 0x7fda94022030 con 0x7fda9c15fc40 2024-04-25T01:16:49.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:49.924+0000 7fdaa1000640 1 -- 172.21.15.90:0/2554142118 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fda7c06be30 msgr2=0x7fda7c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:49.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:49.924+0000 7fdaa1000640 1 --2- 172.21.15.90:0/2554142118 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fda7c06be30 0x7fda7c06e2f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fda8c001640 tx=0x7fda8c009040 comp rx=0 tx=0).stop 2024-04-25T01:16:49.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:49.924+0000 7fdaa1000640 1 -- 172.21.15.90:0/2554142118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda9c15fc40 msgr2=0x7fda9c157150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:49.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:49.924+0000 7fdaa1000640 1 --2- 172.21.15.90:0/2554142118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda9c15fc40 0x7fda9c157150 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fda9400efc0 tx=0x7fda9400c490 comp rx=0 tx=0).stop 2024-04-25T01:16:49.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:49.924+0000 7fdaa1000640 1 -- 172.21.15.90:0/2554142118 shutdown_connections 2024-04-25T01:16:49.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:49.924+0000 7fdaa1000640 1 --2- 172.21.15.90:0/2554142118 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fda7c06be30 0x7fda7c06e2f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:49.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:49.924+0000 7fdaa1000640 1 --2- 172.21.15.90:0/2554142118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda9c15fc40 0x7fda9c157150 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:49.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:49.924+0000 7fdaa1000640 1 --2- 172.21.15.90:0/2554142118 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fda9c0cedc0 0x7fda9c15e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:49.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:49.924+0000 7fdaa1000640 1 --2- 172.21.15.90:0/2554142118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda9c0722e0 0x7fda9c147990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:49.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:49.924+0000 7fdaa1000640 1 -- 172.21.15.90:0/2554142118 >> 172.21.15.90:0/2554142118 conn(0x7fda9c06b720 msgr2=0x7fda9c15c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:49.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:49.924+0000 7fdaa1000640 1 -- 172.21.15.90:0/2554142118 shutdown_connections 2024-04-25T01:16:49.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:49.924+0000 7fdaa1000640 1 -- 172.21.15.90:0/2554142118 wait complete. 2024-04-25T01:16:49.929 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-04-25T01:16:49.929 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-25T01:16:50.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.151+0000 7f9776c00640 1 Processor -- start 2024-04-25T01:16:50.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.151+0000 7f9776c00640 1 -- start start 2024-04-25T01:16:50.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.152+0000 7f9776c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97701596c0 0x7f977006de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:50.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.153+0000 7f9776c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f97700b2880 0x7f977006e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:50.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.153+0000 7f9776c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f977015c200 con 0x7f97700cf7f0 2024-04-25T01:16:50.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.153+0000 7f9776c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9770159aa0 con 0x7f97700b2880 2024-04-25T01:16:50.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.153+0000 7f9776c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f977015c470 con 0x7f97701596c0 2024-04-25T01:16:50.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.153+0000 7f9775800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97701596c0 0x7f977006de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:50.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.153+0000 7f9774e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f97700b2880 0x7f977006e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:50.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.153+0000 7f9775800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97701596c0 0x7f977006de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.90:36652/0 (socket says 172.21.15.90:36652) 2024-04-25T01:16:50.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.153+0000 7f9775800640 1 -- 172.21.15.90:0/978669240 learned_addr learned my addr 172.21.15.90:0/978669240 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:50.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.154+0000 7f9775800640 1 -- 172.21.15.90:0/978669240 >> v2:172.21.15.103:3300/0 conn(0x7f97700b2880 msgr2=0x7f977006e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:50.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.154+0000 7f9775800640 1 --2- 172.21.15.90:0/978669240 >> v2:172.21.15.103:3300/0 conn(0x7f97700b2880 0x7f977006e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:50.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.154+0000 7f9775800640 1 -- 172.21.15.90:0/978669240 >> v1:172.21.15.103:6789/0 conn(0x7f97700cf7f0 legacy=0x7f97700cfc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:50.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.154+0000 7f9775800640 1 -- 172.21.15.90:0/978669240 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f977015c750 con 0x7f97701596c0 2024-04-25T01:16:50.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.156+0000 7f9775800640 1 --2- 172.21.15.90:0/978669240 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97701596c0 0x7f977006de90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9764009870 tx=0x7f976402a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5685ea3989928078 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:50.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.156+0000 7f976fe00640 1 -- 172.21.15.90:0/978669240 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9764037070 con 0x7f97701596c0 2024-04-25T01:16:50.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.156+0000 7f976fe00640 1 -- 172.21.15.90:0/978669240 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9764032440 con 0x7f97701596c0 2024-04-25T01:16:50.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.156+0000 7f976fe00640 1 -- 172.21.15.90:0/978669240 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9764032810 con 0x7f97701596c0 2024-04-25T01:16:50.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.157+0000 7f9776c00640 1 -- 172.21.15.90:0/978669240 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97701596c0 msgr2=0x7f977006de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:50.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.157+0000 7f9776c00640 1 --2- 172.21.15.90:0/978669240 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97701596c0 0x7f977006de90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9764009870 tx=0x7f976402a270 comp rx=0 tx=0).stop 2024-04-25T01:16:50.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.157+0000 7f9776c00640 1 -- 172.21.15.90:0/978669240 shutdown_connections 2024-04-25T01:16:50.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.157+0000 7f9776c00640 1 --2- 172.21.15.90:0/978669240 >> v2:172.21.15.103:3300/0 conn(0x7f97700b2880 0x7f977006e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:50.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.157+0000 7f9776c00640 1 --2- 172.21.15.90:0/978669240 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97701596c0 0x7f977006de90 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:50.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.157+0000 7f9776c00640 1 -- 172.21.15.90:0/978669240 >> 172.21.15.90:0/978669240 conn(0x7f977006b120 msgr2=0x7f977006b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:50.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.157+0000 7f9776c00640 1 -- 172.21.15.90:0/978669240 shutdown_connections 2024-04-25T01:16:50.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.157+0000 7f9776c00640 1 -- 172.21.15.90:0/978669240 wait complete. 2024-04-25T01:16:50.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.159+0000 7f9776c00640 1 Processor -- start 2024-04-25T01:16:50.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.160+0000 7f9776c00640 1 -- start start 2024-04-25T01:16:50.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.161+0000 7f9776c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97700b2880 0x7f97701f3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:50.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.161+0000 7f9776c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97700cf7f0 0x7f977014ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:50.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.161+0000 7f9776c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97701596c0 0x7f977014f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:50.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.161+0000 7f9776c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f977015f770 con 0x7f97700b2880 2024-04-25T01:16:50.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.161+0000 7f9776c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f977015cd10 con 0x7f97701596c0 2024-04-25T01:16:50.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.161+0000 7f9776c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f977015f8f0 con 0x7f97700cf7f0 2024-04-25T01:16:50.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.161+0000 7f9775800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97700b2880 0x7f97701f3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:50.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.161+0000 7f9775800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97700b2880 0x7f97701f3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33434/0 (socket says 172.21.15.90:33434) 2024-04-25T01:16:50.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.161+0000 7f9775800640 1 -- 172.21.15.90:0/1791202783 learned_addr learned my addr 172.21.15.90:0/1791202783 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:50.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.162+0000 7f9774e00640 1 --2- 172.21.15.90:0/1791202783 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97700cf7f0 0x7f977014ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:50.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.162+0000 7f9776200640 1 --2- 172.21.15.90:0/1791202783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97701596c0 0x7f977014f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:50.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.162+0000 7f9775800640 1 -- 172.21.15.90:0/1791202783 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97700cf7f0 msgr2=0x7f977014ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:50.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.162+0000 7f9775800640 1 --2- 172.21.15.90:0/1791202783 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97700cf7f0 0x7f977014ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:50.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.162+0000 7f9775800640 1 -- 172.21.15.90:0/1791202783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97701596c0 msgr2=0x7f977014f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:50.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.162+0000 7f9775800640 1 --2- 172.21.15.90:0/1791202783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97701596c0 0x7f977014f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:50.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.162+0000 7f9775800640 1 -- 172.21.15.90:0/1791202783 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f977015d200 con 0x7f97700b2880 2024-04-25T01:16:50.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.162+0000 7f9775800640 1 --2- 172.21.15.90:0/1791202783 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97700b2880 0x7f97701f3a80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9764000c00 tx=0x7f97640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:50.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.162+0000 7f976d600640 1 -- 172.21.15.90:0/1791202783 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9764037040 con 0x7f97700b2880 2024-04-25T01:16:50.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.163+0000 7f976d600640 1 -- 172.21.15.90:0/1791202783 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9764042070 con 0x7f97700b2880 2024-04-25T01:16:50.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.163+0000 7f976d600640 1 -- 172.21.15.90:0/1791202783 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9764030880 con 0x7f97700b2880 2024-04-25T01:16:50.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.163+0000 7f9776c00640 1 -- 172.21.15.90:0/1791202783 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f977014fb00 con 0x7f97700b2880 2024-04-25T01:16:50.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.163+0000 7f9776c00640 1 -- 172.21.15.90:0/1791202783 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97701500c0 con 0x7f97700b2880 2024-04-25T01:16:50.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.164+0000 7f9776c00640 1 -- 172.21.15.90:0/1791202783 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9770075350 con 0x7f97700b2880 2024-04-25T01:16:50.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.165+0000 7f976d600640 1 -- 172.21.15.90:0/1791202783 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9764030a20 con 0x7f97700b2880 2024-04-25T01:16:50.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.166+0000 7f976d600640 1 --2- 172.21.15.90:0/1791202783 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f975006be30 0x7f975006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:50.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.166+0000 7f976d600640 1 -- 172.21.15.90:0/1791202783 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f97640aba70 con 0x7f97700b2880 2024-04-25T01:16:50.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.166+0000 7f9774e00640 1 --2- 172.21.15.90:0/1791202783 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f975006be30 0x7f975006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:50.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.167+0000 7f9774e00640 1 --2- 172.21.15.90:0/1791202783 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f975006be30 0x7f975006e2f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f977014cff0 tx=0x7f975c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:50.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.170+0000 7f976d600640 1 -- 172.21.15.90:0/1791202783 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9764045360 con 0x7f97700b2880 2024-04-25T01:16:50.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.403+0000 7f9776c00640 1 -- 172.21.15.90:0/1791202783 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f97701f3fc0 con 0x7f97700b2880 2024-04-25T01:16:50.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.931+0000 7f976d600640 1 -- 172.21.15.90:0/1791202783 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) ==== 122+0+0 (secure 0 0 0) 0x7f976407a0d0 con 0x7f97700b2880 2024-04-25T01:16:50.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.935+0000 7f9776c00640 1 -- 172.21.15.90:0/1791202783 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f975006be30 msgr2=0x7f975006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:50.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.935+0000 7f9776c00640 1 --2- 172.21.15.90:0/1791202783 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f975006be30 0x7f975006e2f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f977014cff0 tx=0x7f975c008040 comp rx=0 tx=0).stop 2024-04-25T01:16:50.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.935+0000 7f9776c00640 1 -- 172.21.15.90:0/1791202783 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97700b2880 msgr2=0x7f97701f3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:50.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.935+0000 7f9776c00640 1 --2- 172.21.15.90:0/1791202783 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97700b2880 0x7f97701f3a80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9764000c00 tx=0x7f97640046c0 comp rx=0 tx=0).stop 2024-04-25T01:16:50.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.935+0000 7f9776c00640 1 -- 172.21.15.90:0/1791202783 shutdown_connections 2024-04-25T01:16:50.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.935+0000 7f9776c00640 1 --2- 172.21.15.90:0/1791202783 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f975006be30 0x7f975006e2f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:50.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.935+0000 7f9776c00640 1 --2- 172.21.15.90:0/1791202783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97701596c0 0x7f977014f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:50.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.935+0000 7f9776c00640 1 --2- 172.21.15.90:0/1791202783 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f97700cf7f0 0x7f977014ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:50.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.935+0000 7f9776c00640 1 --2- 172.21.15.90:0/1791202783 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97700b2880 0x7f97701f3a80 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:50.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.935+0000 7f9776c00640 1 -- 172.21.15.90:0/1791202783 >> 172.21.15.90:0/1791202783 conn(0x7f977006b120 msgr2=0x7f97700cff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:50.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.935+0000 7f9776c00640 1 -- 172.21.15.90:0/1791202783 shutdown_connections 2024-04-25T01:16:50.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:50.935+0000 7f9776c00640 1 -- 172.21.15.90:0/1791202783 wait complete. 2024-04-25T01:16:50.940 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-25T01:16:50.941 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:16:51.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.157+0000 7f3267800640 1 Processor -- start 2024-04-25T01:16:51.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.158+0000 7f3267800640 1 -- start start 2024-04-25T01:16:51.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.159+0000 7f3267800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f32600cfaf0 0x7f32600d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:51.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.159+0000 7f3267800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32600d1e90 con 0x7f32600d4a10 2024-04-25T01:16:51.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.159+0000 7f3267800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32600b54a0 con 0x7f326015e4c0 2024-04-25T01:16:51.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.159+0000 7f3267800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f32600d2130 con 0x7f32600cfaf0 2024-04-25T01:16:51.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.159+0000 7f325fe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f32600d4a10 0x7f32600b3ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60736/0 (socket says 172.21.15.90:60736) 2024-04-25T01:16:51.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.159+0000 7f325fe00640 1 -- 172.21.15.90:0/1593504439 learned_addr learned my addr 172.21.15.90:0/1593504439 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:51.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.160+0000 7f3265800640 1 --2- 172.21.15.90:0/1593504439 >> v2:172.21.15.103:3300/0 conn(0x7f32600cfaf0 0x7f32600d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:51.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.160+0000 7f325f400640 1 -- 172.21.15.90:0/1593504439 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4224176971 0 0) 0x7f32600d1e90 con 0x7f32600d4a10 2024-04-25T01:16:51.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.160+0000 7f325f400640 1 -- 172.21.15.90:0/1593504439 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3248003520 con 0x7f32600d4a10 2024-04-25T01:16:51.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.160+0000 7f325f400640 1 -- 172.21.15.90:0/1593504439 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1593387071 0 0) 0x7f3248003520 con 0x7f32600d4a10 2024-04-25T01:16:51.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.160+0000 7f325f400640 1 -- 172.21.15.90:0/1593504439 >> v2:172.21.15.103:3300/0 conn(0x7f32600cfaf0 msgr2=0x7f32600d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:51.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.160+0000 7f325f400640 1 --2- 172.21.15.90:0/1593504439 >> v2:172.21.15.103:3300/0 conn(0x7f32600cfaf0 0x7f32600d0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:51.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.160+0000 7f325f400640 1 -- 172.21.15.90:0/1593504439 >> v1:172.21.15.103:6789/0 conn(0x7f326015e4c0 legacy=0x7f32600d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:51.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.160+0000 7f325f400640 1 -- 172.21.15.90:0/1593504439 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32600b5780 con 0x7f32600d4a10 2024-04-25T01:16:51.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.160+0000 7f325f400640 1 -- 172.21.15.90:0/1593504439 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f324c001670 con 0x7f32600d4a10 2024-04-25T01:16:51.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.160+0000 7f325f400640 1 -- 172.21.15.90:0/1593504439 >> v1:172.21.15.90:6789/0 conn(0x7f32600d4a10 legacy=0x7f32600b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:51.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.160+0000 7f325f400640 1 --2- 172.21.15.90:0/1593504439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32480039b0 0x7f3248003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:51.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.160+0000 7f325f400640 1 --2- 172.21.15.90:0/1593504439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3248004800 0x7f3248004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:51.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.160+0000 7f325f400640 1 --2- 172.21.15.90:0/1593504439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32480051c0 0x7f324800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:51.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.160+0000 7f325f400640 1 -- 172.21.15.90:0/1593504439 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f32600d2130 con 0x7f32480039b0 2024-04-25T01:16:51.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.160+0000 7f325f400640 1 -- 172.21.15.90:0/1593504439 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f324800c160 con 0x7f3248004800 2024-04-25T01:16:51.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.160+0000 7f325f400640 1 -- 172.21.15.90:0/1593504439 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f324800c360 con 0x7f32480051c0 2024-04-25T01:16:51.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.161+0000 7f3264e00640 1 --2- 172.21.15.90:0/1593504439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32480039b0 0x7f3248003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:51.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.161+0000 7f325fe00640 1 --2- 172.21.15.90:0/1593504439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3248004800 0x7f3248004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:51.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.161+0000 7f3264e00640 1 -- 172.21.15.90:0/1593504439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32480051c0 msgr2=0x7f324800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:51.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.161+0000 7f3265800640 1 --2- 172.21.15.90:0/1593504439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32480051c0 0x7f324800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:51.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.161+0000 7f3264e00640 1 --2- 172.21.15.90:0/1593504439 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32480051c0 0x7f324800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:51.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.161+0000 7f3264e00640 1 -- 172.21.15.90:0/1593504439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3248004800 msgr2=0x7f3248004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:51.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.161+0000 7f3264e00640 1 --2- 172.21.15.90:0/1593504439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3248004800 0x7f3248004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:51.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.161+0000 7f3264e00640 1 -- 172.21.15.90:0/1593504439 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3248003520 con 0x7f32480039b0 2024-04-25T01:16:51.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.161+0000 7f3265800640 1 -- 172.21.15.90:0/1593504439 reap_dead start 2024-04-25T01:16:51.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.161+0000 7f325fe00640 1 --2- 172.21.15.90:0/1593504439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3248004800 0x7f3248004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:51.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.162+0000 7f3264e00640 1 --2- 172.21.15.90:0/1593504439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32480039b0 0x7f3248003dd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3254001750 tx=0x7f3248005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=555cbf15a2989adf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:51.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.163+0000 7f325f400640 1 -- 172.21.15.90:0/1593504439 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3254001370 con 0x7f32480039b0 2024-04-25T01:16:51.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.163+0000 7f325f400640 1 -- 172.21.15.90:0/1593504439 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32540052d0 con 0x7f32480039b0 2024-04-25T01:16:51.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.163+0000 7f325f400640 1 -- 172.21.15.90:0/1593504439 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325402abe0 con 0x7f32480039b0 2024-04-25T01:16:51.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.163+0000 7f3267800640 1 -- 172.21.15.90:0/1593504439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32480039b0 msgr2=0x7f3248003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:51.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.163+0000 7f3267800640 1 --2- 172.21.15.90:0/1593504439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32480039b0 0x7f3248003dd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3254001750 tx=0x7f3248005830 comp rx=0 tx=0).stop 2024-04-25T01:16:51.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.163+0000 7f3267800640 1 -- 172.21.15.90:0/1593504439 shutdown_connections 2024-04-25T01:16:51.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.163+0000 7f3267800640 1 --2- 172.21.15.90:0/1593504439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32480039b0 0x7f3248003dd0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:51.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.163+0000 7f3267800640 1 -- 172.21.15.90:0/1593504439 >> 172.21.15.90:0/1593504439 conn(0x7f32600d3ef0 msgr2=0x7f32600d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:51.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.163+0000 7f3267800640 1 -- 172.21.15.90:0/1593504439 shutdown_connections 2024-04-25T01:16:51.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.163+0000 7f3267800640 1 -- 172.21.15.90:0/1593504439 wait complete. 2024-04-25T01:16:51.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.165+0000 7f3267800640 1 Processor -- start 2024-04-25T01:16:51.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.165+0000 7f3267800640 1 -- start start 2024-04-25T01:16:51.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.166+0000 7f3267800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32480039b0 0x7f3260156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:51.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.166+0000 7f3267800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32600cb980 0x7f32601f5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:51.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.166+0000 7f3267800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32601f55d0 0x7f32600c8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:51.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.166+0000 7f3267800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f326015ea20 con 0x7f32601f55d0 2024-04-25T01:16:51.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.166+0000 7f3267800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f326015e8a0 con 0x7f32600cb980 2024-04-25T01:16:51.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.166+0000 7f3267800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f32600d2800 con 0x7f32480039b0 2024-04-25T01:16:51.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.166+0000 7f3265800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32601f55d0 0x7f32600c8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:51.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.166+0000 7f3265800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32601f55d0 0x7f32600c8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33442/0 (socket says 172.21.15.90:33442) 2024-04-25T01:16:51.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.166+0000 7f3265800640 1 -- 172.21.15.90:0/1012186702 learned_addr learned my addr 172.21.15.90:0/1012186702 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:51.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.167+0000 7f325fe00640 1 --2- 172.21.15.90:0/1012186702 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32600cb980 0x7f32601f5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:51.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.167+0000 7f3264e00640 1 --2- 172.21.15.90:0/1012186702 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32480039b0 0x7f3260156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:51.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.167+0000 7f3265800640 1 -- 172.21.15.90:0/1012186702 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32480039b0 msgr2=0x7f3260156740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:51.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.167+0000 7f3265800640 1 --2- 172.21.15.90:0/1012186702 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32480039b0 0x7f3260156740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:51.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.167+0000 7f3265800640 1 -- 172.21.15.90:0/1012186702 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32600cb980 msgr2=0x7f32601f5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:51.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.167+0000 7f3265800640 1 --2- 172.21.15.90:0/1012186702 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32600cb980 0x7f32601f5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:51.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.167+0000 7f3265800640 1 -- 172.21.15.90:0/1012186702 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32600c7bc0 con 0x7f32601f55d0 2024-04-25T01:16:51.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.167+0000 7f3265800640 1 --2- 172.21.15.90:0/1012186702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32601f55d0 0x7f32600c8c60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3258009fd0 tx=0x7f325800f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:51.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.167+0000 7f325cc00640 1 -- 172.21.15.90:0/1012186702 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3258016620 con 0x7f32601f55d0 2024-04-25T01:16:51.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.167+0000 7f325cc00640 1 -- 172.21.15.90:0/1012186702 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f325801b070 con 0x7f32601f55d0 2024-04-25T01:16:51.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.167+0000 7f325cc00640 1 -- 172.21.15.90:0/1012186702 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3258016a20 con 0x7f32601f55d0 2024-04-25T01:16:51.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.167+0000 7f3267800640 1 -- 172.21.15.90:0/1012186702 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32600c7ea0 con 0x7f32601f55d0 2024-04-25T01:16:51.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.168+0000 7f3267800640 1 -- 172.21.15.90:0/1012186702 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32600c8460 con 0x7f32601f55d0 2024-04-25T01:16:51.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.169+0000 7f3267800640 1 -- 172.21.15.90:0/1012186702 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3260068fb0 con 0x7f32601f55d0 2024-04-25T01:16:51.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.169+0000 7f325cc00640 1 -- 172.21.15.90:0/1012186702 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3258016bc0 con 0x7f32601f55d0 2024-04-25T01:16:51.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.171+0000 7f325cc00640 1 --2- 172.21.15.90:0/1012186702 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f324006be30 0x7f324006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:51.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.171+0000 7f325cc00640 1 -- 172.21.15.90:0/1012186702 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f325808bd80 con 0x7f32601f55d0 2024-04-25T01:16:51.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.171+0000 7f3264e00640 1 --2- 172.21.15.90:0/1012186702 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f324006be30 0x7f324006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:51.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.172+0000 7f3264e00640 1 --2- 172.21.15.90:0/1012186702 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f324006be30 0x7f324006e2f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3254032000 tx=0x7f325402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:51.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.175+0000 7f325cc00640 1 -- 172.21.15.90:0/1012186702 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f3258024020 con 0x7f32601f55d0 2024-04-25T01:16:51.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.418+0000 7f3267800640 1 -- 172.21.15.90:0/1012186702 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f326006bdf0 con 0x7f32601f55d0 2024-04-25T01:16:51.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.419+0000 7f325cc00640 1 -- 172.21.15.90:0/1012186702 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5180 (secure 0 0 0) 0x7f325801d310 con 0x7f32601f55d0 2024-04-25T01:16:51.403 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:16:51.403 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6834/2605945289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":2605945289},{"type":"v1","addr":"172.21.15.103:6834","nonce":2605945289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6836/704639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":704639426},{"type":"v1","addr":"172.21.15.90:6836","nonce":704639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4245,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3477867755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6835","nonce":3477867755},{"type":"v1","addr":"172.21.15.90:6837","nonce":3477867755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:16:22.060018+0000","modified":"2024-04-25T01:16:50.405431+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6835/4291578198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6833","nonce":4291578198},{"type":"v1","addr":"172.21.15.103:6835","nonce":4291578198}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1}]} 2024-04-25T01:16:51.404 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 9 2024-04-25T01:16:51.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.422+0000 7f3267800640 1 -- 172.21.15.90:0/1012186702 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f324006be30 msgr2=0x7f324006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:51.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.422+0000 7f3267800640 1 --2- 172.21.15.90:0/1012186702 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f324006be30 0x7f324006e2f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3254032000 tx=0x7f325402e000 comp rx=0 tx=0).stop 2024-04-25T01:16:51.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.422+0000 7f3267800640 1 -- 172.21.15.90:0/1012186702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32601f55d0 msgr2=0x7f32600c8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:51.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.422+0000 7f3267800640 1 --2- 172.21.15.90:0/1012186702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32601f55d0 0x7f32600c8c60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3258009fd0 tx=0x7f325800f520 comp rx=0 tx=0).stop 2024-04-25T01:16:51.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.422+0000 7f3267800640 1 -- 172.21.15.90:0/1012186702 shutdown_connections 2024-04-25T01:16:51.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.422+0000 7f3267800640 1 --2- 172.21.15.90:0/1012186702 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f324006be30 0x7f324006e2f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:51.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.422+0000 7f3267800640 1 --2- 172.21.15.90:0/1012186702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32601f55d0 0x7f32600c8c60 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:51.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.422+0000 7f3267800640 1 --2- 172.21.15.90:0/1012186702 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32600cb980 0x7f32601f5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:51.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.422+0000 7f3267800640 1 --2- 172.21.15.90:0/1012186702 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f32480039b0 0x7f3260156740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:51.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.422+0000 7f3267800640 1 -- 172.21.15.90:0/1012186702 >> 172.21.15.90:0/1012186702 conn(0x7f32600d3ef0 msgr2=0x7f32600d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:51.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.422+0000 7f3267800640 1 -- 172.21.15.90:0/1012186702 shutdown_connections 2024-04-25T01:16:51.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.422+0000 7f3267800640 1 -- 172.21.15.90:0/1012186702 wait complete. 2024-04-25T01:16:51.426 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:16:51.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.635+0000 7fbd4c800640 1 Processor -- start 2024-04-25T01:16:51.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.635+0000 7fbd4c800640 1 -- start start 2024-04-25T01:16:51.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.636+0000 7fbd4c800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbd4415e330 0x7fbd440b5a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:51.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.636+0000 7fbd4c800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd440672d0 0x7fbd4406b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:51.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.636+0000 7fbd4c800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd4415e710 con 0x7fbd440b1380 2024-04-25T01:16:51.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.637+0000 7fbd4c800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fbd440b0ef0 con 0x7fbd4415e330 2024-04-25T01:16:51.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.637+0000 7fbd4c800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbd440b05c0 con 0x7fbd440672d0 2024-04-25T01:16:51.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.637+0000 7fbd4a800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fbd440b1380 0x7fbd440d0290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60742/0 (socket says 172.21.15.90:60742) 2024-04-25T01:16:51.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.637+0000 7fbd4a800640 1 -- 172.21.15.90:0/3335644576 learned_addr learned my addr 172.21.15.90:0/3335644576 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:51.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.637+0000 7fbd49e00640 1 --2- 172.21.15.90:0/3335644576 >> v2:172.21.15.103:3300/0 conn(0x7fbd4415e330 0x7fbd440b5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:51.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.637+0000 7fbd49400640 1 --2- 172.21.15.90:0/3335644576 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd440672d0 0x7fbd4406b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:51.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.637+0000 7fbd48a00640 1 -- 172.21.15.90:0/3335644576 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3642189091 0 0) 0x7fbd4415e710 con 0x7fbd440b1380 2024-04-25T01:16:51.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.637+0000 7fbd48a00640 1 -- 172.21.15.90:0/3335644576 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd2c003520 con 0x7fbd440b1380 2024-04-25T01:16:51.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.638+0000 7fbd48a00640 1 -- 172.21.15.90:0/3335644576 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3425836172 0 0) 0x7fbd2c003520 con 0x7fbd440b1380 2024-04-25T01:16:51.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.638+0000 7fbd48a00640 1 -- 172.21.15.90:0/3335644576 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd440672d0 msgr2=0x7fbd4406b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:51.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.638+0000 7fbd48a00640 1 --2- 172.21.15.90:0/3335644576 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd440672d0 0x7fbd4406b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:51.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.638+0000 7fbd48a00640 1 -- 172.21.15.90:0/3335644576 >> v2:172.21.15.103:3300/0 conn(0x7fbd4415e330 msgr2=0x7fbd440b5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:51.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.638+0000 7fbd48a00640 1 --2- 172.21.15.90:0/3335644576 >> v2:172.21.15.103:3300/0 conn(0x7fbd4415e330 0x7fbd440b5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:51.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.638+0000 7fbd48a00640 1 -- 172.21.15.90:0/3335644576 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd440b08a0 con 0x7fbd440b1380 2024-04-25T01:16:51.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.638+0000 7fbd48a00640 1 -- 172.21.15.90:0/3335644576 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7fbd400019f0 con 0x7fbd440b1380 2024-04-25T01:16:51.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.638+0000 7fbd48a00640 1 -- 172.21.15.90:0/3335644576 >> v1:172.21.15.90:6789/0 conn(0x7fbd440b1380 legacy=0x7fbd440d0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:51.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.638+0000 7fbd48a00640 1 --2- 172.21.15.90:0/3335644576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd2c0039b0 0x7fbd2c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:51.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.638+0000 7fbd48a00640 1 --2- 172.21.15.90:0/3335644576 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd440672d0 0x7fbd2c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:51.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.638+0000 7fbd48a00640 1 --2- 172.21.15.90:0/3335644576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd2c004d40 0x7fbd2c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:51.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.638+0000 7fbd48a00640 1 -- 172.21.15.90:0/3335644576 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbd440b0ef0 con 0x7fbd2c004d40 2024-04-25T01:16:51.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.638+0000 7fbd48a00640 1 -- 172.21.15.90:0/3335644576 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbd440b05c0 con 0x7fbd2c0039b0 2024-04-25T01:16:51.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.638+0000 7fbd48a00640 1 -- 172.21.15.90:0/3335644576 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbd2c00b9f0 con 0x7fbd440672d0 2024-04-25T01:16:51.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.638+0000 7fbd4a800640 1 --2- 172.21.15.90:0/3335644576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd2c004d40 0x7fbd2c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:51.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.638+0000 7fbd49400640 1 --2- 172.21.15.90:0/3335644576 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd440672d0 0x7fbd2c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:51.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.638+0000 7fbd49e00640 1 --2- 172.21.15.90:0/3335644576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd2c0039b0 0x7fbd2c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:51.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.639+0000 7fbd4a800640 1 -- 172.21.15.90:0/3335644576 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd440672d0 msgr2=0x7fbd2c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:51.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.639+0000 7fbd4a800640 1 --2- 172.21.15.90:0/3335644576 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd440672d0 0x7fbd2c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:51.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.639+0000 7fbd4a800640 1 -- 172.21.15.90:0/3335644576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd2c0039b0 msgr2=0x7fbd2c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:51.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.639+0000 7fbd4a800640 1 --2- 172.21.15.90:0/3335644576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd2c0039b0 0x7fbd2c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:51.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.639+0000 7fbd4a800640 1 -- 172.21.15.90:0/3335644576 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd2c003520 con 0x7fbd2c004d40 2024-04-25T01:16:51.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.639+0000 7fbd49e00640 1 --2- 172.21.15.90:0/3335644576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd2c0039b0 0x7fbd2c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:16:51.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.639+0000 7fbd49400640 1 --2- 172.21.15.90:0/3335644576 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd440672d0 0x7fbd2c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:16:51.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.640+0000 7fbd4a800640 1 --2- 172.21.15.90:0/3335644576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd2c004d40 0x7fbd2c0051c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbd4000a7d0 tx=0x7fbd2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7b1adaf054091cb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:51.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.640+0000 7fbd48a00640 1 -- 172.21.15.90:0/3335644576 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4000a3f0 con 0x7fbd2c004d40 2024-04-25T01:16:51.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.640+0000 7fbd48a00640 1 -- 172.21.15.90:0/3335644576 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd4002f070 con 0x7fbd2c004d40 2024-04-25T01:16:51.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.640+0000 7fbd48a00640 1 -- 172.21.15.90:0/3335644576 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4002bc50 con 0x7fbd2c004d40 2024-04-25T01:16:51.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.641+0000 7fbd4c800640 1 -- 172.21.15.90:0/3335644576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd2c004d40 msgr2=0x7fbd2c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:51.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.641+0000 7fbd4c800640 1 --2- 172.21.15.90:0/3335644576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd2c004d40 0x7fbd2c0051c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbd4000a7d0 tx=0x7fbd2c005830 comp rx=0 tx=0).stop 2024-04-25T01:16:51.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.641+0000 7fbd4a800640 1 -- 172.21.15.90:0/3335644576 reap_dead start 2024-04-25T01:16:51.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.641+0000 7fbd4c800640 1 -- 172.21.15.90:0/3335644576 shutdown_connections 2024-04-25T01:16:51.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.641+0000 7fbd4c800640 1 -- 172.21.15.90:0/3335644576 >> 172.21.15.90:0/3335644576 conn(0x7fbd44072920 msgr2=0x7fbd44072d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:51.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.641+0000 7fbd4c800640 1 -- 172.21.15.90:0/3335644576 shutdown_connections 2024-04-25T01:16:51.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.641+0000 7fbd4c800640 1 -- 172.21.15.90:0/3335644576 wait complete. 2024-04-25T01:16:51.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.643+0000 7fbd4c800640 1 Processor -- start 2024-04-25T01:16:51.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.643+0000 7fbd4c800640 1 -- start start 2024-04-25T01:16:51.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.644+0000 7fbd4c800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd44153f60 0x7fbd44154660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:51.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.644+0000 7fbd4c800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd440cc7b0 0x7fbd44161080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:51.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.644+0000 7fbd4c800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd441615c0 0x7fbd440c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:51.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.644+0000 7fbd4c800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbd440b5f80 con 0x7fbd44153f60 2024-04-25T01:16:51.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.644+0000 7fbd4c800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbd440b0270 con 0x7fbd440cc7b0 2024-04-25T01:16:51.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.644+0000 7fbd4c800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbd440b6100 con 0x7fbd441615c0 2024-04-25T01:16:51.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.644+0000 7fbd49e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd44153f60 0x7fbd44154660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:51.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.644+0000 7fbd49e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd44153f60 0x7fbd44154660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33462/0 (socket says 172.21.15.90:33462) 2024-04-25T01:16:51.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.644+0000 7fbd49e00640 1 -- 172.21.15.90:0/520797357 learned_addr learned my addr 172.21.15.90:0/520797357 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:51.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.645+0000 7fbd4a800640 1 --2- 172.21.15.90:0/520797357 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd441615c0 0x7fbd440c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:51.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.645+0000 7fbd49e00640 1 -- 172.21.15.90:0/520797357 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd441615c0 msgr2=0x7fbd440c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:51.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.645+0000 7fbd49e00640 1 --2- 172.21.15.90:0/520797357 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd441615c0 0x7fbd440c8f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:51.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.645+0000 7fbd49e00640 1 -- 172.21.15.90:0/520797357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd440cc7b0 msgr2=0x7fbd44161080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:51.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.645+0000 7fbd49400640 1 --2- 172.21.15.90:0/520797357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd440cc7b0 0x7fbd44161080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:51.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.645+0000 7fbd49e00640 1 --2- 172.21.15.90:0/520797357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd440cc7b0 0x7fbd44161080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:51.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.645+0000 7fbd49e00640 1 -- 172.21.15.90:0/520797357 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd440ccf20 con 0x7fbd44153f60 2024-04-25T01:16:51.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.645+0000 7fbd49e00640 1 --2- 172.21.15.90:0/520797357 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd44153f60 0x7fbd44154660 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbd380115d0 tx=0x7fbd38011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:51.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.645+0000 7fbd3e000640 1 -- 172.21.15.90:0/520797357 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3800f8f0 con 0x7fbd44153f60 2024-04-25T01:16:51.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.645+0000 7fbd3e000640 1 -- 172.21.15.90:0/520797357 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd3800fa90 con 0x7fbd44153f60 2024-04-25T01:16:51.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.645+0000 7fbd3e000640 1 -- 172.21.15.90:0/520797357 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3800fe60 con 0x7fbd44153f60 2024-04-25T01:16:51.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.645+0000 7fbd4c800640 1 -- 172.21.15.90:0/520797357 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd440cd200 con 0x7fbd44153f60 2024-04-25T01:16:51.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.646+0000 7fbd4c800640 1 -- 172.21.15.90:0/520797357 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd440cd7c0 con 0x7fbd44153f60 2024-04-25T01:16:51.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.647+0000 7fbd4c800640 1 -- 172.21.15.90:0/520797357 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd440c6cd0 con 0x7fbd44153f60 2024-04-25T01:16:51.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.648+0000 7fbd3e000640 1 -- 172.21.15.90:0/520797357 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbd38015460 con 0x7fbd44153f60 2024-04-25T01:16:51.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.649+0000 7fbd3e000640 1 --2- 172.21.15.90:0/520797357 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbd2806be30 0x7fbd2806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:51.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.649+0000 7fbd3e000640 1 -- 172.21.15.90:0/520797357 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fbd3808bb70 con 0x7fbd44153f60 2024-04-25T01:16:51.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.649+0000 7fbd49400640 1 --2- 172.21.15.90:0/520797357 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbd2806be30 0x7fbd2806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:51.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.650+0000 7fbd49400640 1 --2- 172.21.15.90:0/520797357 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbd2806be30 0x7fbd2806e2f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbd4406f5d0 tx=0x7fbd34008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:51.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.653+0000 7fbd3e000640 1 -- 172.21.15.90:0/520797357 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fbd440c6cd0 con 0x7fbd44153f60 2024-04-25T01:16:51.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.853+0000 7fbd4c800640 1 -- 172.21.15.90:0/520797357 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbd44068180 con 0x7fbd44153f60 2024-04-25T01:16:51.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.855+0000 7fbd3e000640 1 -- 172.21.15.90:0/520797357 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18482 (secure 0 0 0) 0x7fbd38019030 con 0x7fbd44153f60 2024-04-25T01:16:51.839 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:16:51.840 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":26,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:16:27.092855+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:16:20.094512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:16:21.078268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:16:24.670642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:16:51.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.858+0000 7fbd4c800640 1 -- 172.21.15.90:0/520797357 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbd2806be30 msgr2=0x7fbd2806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:51.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.858+0000 7fbd4c800640 1 --2- 172.21.15.90:0/520797357 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbd2806be30 0x7fbd2806e2f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbd4406f5d0 tx=0x7fbd34008040 comp rx=0 tx=0).stop 2024-04-25T01:16:51.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.858+0000 7fbd4c800640 1 -- 172.21.15.90:0/520797357 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd44153f60 msgr2=0x7fbd44154660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:51.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.858+0000 7fbd4c800640 1 --2- 172.21.15.90:0/520797357 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd44153f60 0x7fbd44154660 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbd380115d0 tx=0x7fbd38011a90 comp rx=0 tx=0).stop 2024-04-25T01:16:51.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.858+0000 7fbd4c800640 1 -- 172.21.15.90:0/520797357 shutdown_connections 2024-04-25T01:16:51.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.858+0000 7fbd4c800640 1 --2- 172.21.15.90:0/520797357 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbd2806be30 0x7fbd2806e2f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:51.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.858+0000 7fbd4c800640 1 --2- 172.21.15.90:0/520797357 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd441615c0 0x7fbd440c8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:51.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.858+0000 7fbd4c800640 1 --2- 172.21.15.90:0/520797357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd440cc7b0 0x7fbd44161080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:51.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.858+0000 7fbd4c800640 1 --2- 172.21.15.90:0/520797357 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd44153f60 0x7fbd44154660 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:51.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.858+0000 7fbd4c800640 1 -- 172.21.15.90:0/520797357 >> 172.21.15.90:0/520797357 conn(0x7fbd44072920 msgr2=0x7fbd4406d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:51.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.858+0000 7fbd4c800640 1 -- 172.21.15.90:0/520797357 shutdown_connections 2024-04-25T01:16:51.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:51.858+0000 7fbd4c800640 1 -- 172.21.15.90:0/520797357 wait complete. 2024-04-25T01:16:51.861 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T01:16:51.861 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:16:52.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.079+0000 7f018ce00640 1 Processor -- start 2024-04-25T01:16:52.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.079+0000 7f018ce00640 1 -- start start 2024-04-25T01:16:52.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.081+0000 7f018ce00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f01880ae9d0 0x7f018815ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:52.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.081+0000 7f018ce00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01880adc10 con 0x7f018806b830 2024-04-25T01:16:52.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.081+0000 7f018ce00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01880adef0 con 0x7f01880b41f0 2024-04-25T01:16:52.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.081+0000 7f018ce00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f01880ae1d0 con 0x7f01880ae9d0 2024-04-25T01:16:52.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.081+0000 7f0187e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f01880ae9d0 0x7f018815ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:52.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.081+0000 7f0187e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f01880ae9d0 0x7f018815ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33478/0 (socket says 172.21.15.90:33478) 2024-04-25T01:16:52.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.081+0000 7f017fe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f018806b830 0x7f018815e6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60744/0 (socket says 172.21.15.90:60744) 2024-04-25T01:16:52.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.081+0000 7f0187e00640 1 -- 172.21.15.90:0/2588469999 learned_addr learned my addr 172.21.15.90:0/2588469999 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:52.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.081+0000 7f0187e00640 1 -- 172.21.15.90:0/2588469999 >> v1:172.21.15.103:6789/0 conn(0x7f01880b41f0 legacy=0x7f018806b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:52.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.081+0000 7f0187e00640 1 -- 172.21.15.90:0/2588469999 >> v1:172.21.15.90:6789/0 conn(0x7f018806b830 legacy=0x7f018815e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:52.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.081+0000 7f0187e00640 1 -- 172.21.15.90:0/2588469999 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01880ae4b0 con 0x7f01880ae9d0 2024-04-25T01:16:52.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.081+0000 7f0186a00640 1 -- 172.21.15.90:0/2588469999 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1551946352 0 0) 0x7f01880adc10 con 0x7f018806b830 2024-04-25T01:16:52.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.082+0000 7f0187e00640 1 --2- 172.21.15.90:0/2588469999 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01880ae9d0 0x7f018815ee00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0180009e50 tx=0x7f018002a760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3afa555b263404e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:52.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.083+0000 7f0186a00640 1 -- 172.21.15.90:0/2588469999 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0180036070 con 0x7f01880ae9d0 2024-04-25T01:16:52.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.083+0000 7f0186a00640 1 -- 172.21.15.90:0/2588469999 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0180031600 con 0x7f01880ae9d0 2024-04-25T01:16:52.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.083+0000 7f0186a00640 1 -- 172.21.15.90:0/2588469999 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01800319d0 con 0x7f01880ae9d0 2024-04-25T01:16:52.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.083+0000 7f018ce00640 1 -- 172.21.15.90:0/2588469999 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01880ae9d0 msgr2=0x7f018815ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:52.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.083+0000 7f018ce00640 1 --2- 172.21.15.90:0/2588469999 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01880ae9d0 0x7f018815ee00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0180009e50 tx=0x7f018002a760 comp rx=0 tx=0).stop 2024-04-25T01:16:52.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.083+0000 7f018ce00640 1 -- 172.21.15.90:0/2588469999 shutdown_connections 2024-04-25T01:16:52.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.083+0000 7f018ce00640 1 --2- 172.21.15.90:0/2588469999 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01880ae9d0 0x7f018815ee00 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:52.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.083+0000 7f018ce00640 1 -- 172.21.15.90:0/2588469999 >> 172.21.15.90:0/2588469999 conn(0x7f01880b1660 msgr2=0x7f01880b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:52.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.084+0000 7f018ce00640 1 -- 172.21.15.90:0/2588469999 shutdown_connections 2024-04-25T01:16:52.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.084+0000 7f018ce00640 1 -- 172.21.15.90:0/2588469999 wait complete. 2024-04-25T01:16:52.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.085+0000 7f018ce00640 1 Processor -- start 2024-04-25T01:16:52.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.085+0000 7f018ce00640 1 -- start start 2024-04-25T01:16:52.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.086+0000 7f018ce00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f018806b830 0x7f0188158df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:52.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.086+0000 7f018ce00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01880ae9d0 0x7f01881572e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:52.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.086+0000 7f018ce00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01880b41f0 0x7f01881e84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:52.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.086+0000 7f018ce00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0188158000 con 0x7f01880b41f0 2024-04-25T01:16:52.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.086+0000 7f018ce00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0188157e80 con 0x7f01880ae9d0 2024-04-25T01:16:52.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.086+0000 7f018ce00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0188158180 con 0x7f018806b830 2024-04-25T01:16:52.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.087+0000 7f0187e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01880b41f0 0x7f01881e84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:52.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.087+0000 7f0187e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01880b41f0 0x7f01881e84e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33492/0 (socket says 172.21.15.90:33492) 2024-04-25T01:16:52.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.087+0000 7f0187e00640 1 -- 172.21.15.90:0/3310839454 learned_addr learned my addr 172.21.15.90:0/3310839454 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:52.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.087+0000 7f0187400640 1 --2- 172.21.15.90:0/3310839454 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f018806b830 0x7f0188158df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:52.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.087+0000 7f0187e00640 1 -- 172.21.15.90:0/3310839454 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f018806b830 msgr2=0x7f0188158df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:52.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.087+0000 7f0187e00640 1 --2- 172.21.15.90:0/3310839454 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f018806b830 0x7f0188158df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:52.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.087+0000 7f017fe00640 1 --2- 172.21.15.90:0/3310839454 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01880ae9d0 0x7f01881572e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:52.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.087+0000 7f0187e00640 1 -- 172.21.15.90:0/3310839454 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01880ae9d0 msgr2=0x7f01881572e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:52.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.087+0000 7f0187e00640 1 --2- 172.21.15.90:0/3310839454 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01880ae9d0 0x7f01881572e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:52.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.087+0000 7f0187e00640 1 -- 172.21.15.90:0/3310839454 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0188159490 con 0x7f01880b41f0 2024-04-25T01:16:52.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.087+0000 7f0187400640 1 --2- 172.21.15.90:0/3310839454 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f018806b830 0x7f0188158df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:52.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.087+0000 7f017fe00640 1 --2- 172.21.15.90:0/3310839454 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01880ae9d0 0x7f01881572e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:52.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.088+0000 7f0187e00640 1 --2- 172.21.15.90:0/3310839454 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01880b41f0 0x7f01881e84e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0180034040 tx=0x7f0180004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:52.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.088+0000 7f017f400640 1 -- 172.21.15.90:0/3310839454 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0180041020 con 0x7f01880b41f0 2024-04-25T01:16:52.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.088+0000 7f017f400640 1 -- 172.21.15.90:0/3310839454 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0180007780 con 0x7f01880b41f0 2024-04-25T01:16:52.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.088+0000 7f017f400640 1 -- 172.21.15.90:0/3310839454 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0180036040 con 0x7f01880b41f0 2024-04-25T01:16:52.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.088+0000 7f018ce00640 1 -- 172.21.15.90:0/3310839454 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01881e7720 con 0x7f01880b41f0 2024-04-25T01:16:52.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.088+0000 7f018ce00640 1 -- 172.21.15.90:0/3310839454 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01881e7fc0 con 0x7f01880b41f0 2024-04-25T01:16:52.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.089+0000 7f018ce00640 1 -- 172.21.15.90:0/3310839454 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0188075350 con 0x7f01880b41f0 2024-04-25T01:16:52.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.090+0000 7f017f400640 1 -- 172.21.15.90:0/3310839454 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0180044020 con 0x7f01880b41f0 2024-04-25T01:16:52.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.091+0000 7f017f400640 1 --2- 172.21.15.90:0/3310839454 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f016406be30 0x7f016406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:52.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.091+0000 7f017f400640 1 -- 172.21.15.90:0/3310839454 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f01800abed0 con 0x7f01880b41f0 2024-04-25T01:16:52.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.092+0000 7f0187400640 1 --2- 172.21.15.90:0/3310839454 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f016406be30 0x7f016406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:52.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.092+0000 7f0187400640 1 --2- 172.21.15.90:0/3310839454 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f016406be30 0x7f016406e2f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f017800a830 tx=0x7f0178008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:52.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.096+0000 7f017f400640 1 -- 172.21.15.90:0/3310839454 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f018004b030 con 0x7f01880b41f0 2024-04-25T01:16:52.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.289+0000 7f018ce00640 1 -- 172.21.15.90:0/3310839454 --> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f01881e7ab0 con 0x7f016406be30 2024-04-25T01:16:52.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.318+0000 7f017f400640 1 -- 172.21.15.90:0/3310839454 <== mgr.4104 v2:172.21.15.90:6832/2198086733 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332534 (secure 0 0 0) 0x7f01881e7ab0 con 0x7f016406be30 2024-04-25T01:16:52.307 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:16:52.309 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-25T01:16:52.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.326+0000 7f018ce00640 1 -- 172.21.15.90:0/3310839454 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f016406be30 msgr2=0x7f016406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:52.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.326+0000 7f018ce00640 1 --2- 172.21.15.90:0/3310839454 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f016406be30 0x7f016406e2f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f017800a830 tx=0x7f0178008040 comp rx=0 tx=0).stop 2024-04-25T01:16:52.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.326+0000 7f018ce00640 1 -- 172.21.15.90:0/3310839454 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01880b41f0 msgr2=0x7f01881e84e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:52.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.326+0000 7f018ce00640 1 --2- 172.21.15.90:0/3310839454 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01880b41f0 0x7f01881e84e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0180034040 tx=0x7f0180004a40 comp rx=0 tx=0).stop 2024-04-25T01:16:52.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.326+0000 7f018ce00640 1 -- 172.21.15.90:0/3310839454 shutdown_connections 2024-04-25T01:16:52.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.326+0000 7f018ce00640 1 --2- 172.21.15.90:0/3310839454 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f016406be30 0x7f016406e2f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:52.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.326+0000 7f018ce00640 1 --2- 172.21.15.90:0/3310839454 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01880b41f0 0x7f01881e84e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:52.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.326+0000 7f018ce00640 1 --2- 172.21.15.90:0/3310839454 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01880ae9d0 0x7f01881572e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:52.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.326+0000 7f018ce00640 1 --2- 172.21.15.90:0/3310839454 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f018806b830 0x7f0188158df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:52.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.326+0000 7f018ce00640 1 -- 172.21.15.90:0/3310839454 >> 172.21.15.90:0/3310839454 conn(0x7f01880b1660 msgr2=0x7f01880afc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:52.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.326+0000 7f018ce00640 1 -- 172.21.15.90:0/3310839454 shutdown_connections 2024-04-25T01:16:52.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.326+0000 7f018ce00640 1 -- 172.21.15.90:0/3310839454 wait complete. 2024-04-25T01:16:52.330 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-04-25T01:16:51.833739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219536,"kb_used_data":3368,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755184,"statfs":{"total":773094113280,"available":772869308416,"internally_reserved":0,"allocated":3448832,"data_stored":2055328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1781,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004702"},"pg_stats":[{"pgid":"2.1f","version":"21'4","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.698700+0000","last_change":"2024-04-25T01:16:21.622879+0000","last_active":"2024-04-25T01:16:27.698700+0000","last_peered":"2024-04-25T01:16:27.698700+0000","last_clean":"2024-04-25T01:16:27.698700+0000","last_became_active":"2024-04-25T01:16:21.622586+0000","last_became_peered":"2024-04-25T01:16:21.622586+0000","last_unstale":"2024-04-25T01:16:27.698700+0000","last_undegraded":"2024-04-25T01:16:27.698700+0000","last_fullsized":"2024-04-25T01:16:27.698700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:50.882927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699904+0000","last_change":"2024-04-25T01:16:22.615827+0000","last_active":"2024-04-25T01:16:27.699904+0000","last_peered":"2024-04-25T01:16:27.699904+0000","last_clean":"2024-04-25T01:16:27.699904+0000","last_became_active":"2024-04-25T01:16:22.615497+0000","last_became_peered":"2024-04-25T01:16:22.615497+0000","last_unstale":"2024-04-25T01:16:27.699904+0000","last_undegraded":"2024-04-25T01:16:27.699904+0000","last_fullsized":"2024-04-25T01:16:27.699904+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:03:53.876481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.223962+0000","last_change":"2024-04-25T01:16:26.672437+0000","last_active":"2024-04-25T01:16:28.223962+0000","last_peered":"2024-04-25T01:16:28.223962+0000","last_clean":"2024-04-25T01:16:28.223962+0000","last_became_active":"2024-04-25T01:16:26.671999+0000","last_became_peered":"2024-04-25T01:16:26.671999+0000","last_unstale":"2024-04-25T01:16:28.223962+0000","last_undegraded":"2024-04-25T01:16:28.223962+0000","last_fullsized":"2024-04-25T01:16:28.223962+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:56.586900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4,1],"acting":[7,6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.700145+0000","last_change":"2024-04-25T01:16:21.609447+0000","last_active":"2024-04-25T01:16:27.700145+0000","last_peered":"2024-04-25T01:16:27.700145+0000","last_clean":"2024-04-25T01:16:27.700145+0000","last_became_active":"2024-04-25T01:16:21.608490+0000","last_became_peered":"2024-04-25T01:16:21.608490+0000","last_unstale":"2024-04-25T01:16:27.700145+0000","last_undegraded":"2024-04-25T01:16:27.700145+0000","last_fullsized":"2024-04-25T01:16:27.700145+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:15.007939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699632+0000","last_change":"2024-04-25T01:16:22.613891+0000","last_active":"2024-04-25T01:16:27.699632+0000","last_peered":"2024-04-25T01:16:27.699632+0000","last_clean":"2024-04-25T01:16:27.699632+0000","last_became_active":"2024-04-25T01:16:22.613471+0000","last_became_peered":"2024-04-25T01:16:22.613471+0000","last_unstale":"2024-04-25T01:16:27.699632+0000","last_undegraded":"2024-04-25T01:16:27.699632+0000","last_fullsized":"2024-04-25T01:16:27.699632+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:37:00.182534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.874736+0000","last_change":"2024-04-25T01:16:26.653705+0000","last_active":"2024-04-25T01:16:27.874736+0000","last_peered":"2024-04-25T01:16:27.874736+0000","last_clean":"2024-04-25T01:16:27.874736+0000","last_became_active":"2024-04-25T01:16:26.653348+0000","last_became_peered":"2024-04-25T01:16:26.653348+0000","last_unstale":"2024-04-25T01:16:27.874736+0000","last_undegraded":"2024-04-25T01:16:27.874736+0000","last_fullsized":"2024-04-25T01:16:27.874736+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:58:37.794558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4,6],"acting":[2,7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.772478+0000","last_change":"2024-04-25T01:16:21.611644+0000","last_active":"2024-04-25T01:16:29.772478+0000","last_peered":"2024-04-25T01:16:29.772478+0000","last_clean":"2024-04-25T01:16:29.772478+0000","last_became_active":"2024-04-25T01:16:21.611346+0000","last_became_peered":"2024-04-25T01:16:21.611346+0000","last_unstale":"2024-04-25T01:16:29.772478+0000","last_undegraded":"2024-04-25T01:16:29.772478+0000","last_fullsized":"2024-04-25T01:16:29.772478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:44.675184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.971313+0000","last_change":"2024-04-25T01:16:22.627961+0000","last_active":"2024-04-25T01:16:27.971313+0000","last_peered":"2024-04-25T01:16:27.971313+0000","last_clean":"2024-04-25T01:16:27.971313+0000","last_became_active":"2024-04-25T01:16:22.626767+0000","last_became_peered":"2024-04-25T01:16:22.626767+0000","last_unstale":"2024-04-25T01:16:27.971313+0000","last_undegraded":"2024-04-25T01:16:27.971313+0000","last_fullsized":"2024-04-25T01:16:27.971313+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:49:05.483350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.875764+0000","last_change":"2024-04-25T01:16:26.657547+0000","last_active":"2024-04-25T01:16:27.875764+0000","last_peered":"2024-04-25T01:16:27.875764+0000","last_clean":"2024-04-25T01:16:27.875764+0000","last_became_active":"2024-04-25T01:16:26.656643+0000","last_became_peered":"2024-04-25T01:16:26.656643+0000","last_unstale":"2024-04-25T01:16:27.875764+0000","last_undegraded":"2024-04-25T01:16:27.875764+0000","last_fullsized":"2024-04-25T01:16:27.875764+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:05.563072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3,6],"acting":[2,5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.772473+0000","last_change":"2024-04-25T01:16:21.617309+0000","last_active":"2024-04-25T01:16:29.772473+0000","last_peered":"2024-04-25T01:16:29.772473+0000","last_clean":"2024-04-25T01:16:29.772473+0000","last_became_active":"2024-04-25T01:16:21.617021+0000","last_became_peered":"2024-04-25T01:16:21.617021+0000","last_unstale":"2024-04-25T01:16:29.772473+0000","last_undegraded":"2024-04-25T01:16:29.772473+0000","last_fullsized":"2024-04-25T01:16:29.772473+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:06.940097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694375+0000","last_change":"2024-04-25T01:16:22.639034+0000","last_active":"2024-04-25T01:16:27.694375+0000","last_peered":"2024-04-25T01:16:27.694375+0000","last_clean":"2024-04-25T01:16:27.694375+0000","last_became_active":"2024-04-25T01:16:22.638661+0000","last_became_peered":"2024-04-25T01:16:22.638661+0000","last_unstale":"2024-04-25T01:16:27.694375+0000","last_undegraded":"2024-04-25T01:16:27.694375+0000","last_fullsized":"2024-04-25T01:16:27.694375+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:28.044482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.875219+0000","last_change":"2024-04-25T01:16:26.656419+0000","last_active":"2024-04-25T01:16:27.875219+0000","last_peered":"2024-04-25T01:16:27.875219+0000","last_clean":"2024-04-25T01:16:27.875219+0000","last_became_active":"2024-04-25T01:16:26.655743+0000","last_became_peered":"2024-04-25T01:16:26.655743+0000","last_unstale":"2024-04-25T01:16:27.875219+0000","last_undegraded":"2024-04-25T01:16:27.875219+0000","last_fullsized":"2024-04-25T01:16:27.875219+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:41:04.510055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5,7],"acting":[2,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694711+0000","last_change":"2024-04-25T01:16:22.621126+0000","last_active":"2024-04-25T01:16:27.694711+0000","last_peered":"2024-04-25T01:16:27.694711+0000","last_clean":"2024-04-25T01:16:27.694711+0000","last_became_active":"2024-04-25T01:16:22.620692+0000","last_became_peered":"2024-04-25T01:16:22.620692+0000","last_unstale":"2024-04-25T01:16:27.694711+0000","last_undegraded":"2024-04-25T01:16:27.694711+0000","last_fullsized":"2024-04-25T01:16:27.694711+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:28.261961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694834+0000","last_change":"2024-04-25T01:16:21.608564+0000","last_active":"2024-04-25T01:16:27.694834+0000","last_peered":"2024-04-25T01:16:27.694834+0000","last_clean":"2024-04-25T01:16:27.694834+0000","last_became_active":"2024-04-25T01:16:21.608227+0000","last_became_peered":"2024-04-25T01:16:21.608227+0000","last_unstale":"2024-04-25T01:16:27.694834+0000","last_undegraded":"2024-04-25T01:16:27.694834+0000","last_fullsized":"2024-04-25T01:16:27.694834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:38:08.656990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694160+0000","last_change":"2024-04-25T01:16:26.656167+0000","last_active":"2024-04-25T01:16:27.694160+0000","last_peered":"2024-04-25T01:16:27.694160+0000","last_clean":"2024-04-25T01:16:27.694160+0000","last_became_active":"2024-04-25T01:16:26.655643+0000","last_became_peered":"2024-04-25T01:16:26.655643+0000","last_unstale":"2024-04-25T01:16:27.694160+0000","last_undegraded":"2024-04-25T01:16:27.694160+0000","last_fullsized":"2024-04-25T01:16:27.694160+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:17:57.227839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,5],"acting":[3,1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.224130+0000","last_change":"2024-04-25T01:16:22.630130+0000","last_active":"2024-04-25T01:16:28.224130+0000","last_peered":"2024-04-25T01:16:28.224130+0000","last_clean":"2024-04-25T01:16:28.224130+0000","last_became_active":"2024-04-25T01:16:22.629829+0000","last_became_peered":"2024-04-25T01:16:22.629829+0000","last_unstale":"2024-04-25T01:16:28.224130+0000","last_undegraded":"2024-04-25T01:16:28.224130+0000","last_fullsized":"2024-04-25T01:16:28.224130+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:48:42.505862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.086752+0000","last_change":"2024-04-25T01:16:21.615603+0000","last_active":"2024-04-25T01:16:30.086752+0000","last_peered":"2024-04-25T01:16:30.086752+0000","last_clean":"2024-04-25T01:16:30.086752+0000","last_became_active":"2024-04-25T01:16:21.615217+0000","last_became_peered":"2024-04-25T01:16:21.615217+0000","last_unstale":"2024-04-25T01:16:30.086752+0000","last_undegraded":"2024-04-25T01:16:30.086752+0000","last_fullsized":"2024-04-25T01:16:30.086752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:10:00.414303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.773845+0000","last_change":"2024-04-25T01:16:26.677623+0000","last_active":"2024-04-25T01:16:29.773845+0000","last_peered":"2024-04-25T01:16:29.773845+0000","last_clean":"2024-04-25T01:16:29.773845+0000","last_became_active":"2024-04-25T01:16:26.677212+0000","last_became_peered":"2024-04-25T01:16:26.677212+0000","last_unstale":"2024-04-25T01:16:29.773845+0000","last_undegraded":"2024-04-25T01:16:29.773845+0000","last_fullsized":"2024-04-25T01:16:29.773845+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:58.601828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6,1],"acting":[0,4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694468+0000","last_change":"2024-04-25T01:16:22.617565+0000","last_active":"2024-04-25T01:16:27.694468+0000","last_peered":"2024-04-25T01:16:27.694468+0000","last_clean":"2024-04-25T01:16:27.694468+0000","last_became_active":"2024-04-25T01:16:22.617247+0000","last_became_peered":"2024-04-25T01:16:22.617247+0000","last_unstale":"2024-04-25T01:16:27.694468+0000","last_undegraded":"2024-04-25T01:16:27.694468+0000","last_fullsized":"2024-04-25T01:16:27.694468+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:00:17.366512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695722+0000","last_change":"2024-04-25T01:16:21.620221+0000","last_active":"2024-04-25T01:16:27.695722+0000","last_peered":"2024-04-25T01:16:27.695722+0000","last_clean":"2024-04-25T01:16:27.695722+0000","last_became_active":"2024-04-25T01:16:21.619923+0000","last_became_peered":"2024-04-25T01:16:21.619923+0000","last_unstale":"2024-04-25T01:16:27.695722+0000","last_undegraded":"2024-04-25T01:16:27.695722+0000","last_fullsized":"2024-04-25T01:16:27.695722+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:06.940097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.225195+0000","last_change":"2024-04-25T01:16:26.674824+0000","last_active":"2024-04-25T01:16:28.225195+0000","last_peered":"2024-04-25T01:16:28.225195+0000","last_clean":"2024-04-25T01:16:28.225195+0000","last_became_active":"2024-04-25T01:16:26.672608+0000","last_became_peered":"2024-04-25T01:16:26.672608+0000","last_unstale":"2024-04-25T01:16:28.225195+0000","last_undegraded":"2024-04-25T01:16:28.225195+0000","last_fullsized":"2024-04-25T01:16:28.225195+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:46:37.320887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2,3],"acting":[7,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.693656+0000","last_change":"2024-04-25T01:16:22.624853+0000","last_active":"2024-04-25T01:16:27.693656+0000","last_peered":"2024-04-25T01:16:27.693656+0000","last_clean":"2024-04-25T01:16:27.693656+0000","last_became_active":"2024-04-25T01:16:22.624275+0000","last_became_peered":"2024-04-25T01:16:22.624275+0000","last_unstale":"2024-04-25T01:16:27.693656+0000","last_undegraded":"2024-04-25T01:16:27.693656+0000","last_fullsized":"2024-04-25T01:16:27.693656+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:09.176572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.772092+0000","last_change":"2024-04-25T01:16:21.616953+0000","last_active":"2024-04-25T01:16:29.772092+0000","last_peered":"2024-04-25T01:16:29.772092+0000","last_clean":"2024-04-25T01:16:29.772092+0000","last_became_active":"2024-04-25T01:16:21.615656+0000","last_became_peered":"2024-04-25T01:16:21.615656+0000","last_unstale":"2024-04-25T01:16:29.772092+0000","last_undegraded":"2024-04-25T01:16:29.772092+0000","last_fullsized":"2024-04-25T01:16:29.772092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:48:41.491267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.876501+0000","last_change":"2024-04-25T01:16:26.661383+0000","last_active":"2024-04-25T01:16:27.876501+0000","last_peered":"2024-04-25T01:16:27.876501+0000","last_clean":"2024-04-25T01:16:27.876501+0000","last_became_active":"2024-04-25T01:16:26.658181+0000","last_became_peered":"2024-04-25T01:16:26.658181+0000","last_unstale":"2024-04-25T01:16:27.876501+0000","last_undegraded":"2024-04-25T01:16:27.876501+0000","last_fullsized":"2024-04-25T01:16:27.876501+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:23:19.810360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,7],"acting":[2,1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.697025+0000","last_change":"2024-04-25T01:16:22.618495+0000","last_active":"2024-04-25T01:16:27.697025+0000","last_peered":"2024-04-25T01:16:27.697025+0000","last_clean":"2024-04-25T01:16:27.697025+0000","last_became_active":"2024-04-25T01:16:22.618155+0000","last_became_peered":"2024-04-25T01:16:22.618155+0000","last_unstale":"2024-04-25T01:16:27.697025+0000","last_undegraded":"2024-04-25T01:16:27.697025+0000","last_fullsized":"2024-04-25T01:16:27.697025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:53.625025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695692+0000","last_change":"2024-04-25T01:16:21.618714+0000","last_active":"2024-04-25T01:16:27.695692+0000","last_peered":"2024-04-25T01:16:27.695692+0000","last_clean":"2024-04-25T01:16:27.695692+0000","last_became_active":"2024-04-25T01:16:21.618439+0000","last_became_peered":"2024-04-25T01:16:21.618439+0000","last_unstale":"2024-04-25T01:16:27.695692+0000","last_undegraded":"2024-04-25T01:16:27.695692+0000","last_fullsized":"2024-04-25T01:16:27.695692+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:17:15.320611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.876136+0000","last_change":"2024-04-25T01:16:26.649207+0000","last_active":"2024-04-25T01:16:27.876136+0000","last_peered":"2024-04-25T01:16:27.876136+0000","last_clean":"2024-04-25T01:16:27.876136+0000","last_became_active":"2024-04-25T01:16:26.648837+0000","last_became_peered":"2024-04-25T01:16:26.648837+0000","last_unstale":"2024-04-25T01:16:27.876136+0000","last_undegraded":"2024-04-25T01:16:27.876136+0000","last_fullsized":"2024-04-25T01:16:27.876136+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:13.227403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,1],"acting":[2,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.698475+0000","last_change":"2024-04-25T01:16:22.627735+0000","last_active":"2024-04-25T01:16:27.698475+0000","last_peered":"2024-04-25T01:16:27.698475+0000","last_clean":"2024-04-25T01:16:27.698475+0000","last_became_active":"2024-04-25T01:16:22.627430+0000","last_became_peered":"2024-04-25T01:16:22.627430+0000","last_unstale":"2024-04-25T01:16:27.698475+0000","last_undegraded":"2024-04-25T01:16:27.698475+0000","last_fullsized":"2024-04-25T01:16:27.698475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:06:31.257662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.697307+0000","last_change":"2024-04-25T01:16:21.620348+0000","last_active":"2024-04-25T01:16:27.697307+0000","last_peered":"2024-04-25T01:16:27.697307+0000","last_clean":"2024-04-25T01:16:27.697307+0000","last_became_active":"2024-04-25T01:16:21.620049+0000","last_became_peered":"2024-04-25T01:16:21.620049+0000","last_unstale":"2024-04-25T01:16:27.697307+0000","last_undegraded":"2024-04-25T01:16:27.697307+0000","last_fullsized":"2024-04-25T01:16:27.697307+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:17:59.668250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694701+0000","last_change":"2024-04-25T01:16:26.672455+0000","last_active":"2024-04-25T01:16:27.694701+0000","last_peered":"2024-04-25T01:16:27.694701+0000","last_clean":"2024-04-25T01:16:27.694701+0000","last_became_active":"2024-04-25T01:16:26.671952+0000","last_became_peered":"2024-04-25T01:16:26.671952+0000","last_unstale":"2024-04-25T01:16:27.694701+0000","last_undegraded":"2024-04-25T01:16:27.694701+0000","last_fullsized":"2024-04-25T01:16:27.694701+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:13.227403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6,3],"acting":[4,1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.088865+0000","last_change":"2024-04-25T01:16:22.628050+0000","last_active":"2024-04-25T01:16:30.088865+0000","last_peered":"2024-04-25T01:16:30.088865+0000","last_clean":"2024-04-25T01:16:30.088865+0000","last_became_active":"2024-04-25T01:16:22.626875+0000","last_became_peered":"2024-04-25T01:16:22.626875+0000","last_unstale":"2024-04-25T01:16:30.088865+0000","last_undegraded":"2024-04-25T01:16:30.088865+0000","last_fullsized":"2024-04-25T01:16:30.088865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:24:16.000792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695500+0000","last_change":"2024-04-25T01:16:21.621352+0000","last_active":"2024-04-25T01:16:27.695500+0000","last_peered":"2024-04-25T01:16:27.695500+0000","last_clean":"2024-04-25T01:16:27.695500+0000","last_became_active":"2024-04-25T01:16:21.620116+0000","last_became_peered":"2024-04-25T01:16:21.620116+0000","last_unstale":"2024-04-25T01:16:27.695500+0000","last_undegraded":"2024-04-25T01:16:27.695500+0000","last_fullsized":"2024-04-25T01:16:27.695500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:07:31.151855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.087941+0000","last_change":"2024-04-25T01:16:26.663371+0000","last_active":"2024-04-25T01:16:30.087941+0000","last_peered":"2024-04-25T01:16:30.087941+0000","last_clean":"2024-04-25T01:16:30.087941+0000","last_became_active":"2024-04-25T01:16:26.662972+0000","last_became_peered":"2024-04-25T01:16:26.662972+0000","last_unstale":"2024-04-25T01:16:30.087941+0000","last_undegraded":"2024-04-25T01:16:30.087941+0000","last_fullsized":"2024-04-25T01:16:30.087941+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:18.289163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2,7],"acting":[6,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.875472+0000","last_change":"2024-04-25T01:16:22.628735+0000","last_active":"2024-04-25T01:16:27.875472+0000","last_peered":"2024-04-25T01:16:27.875472+0000","last_clean":"2024-04-25T01:16:27.875472+0000","last_became_active":"2024-04-25T01:16:22.628429+0000","last_became_peered":"2024-04-25T01:16:22.628429+0000","last_unstale":"2024-04-25T01:16:27.875472+0000","last_undegraded":"2024-04-25T01:16:27.875472+0000","last_fullsized":"2024-04-25T01:16:27.875472+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:20.635412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.697942+0000","last_change":"2024-04-25T01:16:21.622977+0000","last_active":"2024-04-25T01:16:27.697942+0000","last_peered":"2024-04-25T01:16:27.697942+0000","last_clean":"2024-04-25T01:16:27.697942+0000","last_became_active":"2024-04-25T01:16:21.622298+0000","last_became_peered":"2024-04-25T01:16:21.622298+0000","last_unstale":"2024-04-25T01:16:27.697942+0000","last_undegraded":"2024-04-25T01:16:27.697942+0000","last_fullsized":"2024-04-25T01:16:27.697942+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:34.732380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.087427+0000","last_change":"2024-04-25T01:16:26.664207+0000","last_active":"2024-04-25T01:16:30.087427+0000","last_peered":"2024-04-25T01:16:30.087427+0000","last_clean":"2024-04-25T01:16:30.087427+0000","last_became_active":"2024-04-25T01:16:26.663749+0000","last_became_peered":"2024-04-25T01:16:26.663749+0000","last_unstale":"2024-04-25T01:16:30.087427+0000","last_undegraded":"2024-04-25T01:16:30.087427+0000","last_fullsized":"2024-04-25T01:16:30.087427+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:05.467463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5,7],"acting":[6,3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.773873+0000","last_change":"2024-04-25T01:16:22.635293+0000","last_active":"2024-04-25T01:16:29.773873+0000","last_peered":"2024-04-25T01:16:29.773873+0000","last_clean":"2024-04-25T01:16:29.773873+0000","last_became_active":"2024-04-25T01:16:22.635026+0000","last_became_peered":"2024-04-25T01:16:22.635026+0000","last_unstale":"2024-04-25T01:16:29.773873+0000","last_undegraded":"2024-04-25T01:16:29.773873+0000","last_fullsized":"2024-04-25T01:16:29.773873+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:46:16.608442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696109+0000","last_change":"2024-04-25T01:16:21.623621+0000","last_active":"2024-04-25T01:16:27.696109+0000","last_peered":"2024-04-25T01:16:27.696109+0000","last_clean":"2024-04-25T01:16:27.696109+0000","last_became_active":"2024-04-25T01:16:21.623351+0000","last_became_peered":"2024-04-25T01:16:21.623351+0000","last_unstale":"2024-04-25T01:16:27.696109+0000","last_undegraded":"2024-04-25T01:16:27.696109+0000","last_fullsized":"2024-04-25T01:16:27.696109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:49:04.468754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.088614+0000","last_change":"2024-04-25T01:16:26.656694+0000","last_active":"2024-04-25T01:16:30.088614+0000","last_peered":"2024-04-25T01:16:30.088614+0000","last_clean":"2024-04-25T01:16:30.088614+0000","last_became_active":"2024-04-25T01:16:26.656201+0000","last_became_peered":"2024-04-25T01:16:26.656201+0000","last_unstale":"2024-04-25T01:16:30.088614+0000","last_undegraded":"2024-04-25T01:16:30.088614+0000","last_fullsized":"2024-04-25T01:16:30.088614+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:39:18.953803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0,3],"acting":[6,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694802+0000","last_change":"2024-04-25T01:16:22.615076+0000","last_active":"2024-04-25T01:16:27.694802+0000","last_peered":"2024-04-25T01:16:27.694802+0000","last_clean":"2024-04-25T01:16:27.694802+0000","last_became_active":"2024-04-25T01:16:22.613139+0000","last_became_peered":"2024-04-25T01:16:22.613139+0000","last_unstale":"2024-04-25T01:16:27.694802+0000","last_undegraded":"2024-04-25T01:16:27.694802+0000","last_fullsized":"2024-04-25T01:16:27.694802+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:22:36.726844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.224133+0000","last_change":"2024-04-25T01:16:21.614492+0000","last_active":"2024-04-25T01:16:28.224133+0000","last_peered":"2024-04-25T01:16:28.224133+0000","last_clean":"2024-04-25T01:16:28.224133+0000","last_became_active":"2024-04-25T01:16:21.613490+0000","last_became_peered":"2024-04-25T01:16:21.613490+0000","last_unstale":"2024-04-25T01:16:28.224133+0000","last_undegraded":"2024-04-25T01:16:28.224133+0000","last_fullsized":"2024-04-25T01:16:28.224133+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:41:30.225989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.088458+0000","last_change":"2024-04-25T01:16:26.656061+0000","last_active":"2024-04-25T01:16:30.088458+0000","last_peered":"2024-04-25T01:16:30.088458+0000","last_clean":"2024-04-25T01:16:30.088458+0000","last_became_active":"2024-04-25T01:16:26.655189+0000","last_became_peered":"2024-04-25T01:16:26.655189+0000","last_unstale":"2024-04-25T01:16:30.088458+0000","last_undegraded":"2024-04-25T01:16:30.088458+0000","last_fullsized":"2024-04-25T01:16:30.088458+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:37:04.233365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,7],"acting":[6,0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.087313+0000","last_change":"2024-04-25T01:16:22.634160+0000","last_active":"2024-04-25T01:16:30.087313+0000","last_peered":"2024-04-25T01:16:30.087313+0000","last_clean":"2024-04-25T01:16:30.087313+0000","last_became_active":"2024-04-25T01:16:22.633856+0000","last_became_peered":"2024-04-25T01:16:22.633856+0000","last_unstale":"2024-04-25T01:16:30.087313+0000","last_undegraded":"2024-04-25T01:16:30.087313+0000","last_fullsized":"2024-04-25T01:16:30.087313+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:46:05.184032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695122+0000","last_change":"2024-04-25T01:16:21.617465+0000","last_active":"2024-04-25T01:16:27.695122+0000","last_peered":"2024-04-25T01:16:27.695122+0000","last_clean":"2024-04-25T01:16:27.695122+0000","last_became_active":"2024-04-25T01:16:21.617201+0000","last_became_peered":"2024-04-25T01:16:21.617201+0000","last_unstale":"2024-04-25T01:16:27.695122+0000","last_undegraded":"2024-04-25T01:16:27.695122+0000","last_fullsized":"2024-04-25T01:16:27.695122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:48:41.491267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699450+0000","last_change":"2024-04-25T01:16:26.666150+0000","last_active":"2024-04-25T01:16:27.699450+0000","last_peered":"2024-04-25T01:16:27.699450+0000","last_clean":"2024-04-25T01:16:27.699450+0000","last_became_active":"2024-04-25T01:16:26.665620+0000","last_became_peered":"2024-04-25T01:16:26.665620+0000","last_unstale":"2024-04-25T01:16:27.699450+0000","last_undegraded":"2024-04-25T01:16:27.699450+0000","last_fullsized":"2024-04-25T01:16:27.699450+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:51:23.702657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6,1],"acting":[5,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695397+0000","last_change":"2024-04-25T01:16:22.637692+0000","last_active":"2024-04-25T01:16:27.695397+0000","last_peered":"2024-04-25T01:16:27.695397+0000","last_clean":"2024-04-25T01:16:27.695397+0000","last_became_active":"2024-04-25T01:16:22.635225+0000","last_became_peered":"2024-04-25T01:16:22.635225+0000","last_unstale":"2024-04-25T01:16:27.695397+0000","last_undegraded":"2024-04-25T01:16:27.695397+0000","last_fullsized":"2024-04-25T01:16:27.695397+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:11.514022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695136+0000","last_change":"2024-04-25T01:16:21.608226+0000","last_active":"2024-04-25T01:16:27.695136+0000","last_peered":"2024-04-25T01:16:27.695136+0000","last_clean":"2024-04-25T01:16:27.695136+0000","last_became_active":"2024-04-25T01:16:21.607851+0000","last_became_peered":"2024-04-25T01:16:21.607851+0000","last_unstale":"2024-04-25T01:16:27.695136+0000","last_undegraded":"2024-04-25T01:16:27.695136+0000","last_fullsized":"2024-04-25T01:16:27.695136+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:51.521474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":178,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.970532+0000","last_change":"2024-04-25T01:16:17.466569+0000","last_active":"2024-04-25T01:16:27.970532+0000","last_peered":"2024-04-25T01:16:27.970532+0000","last_clean":"2024-04-25T01:16:27.970532+0000","last_became_active":"2024-04-25T01:16:17.465687+0000","last_became_peered":"2024-04-25T01:16:17.465687+0000","last_unstale":"2024-04-25T01:16:27.970532+0000","last_undegraded":"2024-04-25T01:16:27.970532+0000","last_fullsized":"2024-04-25T01:16:27.970532+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:16.440559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:16.440559+0000","last_clean_scrub_stamp":"2024-04-25T01:16:16.440559+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:18:17.493555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695469+0000","last_change":"2024-04-25T01:16:26.646040+0000","last_active":"2024-04-25T01:16:27.695469+0000","last_peered":"2024-04-25T01:16:27.695469+0000","last_clean":"2024-04-25T01:16:27.695469+0000","last_became_active":"2024-04-25T01:16:26.645500+0000","last_became_peered":"2024-04-25T01:16:26.645500+0000","last_unstale":"2024-04-25T01:16:27.695469+0000","last_undegraded":"2024-04-25T01:16:27.695469+0000","last_fullsized":"2024-04-25T01:16:27.695469+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:01:53.032237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2,0],"acting":[3,1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694653+0000","last_change":"2024-04-25T01:16:22.631511+0000","last_active":"2024-04-25T01:16:27.694653+0000","last_peered":"2024-04-25T01:16:27.694653+0000","last_clean":"2024-04-25T01:16:27.694653+0000","last_became_active":"2024-04-25T01:16:22.631169+0000","last_became_peered":"2024-04-25T01:16:22.631169+0000","last_unstale":"2024-04-25T01:16:27.694653+0000","last_undegraded":"2024-04-25T01:16:27.694653+0000","last_fullsized":"2024-04-25T01:16:27.694653+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:54.550997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.089006+0000","last_change":"2024-04-25T01:16:21.621474+0000","last_active":"2024-04-25T01:16:30.089006+0000","last_peered":"2024-04-25T01:16:30.089006+0000","last_clean":"2024-04-25T01:16:30.089006+0000","last_became_active":"2024-04-25T01:16:21.621139+0000","last_became_peered":"2024-04-25T01:16:21.621139+0000","last_unstale":"2024-04-25T01:16:30.089006+0000","last_undegraded":"2024-04-25T01:16:30.089006+0000","last_fullsized":"2024-04-25T01:16:30.089006+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:35:01.136892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.875567+0000","last_change":"2024-04-25T01:16:26.669916+0000","last_active":"2024-04-25T01:16:27.875567+0000","last_peered":"2024-04-25T01:16:27.875567+0000","last_clean":"2024-04-25T01:16:27.875567+0000","last_became_active":"2024-04-25T01:16:26.668054+0000","last_became_peered":"2024-04-25T01:16:26.668054+0000","last_unstale":"2024-04-25T01:16:27.875567+0000","last_undegraded":"2024-04-25T01:16:27.875567+0000","last_fullsized":"2024-04-25T01:16:27.875567+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:58:10.131890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7,4],"acting":[2,3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774586+0000","last_change":"2024-04-25T01:16:22.617183+0000","last_active":"2024-04-25T01:16:29.774586+0000","last_peered":"2024-04-25T01:16:29.774586+0000","last_clean":"2024-04-25T01:16:29.774586+0000","last_became_active":"2024-04-25T01:16:22.615748+0000","last_became_peered":"2024-04-25T01:16:22.615748+0000","last_unstale":"2024-04-25T01:16:29.774586+0000","last_undegraded":"2024-04-25T01:16:29.774586+0000","last_fullsized":"2024-04-25T01:16:29.774586+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:41:00.459225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699060+0000","last_change":"2024-04-25T01:16:21.617685+0000","last_active":"2024-04-25T01:16:27.699060+0000","last_peered":"2024-04-25T01:16:27.699060+0000","last_clean":"2024-04-25T01:16:27.699060+0000","last_became_active":"2024-04-25T01:16:21.617209+0000","last_became_peered":"2024-04-25T01:16:21.617209+0000","last_unstale":"2024-04-25T01:16:27.699060+0000","last_undegraded":"2024-04-25T01:16:27.699060+0000","last_fullsized":"2024-04-25T01:16:27.699060+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:34:39.714953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696564+0000","last_change":"2024-04-25T01:16:26.661702+0000","last_active":"2024-04-25T01:16:27.696564+0000","last_peered":"2024-04-25T01:16:27.696564+0000","last_clean":"2024-04-25T01:16:27.696564+0000","last_became_active":"2024-04-25T01:16:26.660914+0000","last_became_peered":"2024-04-25T01:16:26.660914+0000","last_unstale":"2024-04-25T01:16:27.696564+0000","last_undegraded":"2024-04-25T01:16:27.696564+0000","last_fullsized":"2024-04-25T01:16:27.696564+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:04:40.070035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1,3],"acting":[5,4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774509+0000","last_change":"2024-04-25T01:16:22.626033+0000","last_active":"2024-04-25T01:16:29.774509+0000","last_peered":"2024-04-25T01:16:29.774509+0000","last_clean":"2024-04-25T01:16:29.774509+0000","last_became_active":"2024-04-25T01:16:22.625646+0000","last_became_peered":"2024-04-25T01:16:22.625646+0000","last_unstale":"2024-04-25T01:16:29.774509+0000","last_undegraded":"2024-04-25T01:16:29.774509+0000","last_fullsized":"2024-04-25T01:16:29.774509+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:28.261961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.697212+0000","last_change":"2024-04-25T01:16:21.614628+0000","last_active":"2024-04-25T01:16:27.697212+0000","last_peered":"2024-04-25T01:16:27.697212+0000","last_clean":"2024-04-25T01:16:27.697212+0000","last_became_active":"2024-04-25T01:16:21.614110+0000","last_became_peered":"2024-04-25T01:16:21.614110+0000","last_unstale":"2024-04-25T01:16:27.697212+0000","last_undegraded":"2024-04-25T01:16:27.697212+0000","last_fullsized":"2024-04-25T01:16:27.697212+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:44.675184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.693414+0000","last_change":"2024-04-25T01:16:26.652814+0000","last_active":"2024-04-25T01:16:27.693414+0000","last_peered":"2024-04-25T01:16:27.693414+0000","last_clean":"2024-04-25T01:16:27.693414+0000","last_became_active":"2024-04-25T01:16:26.652030+0000","last_became_peered":"2024-04-25T01:16:26.652030+0000","last_unstale":"2024-04-25T01:16:27.693414+0000","last_undegraded":"2024-04-25T01:16:27.693414+0000","last_fullsized":"2024-04-25T01:16:27.693414+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:22.783625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.876518+0000","last_change":"2024-04-25T01:16:22.617280+0000","last_active":"2024-04-25T01:16:27.876518+0000","last_peered":"2024-04-25T01:16:27.876518+0000","last_clean":"2024-04-25T01:16:27.876518+0000","last_became_active":"2024-04-25T01:16:22.616985+0000","last_became_peered":"2024-04-25T01:16:22.616985+0000","last_unstale":"2024-04-25T01:16:27.876518+0000","last_undegraded":"2024-04-25T01:16:27.876518+0000","last_fullsized":"2024-04-25T01:16:27.876518+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:17:16.335206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.772458+0000","last_change":"2024-04-25T01:16:21.612374+0000","last_active":"2024-04-25T01:16:29.772458+0000","last_peered":"2024-04-25T01:16:29.772458+0000","last_clean":"2024-04-25T01:16:29.772458+0000","last_became_active":"2024-04-25T01:16:21.610765+0000","last_became_peered":"2024-04-25T01:16:21.610765+0000","last_unstale":"2024-04-25T01:16:29.772458+0000","last_undegraded":"2024-04-25T01:16:29.772458+0000","last_fullsized":"2024-04-25T01:16:29.772458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:17:59.668250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.971020+0000","last_change":"2024-04-25T01:16:26.655048+0000","last_active":"2024-04-25T01:16:27.971020+0000","last_peered":"2024-04-25T01:16:27.971020+0000","last_clean":"2024-04-25T01:16:27.971020+0000","last_became_active":"2024-04-25T01:16:26.654534+0000","last_became_peered":"2024-04-25T01:16:26.654534+0000","last_unstale":"2024-04-25T01:16:27.971020+0000","last_undegraded":"2024-04-25T01:16:27.971020+0000","last_fullsized":"2024-04-25T01:16:27.971020+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:17:20.386037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,2],"acting":[1,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.693580+0000","last_change":"2024-04-25T01:16:22.621853+0000","last_active":"2024-04-25T01:16:27.693580+0000","last_peered":"2024-04-25T01:16:27.693580+0000","last_clean":"2024-04-25T01:16:27.693580+0000","last_became_active":"2024-04-25T01:16:22.621537+0000","last_became_peered":"2024-04-25T01:16:22.621537+0000","last_unstale":"2024-04-25T01:16:27.693580+0000","last_undegraded":"2024-04-25T01:16:27.693580+0000","last_fullsized":"2024-04-25T01:16:27.693580+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:01.512241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694812+0000","last_change":"2024-04-25T01:16:21.607981+0000","last_active":"2024-04-25T01:16:27.694812+0000","last_peered":"2024-04-25T01:16:27.694812+0000","last_clean":"2024-04-25T01:16:27.694812+0000","last_became_active":"2024-04-25T01:16:21.607619+0000","last_became_peered":"2024-04-25T01:16:21.607619+0000","last_unstale":"2024-04-25T01:16:27.694812+0000","last_undegraded":"2024-04-25T01:16:27.694812+0000","last_fullsized":"2024-04-25T01:16:27.694812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:48:41.491267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699998+0000","last_change":"2024-04-25T01:16:26.664351+0000","last_active":"2024-04-25T01:16:27.699998+0000","last_peered":"2024-04-25T01:16:27.699998+0000","last_clean":"2024-04-25T01:16:27.699998+0000","last_became_active":"2024-04-25T01:16:26.663469+0000","last_became_peered":"2024-04-25T01:16:26.663469+0000","last_unstale":"2024-04-25T01:16:27.699998+0000","last_undegraded":"2024-04-25T01:16:27.699998+0000","last_fullsized":"2024-04-25T01:16:27.699998+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:10.152413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1,6],"acting":[5,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.970542+0000","last_change":"2024-04-25T01:16:22.609467+0000","last_active":"2024-04-25T01:16:27.970542+0000","last_peered":"2024-04-25T01:16:27.970542+0000","last_clean":"2024-04-25T01:16:27.970542+0000","last_became_active":"2024-04-25T01:16:22.609021+0000","last_became_peered":"2024-04-25T01:16:22.609021+0000","last_unstale":"2024-04-25T01:16:27.970542+0000","last_undegraded":"2024-04-25T01:16:27.970542+0000","last_fullsized":"2024-04-25T01:16:27.970542+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:45.689780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.772174+0000","last_change":"2024-04-25T01:16:21.618665+0000","last_active":"2024-04-25T01:16:29.772174+0000","last_peered":"2024-04-25T01:16:29.772174+0000","last_clean":"2024-04-25T01:16:29.772174+0000","last_became_active":"2024-04-25T01:16:21.618387+0000","last_became_peered":"2024-04-25T01:16:21.618387+0000","last_unstale":"2024-04-25T01:16:29.772174+0000","last_undegraded":"2024-04-25T01:16:29.772174+0000","last_fullsized":"2024-04-25T01:16:29.772174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:50.882927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699704+0000","last_change":"2024-04-25T01:16:26.676113+0000","last_active":"2024-04-25T01:16:27.699704+0000","last_peered":"2024-04-25T01:16:27.699704+0000","last_clean":"2024-04-25T01:16:27.699704+0000","last_became_active":"2024-04-25T01:16:26.675447+0000","last_became_peered":"2024-04-25T01:16:26.675447+0000","last_unstale":"2024-04-25T01:16:27.699704+0000","last_undegraded":"2024-04-25T01:16:27.699704+0000","last_fullsized":"2024-04-25T01:16:27.699704+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:07:54.641757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6,4],"acting":[5,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694284+0000","last_change":"2024-04-25T01:16:22.633079+0000","last_active":"2024-04-25T01:16:27.694284+0000","last_peered":"2024-04-25T01:16:27.694284+0000","last_clean":"2024-04-25T01:16:27.694284+0000","last_became_active":"2024-04-25T01:16:22.632813+0000","last_became_peered":"2024-04-25T01:16:22.632813+0000","last_unstale":"2024-04-25T01:16:27.694284+0000","last_undegraded":"2024-04-25T01:16:27.694284+0000","last_fullsized":"2024-04-25T01:16:27.694284+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:45:08.158420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695843+0000","last_change":"2024-04-25T01:16:21.610362+0000","last_active":"2024-04-25T01:16:27.695843+0000","last_peered":"2024-04-25T01:16:27.695843+0000","last_clean":"2024-04-25T01:16:27.695843+0000","last_became_active":"2024-04-25T01:16:21.608581+0000","last_became_peered":"2024-04-25T01:16:21.608581+0000","last_unstale":"2024-04-25T01:16:27.695843+0000","last_undegraded":"2024-04-25T01:16:27.695843+0000","last_fullsized":"2024-04-25T01:16:27.695843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:17:59.668250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.088195+0000","last_change":"2024-04-25T01:16:26.678631+0000","last_active":"2024-04-25T01:16:30.088195+0000","last_peered":"2024-04-25T01:16:30.088195+0000","last_clean":"2024-04-25T01:16:30.088195+0000","last_became_active":"2024-04-25T01:16:26.676876+0000","last_became_peered":"2024-04-25T01:16:26.676876+0000","last_unstale":"2024-04-25T01:16:30.088195+0000","last_undegraded":"2024-04-25T01:16:30.088195+0000","last_fullsized":"2024-04-25T01:16:30.088195+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:03:57.927312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0,7],"acting":[6,4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.087763+0000","last_change":"2024-04-25T01:16:22.627367+0000","last_active":"2024-04-25T01:16:30.087763+0000","last_peered":"2024-04-25T01:16:30.087763+0000","last_clean":"2024-04-25T01:16:30.087763+0000","last_became_active":"2024-04-25T01:16:22.626338+0000","last_became_peered":"2024-04-25T01:16:22.626338+0000","last_unstale":"2024-04-25T01:16:30.087763+0000","last_undegraded":"2024-04-25T01:16:30.087763+0000","last_fullsized":"2024-04-25T01:16:30.087763+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:50:15.794188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.224430+0000","last_change":"2024-04-25T01:16:21.613871+0000","last_active":"2024-04-25T01:16:28.224430+0000","last_peered":"2024-04-25T01:16:28.224430+0000","last_clean":"2024-04-25T01:16:28.224430+0000","last_became_active":"2024-04-25T01:16:21.613496+0000","last_became_peered":"2024-04-25T01:16:21.613496+0000","last_unstale":"2024-04-25T01:16:28.224430+0000","last_undegraded":"2024-04-25T01:16:28.224430+0000","last_fullsized":"2024-04-25T01:16:28.224430+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:37:47.297259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694895+0000","last_change":"2024-04-25T01:16:26.653921+0000","last_active":"2024-04-25T01:16:27.694895+0000","last_peered":"2024-04-25T01:16:27.694895+0000","last_clean":"2024-04-25T01:16:27.694895+0000","last_became_active":"2024-04-25T01:16:26.653438+0000","last_became_peered":"2024-04-25T01:16:26.653438+0000","last_unstale":"2024-04-25T01:16:27.694895+0000","last_undegraded":"2024-04-25T01:16:27.694895+0000","last_fullsized":"2024-04-25T01:16:27.694895+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:58:40.113666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5,6],"acting":[3,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774620+0000","last_change":"2024-04-25T01:16:22.631703+0000","last_active":"2024-04-25T01:16:29.774620+0000","last_peered":"2024-04-25T01:16:29.774620+0000","last_clean":"2024-04-25T01:16:29.774620+0000","last_became_active":"2024-04-25T01:16:22.631211+0000","last_became_peered":"2024-04-25T01:16:22.631211+0000","last_unstale":"2024-04-25T01:16:29.774620+0000","last_undegraded":"2024-04-25T01:16:29.774620+0000","last_fullsized":"2024-04-25T01:16:29.774620+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:01.512241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"26'10","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:41.501751+0000","last_change":"2024-04-25T01:16:21.614845+0000","last_active":"2024-04-25T01:16:41.501751+0000","last_peered":"2024-04-25T01:16:41.501751+0000","last_clean":"2024-04-25T01:16:41.501751+0000","last_became_active":"2024-04-25T01:16:21.614378+0000","last_became_peered":"2024-04-25T01:16:21.614378+0000","last_unstale":"2024-04-25T01:16:41.501751+0000","last_undegraded":"2024-04-25T01:16:41.501751+0000","last_fullsized":"2024-04-25T01:16:41.501751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:41:30.225989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695086+0000","last_change":"2024-04-25T01:16:26.676618+0000","last_active":"2024-04-25T01:16:27.695086+0000","last_peered":"2024-04-25T01:16:27.695086+0000","last_clean":"2024-04-25T01:16:27.695086+0000","last_became_active":"2024-04-25T01:16:26.667226+0000","last_became_peered":"2024-04-25T01:16:26.667226+0000","last_unstale":"2024-04-25T01:16:27.695086+0000","last_undegraded":"2024-04-25T01:16:27.695086+0000","last_fullsized":"2024-04-25T01:16:27.695086+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:23:19.810360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5,0],"acting":[4,3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.693968+0000","last_change":"2024-04-25T01:16:22.624757+0000","last_active":"2024-04-25T01:16:27.693968+0000","last_peered":"2024-04-25T01:16:27.693968+0000","last_clean":"2024-04-25T01:16:27.693968+0000","last_became_active":"2024-04-25T01:16:22.619992+0000","last_became_peered":"2024-04-25T01:16:22.619992+0000","last_unstale":"2024-04-25T01:16:27.693968+0000","last_undegraded":"2024-04-25T01:16:27.693968+0000","last_fullsized":"2024-04-25T01:16:27.693968+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:26:11.534238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.225000+0000","last_change":"2024-04-25T01:16:21.615700+0000","last_active":"2024-04-25T01:16:28.225000+0000","last_peered":"2024-04-25T01:16:28.225000+0000","last_clean":"2024-04-25T01:16:28.225000+0000","last_became_active":"2024-04-25T01:16:21.615207+0000","last_became_peered":"2024-04-25T01:16:21.615207+0000","last_unstale":"2024-04-25T01:16:28.225000+0000","last_undegraded":"2024-04-25T01:16:28.225000+0000","last_fullsized":"2024-04-25T01:16:28.225000+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:13.115669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.875349+0000","last_change":"2024-04-25T01:16:26.666133+0000","last_active":"2024-04-25T01:16:27.875349+0000","last_peered":"2024-04-25T01:16:27.875349+0000","last_clean":"2024-04-25T01:16:27.875349+0000","last_became_active":"2024-04-25T01:16:26.665359+0000","last_became_peered":"2024-04-25T01:16:26.665359+0000","last_unstale":"2024-04-25T01:16:27.875349+0000","last_undegraded":"2024-04-25T01:16:27.875349+0000","last_fullsized":"2024-04-25T01:16:27.875349+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:46:20.659273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1,7],"acting":[2,3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699852+0000","last_change":"2024-04-25T01:16:22.626495+0000","last_active":"2024-04-25T01:16:27.699852+0000","last_peered":"2024-04-25T01:16:27.699852+0000","last_clean":"2024-04-25T01:16:27.699852+0000","last_became_active":"2024-04-25T01:16:22.625913+0000","last_became_peered":"2024-04-25T01:16:22.625913+0000","last_unstale":"2024-04-25T01:16:27.699852+0000","last_undegraded":"2024-04-25T01:16:27.699852+0000","last_fullsized":"2024-04-25T01:16:27.699852+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:36.984987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.772343+0000","last_change":"2024-04-25T01:16:21.611655+0000","last_active":"2024-04-25T01:16:29.772343+0000","last_peered":"2024-04-25T01:16:29.772343+0000","last_clean":"2024-04-25T01:16:29.772343+0000","last_became_active":"2024-04-25T01:16:21.611388+0000","last_became_peered":"2024-04-25T01:16:21.611388+0000","last_unstale":"2024-04-25T01:16:29.772343+0000","last_undegraded":"2024-04-25T01:16:29.772343+0000","last_fullsized":"2024-04-25T01:16:29.772343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:39:20.330039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.223876+0000","last_change":"2024-04-25T01:16:26.668770+0000","last_active":"2024-04-25T01:16:28.223876+0000","last_peered":"2024-04-25T01:16:28.223876+0000","last_clean":"2024-04-25T01:16:28.223876+0000","last_became_active":"2024-04-25T01:16:26.668309+0000","last_became_peered":"2024-04-25T01:16:26.668309+0000","last_unstale":"2024-04-25T01:16:28.223876+0000","last_undegraded":"2024-04-25T01:16:28.223876+0000","last_fullsized":"2024-04-25T01:16:28.223876+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:17:20.386037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3,2],"acting":[7,1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.876399+0000","last_change":"2024-04-25T01:16:22.632717+0000","last_active":"2024-04-25T01:16:27.876399+0000","last_peered":"2024-04-25T01:16:27.876399+0000","last_clean":"2024-04-25T01:16:27.876399+0000","last_became_active":"2024-04-25T01:16:22.630174+0000","last_became_peered":"2024-04-25T01:16:22.630174+0000","last_unstale":"2024-04-25T01:16:27.876399+0000","last_undegraded":"2024-04-25T01:16:27.876399+0000","last_fullsized":"2024-04-25T01:16:27.876399+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:52.536070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.971972+0000","last_change":"2024-04-25T01:16:21.619013+0000","last_active":"2024-04-25T01:16:27.971972+0000","last_peered":"2024-04-25T01:16:27.971972+0000","last_clean":"2024-04-25T01:16:27.971972+0000","last_became_active":"2024-04-25T01:16:21.615903+0000","last_became_peered":"2024-04-25T01:16:21.615903+0000","last_unstale":"2024-04-25T01:16:27.971972+0000","last_undegraded":"2024-04-25T01:16:27.971972+0000","last_fullsized":"2024-04-25T01:16:27.971972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:37:47.297259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695313+0000","last_change":"2024-04-25T01:16:26.673940+0000","last_active":"2024-04-25T01:16:27.695313+0000","last_peered":"2024-04-25T01:16:27.695313+0000","last_clean":"2024-04-25T01:16:27.695313+0000","last_became_active":"2024-04-25T01:16:26.667111+0000","last_became_peered":"2024-04-25T01:16:26.667111+0000","last_unstale":"2024-04-25T01:16:27.695313+0000","last_undegraded":"2024-04-25T01:16:27.695313+0000","last_fullsized":"2024-04-25T01:16:27.695313+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:32.312792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3,7],"acting":[4,0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696886+0000","last_change":"2024-04-25T01:16:26.679002+0000","last_active":"2024-04-25T01:16:27.696886+0000","last_peered":"2024-04-25T01:16:27.696886+0000","last_clean":"2024-04-25T01:16:27.696886+0000","last_became_active":"2024-04-25T01:16:26.678398+0000","last_became_peered":"2024-04-25T01:16:26.678398+0000","last_unstale":"2024-04-25T01:16:27.696886+0000","last_undegraded":"2024-04-25T01:16:27.696886+0000","last_fullsized":"2024-04-25T01:16:27.696886+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:45:12.209250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5,3],"acting":[4,1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695447+0000","last_change":"2024-04-25T01:16:22.619871+0000","last_active":"2024-04-25T01:16:27.695447+0000","last_peered":"2024-04-25T01:16:27.695447+0000","last_clean":"2024-04-25T01:16:27.695447+0000","last_became_active":"2024-04-25T01:16:22.619553+0000","last_became_peered":"2024-04-25T01:16:22.619553+0000","last_unstale":"2024-04-25T01:16:27.695447+0000","last_undegraded":"2024-04-25T01:16:27.695447+0000","last_fullsized":"2024-04-25T01:16:27.695447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:46:33.270057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694865+0000","last_change":"2024-04-25T01:16:21.625865+0000","last_active":"2024-04-25T01:16:27.694865+0000","last_peered":"2024-04-25T01:16:27.694865+0000","last_clean":"2024-04-25T01:16:27.694865+0000","last_became_active":"2024-04-25T01:16:21.625566+0000","last_became_peered":"2024-04-25T01:16:21.625566+0000","last_unstale":"2024-04-25T01:16:27.694865+0000","last_undegraded":"2024-04-25T01:16:27.694865+0000","last_fullsized":"2024-04-25T01:16:27.694865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:44.675184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695078+0000","last_change":"2024-04-25T01:16:26.650705+0000","last_active":"2024-04-25T01:16:27.695078+0000","last_peered":"2024-04-25T01:16:27.695078+0000","last_clean":"2024-04-25T01:16:27.695078+0000","last_became_active":"2024-04-25T01:16:26.650332+0000","last_became_peered":"2024-04-25T01:16:26.650332+0000","last_unstale":"2024-04-25T01:16:27.695078+0000","last_undegraded":"2024-04-25T01:16:27.695078+0000","last_fullsized":"2024-04-25T01:16:27.695078+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:47:34.491567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,1],"acting":[3,2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694898+0000","last_change":"2024-04-25T01:16:22.622105+0000","last_active":"2024-04-25T01:16:27.694898+0000","last_peered":"2024-04-25T01:16:27.694898+0000","last_clean":"2024-04-25T01:16:27.694898+0000","last_became_active":"2024-04-25T01:16:22.621798+0000","last_became_peered":"2024-04-25T01:16:22.621798+0000","last_unstale":"2024-04-25T01:16:27.694898+0000","last_undegraded":"2024-04-25T01:16:27.694898+0000","last_fullsized":"2024-04-25T01:16:27.694898+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:47.769440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.087421+0000","last_change":"2024-04-25T01:16:21.621471+0000","last_active":"2024-04-25T01:16:30.087421+0000","last_peered":"2024-04-25T01:16:30.087421+0000","last_clean":"2024-04-25T01:16:30.087421+0000","last_became_active":"2024-04-25T01:16:21.621210+0000","last_became_peered":"2024-04-25T01:16:21.621210+0000","last_unstale":"2024-04-25T01:16:30.087421+0000","last_undegraded":"2024-04-25T01:16:30.087421+0000","last_fullsized":"2024-04-25T01:16:30.087421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:34.732380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695224+0000","last_change":"2024-04-25T01:16:21.619390+0000","last_active":"2024-04-25T01:16:27.695224+0000","last_peered":"2024-04-25T01:16:27.695224+0000","last_clean":"2024-04-25T01:16:27.695224+0000","last_became_active":"2024-04-25T01:16:21.619110+0000","last_became_peered":"2024-04-25T01:16:21.619110+0000","last_unstale":"2024-04-25T01:16:27.695224+0000","last_undegraded":"2024-04-25T01:16:27.695224+0000","last_fullsized":"2024-04-25T01:16:27.695224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:50.882927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696696+0000","last_change":"2024-04-25T01:16:22.638518+0000","last_active":"2024-04-25T01:16:27.696696+0000","last_peered":"2024-04-25T01:16:27.696696+0000","last_clean":"2024-04-25T01:16:27.696696+0000","last_became_active":"2024-04-25T01:16:22.638204+0000","last_became_peered":"2024-04-25T01:16:22.638204+0000","last_unstale":"2024-04-25T01:16:27.696696+0000","last_undegraded":"2024-04-25T01:16:27.696696+0000","last_fullsized":"2024-04-25T01:16:27.696696+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:06:58.102648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.697794+0000","last_change":"2024-04-25T01:16:26.673686+0000","last_active":"2024-04-25T01:16:27.697794+0000","last_peered":"2024-04-25T01:16:27.697794+0000","last_clean":"2024-04-25T01:16:27.697794+0000","last_became_active":"2024-04-25T01:16:26.673316+0000","last_became_peered":"2024-04-25T01:16:26.673316+0000","last_unstale":"2024-04-25T01:16:27.697794+0000","last_undegraded":"2024-04-25T01:16:27.697794+0000","last_fullsized":"2024-04-25T01:16:27.697794+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:26:15.585069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,7],"acting":[4,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696890+0000","last_change":"2024-04-25T01:16:21.623660+0000","last_active":"2024-04-25T01:16:27.696890+0000","last_peered":"2024-04-25T01:16:27.696890+0000","last_clean":"2024-04-25T01:16:27.696890+0000","last_became_active":"2024-04-25T01:16:21.623239+0000","last_became_peered":"2024-04-25T01:16:21.623239+0000","last_unstale":"2024-04-25T01:16:27.696890+0000","last_undegraded":"2024-04-25T01:16:27.696890+0000","last_fullsized":"2024-04-25T01:16:27.696890+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:10:00.414303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.970886+0000","last_change":"2024-04-25T01:16:22.606098+0000","last_active":"2024-04-25T01:16:27.970886+0000","last_peered":"2024-04-25T01:16:27.970886+0000","last_clean":"2024-04-25T01:16:27.970886+0000","last_became_active":"2024-04-25T01:16:22.605632+0000","last_became_peered":"2024-04-25T01:16:22.605632+0000","last_unstale":"2024-04-25T01:16:27.970886+0000","last_undegraded":"2024-04-25T01:16:27.970886+0000","last_fullsized":"2024-04-25T01:16:27.970886+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:18:00.682845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.224854+0000","last_change":"2024-04-25T01:16:26.667852+0000","last_active":"2024-04-25T01:16:28.224854+0000","last_peered":"2024-04-25T01:16:28.224854+0000","last_clean":"2024-04-25T01:16:28.224854+0000","last_became_active":"2024-04-25T01:16:26.667070+0000","last_became_peered":"2024-04-25T01:16:26.667070+0000","last_unstale":"2024-04-25T01:16:28.224854+0000","last_undegraded":"2024-04-25T01:16:28.224854+0000","last_fullsized":"2024-04-25T01:16:28.224854+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:35:37.472927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,0],"acting":[7,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.874985+0000","last_change":"2024-04-25T01:16:22.621505+0000","last_active":"2024-04-25T01:16:27.874985+0000","last_peered":"2024-04-25T01:16:27.874985+0000","last_clean":"2024-04-25T01:16:27.874985+0000","last_became_active":"2024-04-25T01:16:22.621239+0000","last_became_peered":"2024-04-25T01:16:22.621239+0000","last_unstale":"2024-04-25T01:16:27.874985+0000","last_undegraded":"2024-04-25T01:16:27.874985+0000","last_fullsized":"2024-04-25T01:16:27.874985+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:18:00.682845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.223851+0000","last_change":"2024-04-25T01:16:22.619015+0000","last_active":"2024-04-25T01:16:28.223851+0000","last_peered":"2024-04-25T01:16:28.223851+0000","last_clean":"2024-04-25T01:16:28.223851+0000","last_became_active":"2024-04-25T01:16:22.618739+0000","last_became_peered":"2024-04-25T01:16:22.618739+0000","last_unstale":"2024-04-25T01:16:28.223851+0000","last_undegraded":"2024-04-25T01:16:28.223851+0000","last_fullsized":"2024-04-25T01:16:28.223851+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:07.954692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694435+0000","last_change":"2024-04-25T01:16:22.621371+0000","last_active":"2024-04-25T01:16:27.694435+0000","last_peered":"2024-04-25T01:16:27.694435+0000","last_clean":"2024-04-25T01:16:27.694435+0000","last_became_active":"2024-04-25T01:16:22.621094+0000","last_became_peered":"2024-04-25T01:16:22.621094+0000","last_unstale":"2024-04-25T01:16:27.694435+0000","last_undegraded":"2024-04-25T01:16:27.694435+0000","last_fullsized":"2024-04-25T01:16:27.694435+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:23:15.759529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696710+0000","last_change":"2024-04-25T01:16:22.613471+0000","last_active":"2024-04-25T01:16:27.696710+0000","last_peered":"2024-04-25T01:16:27.696710+0000","last_clean":"2024-04-25T01:16:27.696710+0000","last_became_active":"2024-04-25T01:16:22.613036+0000","last_became_peered":"2024-04-25T01:16:22.613036+0000","last_unstale":"2024-04-25T01:16:27.696710+0000","last_undegraded":"2024-04-25T01:16:27.696710+0000","last_fullsized":"2024-04-25T01:16:27.696710+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:24:38.413314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.088877+0000","last_change":"2024-04-25T01:16:22.634780+0000","last_active":"2024-04-25T01:16:30.088877+0000","last_peered":"2024-04-25T01:16:30.088877+0000","last_clean":"2024-04-25T01:16:30.088877+0000","last_became_active":"2024-04-25T01:16:22.634412+0000","last_became_peered":"2024-04-25T01:16:22.634412+0000","last_unstale":"2024-04-25T01:16:30.088877+0000","last_undegraded":"2024-04-25T01:16:30.088877+0000","last_fullsized":"2024-04-25T01:16:30.088877+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:18:36.768908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.088852+0000","last_change":"2024-04-25T01:16:22.629001+0000","last_active":"2024-04-25T01:16:30.088852+0000","last_peered":"2024-04-25T01:16:30.088852+0000","last_clean":"2024-04-25T01:16:30.088852+0000","last_became_active":"2024-04-25T01:16:22.626649+0000","last_became_peered":"2024-04-25T01:16:22.626649+0000","last_unstale":"2024-04-25T01:16:30.088852+0000","last_undegraded":"2024-04-25T01:16:30.088852+0000","last_fullsized":"2024-04-25T01:16:30.088852+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:27:44.210712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.876273+0000","last_change":"2024-04-25T01:16:22.633405+0000","last_active":"2024-04-25T01:16:27.876273+0000","last_peered":"2024-04-25T01:16:27.876273+0000","last_clean":"2024-04-25T01:16:27.876273+0000","last_became_active":"2024-04-25T01:16:22.632539+0000","last_became_peered":"2024-04-25T01:16:22.632539+0000","last_unstale":"2024-04-25T01:16:27.876273+0000","last_undegraded":"2024-04-25T01:16:27.876273+0000","last_fullsized":"2024-04-25T01:16:27.876273+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:38:09.671586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699448+0000","last_change":"2024-04-25T01:16:22.615235+0000","last_active":"2024-04-25T01:16:27.699448+0000","last_peered":"2024-04-25T01:16:27.699448+0000","last_clean":"2024-04-25T01:16:27.699448+0000","last_became_active":"2024-04-25T01:16:22.614864+0000","last_became_peered":"2024-04-25T01:16:22.614864+0000","last_unstale":"2024-04-25T01:16:27.699448+0000","last_undegraded":"2024-04-25T01:16:27.699448+0000","last_fullsized":"2024-04-25T01:16:27.699448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:39:14.902973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.693959+0000","last_change":"2024-04-25T01:16:22.615333+0000","last_active":"2024-04-25T01:16:27.693959+0000","last_peered":"2024-04-25T01:16:27.693959+0000","last_clean":"2024-04-25T01:16:27.693959+0000","last_became_active":"2024-04-25T01:16:22.614510+0000","last_became_peered":"2024-04-25T01:16:22.614510+0000","last_unstale":"2024-04-25T01:16:27.693959+0000","last_undegraded":"2024-04-25T01:16:27.693959+0000","last_fullsized":"2024-04-25T01:16:27.693959+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:58:06.081060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696037+0000","last_change":"2024-04-25T01:16:22.615852+0000","last_active":"2024-04-25T01:16:27.696037+0000","last_peered":"2024-04-25T01:16:27.696037+0000","last_clean":"2024-04-25T01:16:27.696037+0000","last_became_active":"2024-04-25T01:16:22.615517+0000","last_became_peered":"2024-04-25T01:16:22.615517+0000","last_unstale":"2024-04-25T01:16:27.696037+0000","last_undegraded":"2024-04-25T01:16:27.696037+0000","last_fullsized":"2024-04-25T01:16:27.696037+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:31.459176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.970579+0000","last_change":"2024-04-25T01:16:22.627717+0000","last_active":"2024-04-25T01:16:27.970579+0000","last_peered":"2024-04-25T01:16:27.970579+0000","last_clean":"2024-04-25T01:16:27.970579+0000","last_became_active":"2024-04-25T01:16:22.627359+0000","last_became_peered":"2024-04-25T01:16:22.627359+0000","last_unstale":"2024-04-25T01:16:27.970579+0000","last_undegraded":"2024-04-25T01:16:27.970579+0000","last_fullsized":"2024-04-25T01:16:27.970579+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:51.897522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696423+0000","last_change":"2024-04-25T01:16:22.634253+0000","last_active":"2024-04-25T01:16:27.696423+0000","last_peered":"2024-04-25T01:16:27.696423+0000","last_clean":"2024-04-25T01:16:27.696423+0000","last_became_active":"2024-04-25T01:16:22.632643+0000","last_became_peered":"2024-04-25T01:16:22.632643+0000","last_unstale":"2024-04-25T01:16:27.696423+0000","last_undegraded":"2024-04-25T01:16:27.696423+0000","last_fullsized":"2024-04-25T01:16:27.696423+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:14:10.348493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774424+0000","last_change":"2024-04-25T01:16:22.627377+0000","last_active":"2024-04-25T01:16:29.774424+0000","last_peered":"2024-04-25T01:16:29.774424+0000","last_clean":"2024-04-25T01:16:29.774424+0000","last_became_active":"2024-04-25T01:16:22.626141+0000","last_became_peered":"2024-04-25T01:16:22.626141+0000","last_unstale":"2024-04-25T01:16:29.774424+0000","last_undegraded":"2024-04-25T01:16:29.774424+0000","last_fullsized":"2024-04-25T01:16:29.774424+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:58:33.743727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695050+0000","last_change":"2024-04-25T01:16:22.622416+0000","last_active":"2024-04-25T01:16:27.695050+0000","last_peered":"2024-04-25T01:16:27.695050+0000","last_clean":"2024-04-25T01:16:27.695050+0000","last_became_active":"2024-04-25T01:16:22.620963+0000","last_became_peered":"2024-04-25T01:16:22.620963+0000","last_unstale":"2024-04-25T01:16:27.695050+0000","last_undegraded":"2024-04-25T01:16:27.695050+0000","last_fullsized":"2024-04-25T01:16:27.695050+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:58:32.477986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.225310+0000","last_change":"2024-04-25T01:16:22.614902+0000","last_active":"2024-04-25T01:16:28.225310+0000","last_peered":"2024-04-25T01:16:28.225310+0000","last_clean":"2024-04-25T01:16:28.225310+0000","last_became_active":"2024-04-25T01:16:22.614621+0000","last_became_peered":"2024-04-25T01:16:22.614621+0000","last_unstale":"2024-04-25T01:16:28.225310+0000","last_undegraded":"2024-04-25T01:16:28.225310+0000","last_fullsized":"2024-04-25T01:16:28.225310+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:45.689780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695622+0000","last_change":"2024-04-25T01:16:22.637378+0000","last_active":"2024-04-25T01:16:27.695622+0000","last_peered":"2024-04-25T01:16:27.695622+0000","last_clean":"2024-04-25T01:16:27.695622+0000","last_became_active":"2024-04-25T01:16:22.636599+0000","last_became_peered":"2024-04-25T01:16:22.636599+0000","last_unstale":"2024-04-25T01:16:27.695622+0000","last_undegraded":"2024-04-25T01:16:27.695622+0000","last_fullsized":"2024-04-25T01:16:27.695622+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:44:52.172147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.693572+0000","last_change":"2024-04-25T01:16:22.621416+0000","last_active":"2024-04-25T01:16:27.693572+0000","last_peered":"2024-04-25T01:16:27.693572+0000","last_clean":"2024-04-25T01:16:27.693572+0000","last_became_active":"2024-04-25T01:16:22.619180+0000","last_became_peered":"2024-04-25T01:16:22.619180+0000","last_unstale":"2024-04-25T01:16:27.693572+0000","last_undegraded":"2024-04-25T01:16:27.693572+0000","last_fullsized":"2024-04-25T01:16:27.693572+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:46:16.608442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695613+0000","last_change":"2024-04-25T01:16:22.623812+0000","last_active":"2024-04-25T01:16:27.695613+0000","last_peered":"2024-04-25T01:16:27.695613+0000","last_clean":"2024-04-25T01:16:27.695613+0000","last_became_active":"2024-04-25T01:16:22.623468+0000","last_became_peered":"2024-04-25T01:16:22.623468+0000","last_unstale":"2024-04-25T01:16:27.695613+0000","last_undegraded":"2024-04-25T01:16:27.695613+0000","last_fullsized":"2024-04-25T01:16:27.695613+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:07:25.313465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.087421+0000","last_change":"2024-04-25T01:16:22.622280+0000","last_active":"2024-04-25T01:16:30.087421+0000","last_peered":"2024-04-25T01:16:30.087421+0000","last_clean":"2024-04-25T01:16:30.087421+0000","last_became_active":"2024-04-25T01:16:22.621901+0000","last_became_peered":"2024-04-25T01:16:22.621901+0000","last_unstale":"2024-04-25T01:16:30.087421+0000","last_undegraded":"2024-04-25T01:16:30.087421+0000","last_fullsized":"2024-04-25T01:16:30.087421+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:09:42.404928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.087951+0000","last_change":"2024-04-25T01:16:22.635778+0000","last_active":"2024-04-25T01:16:30.087951+0000","last_peered":"2024-04-25T01:16:30.087951+0000","last_clean":"2024-04-25T01:16:30.087951+0000","last_became_active":"2024-04-25T01:16:22.635492+0000","last_became_peered":"2024-04-25T01:16:22.635492+0000","last_unstale":"2024-04-25T01:16:30.087951+0000","last_undegraded":"2024-04-25T01:16:30.087951+0000","last_fullsized":"2024-04-25T01:16:30.087951+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:16:27.803006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774190+0000","last_change":"2024-04-25T01:16:22.627185+0000","last_active":"2024-04-25T01:16:29.774190+0000","last_peered":"2024-04-25T01:16:29.774190+0000","last_clean":"2024-04-25T01:16:29.774190+0000","last_became_active":"2024-04-25T01:16:22.626147+0000","last_became_peered":"2024-04-25T01:16:22.626147+0000","last_unstale":"2024-04-25T01:16:29.774190+0000","last_undegraded":"2024-04-25T01:16:29.774190+0000","last_fullsized":"2024-04-25T01:16:29.774190+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:06:31.257662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774396+0000","last_change":"2024-04-25T01:16:22.627705+0000","last_active":"2024-04-25T01:16:29.774396+0000","last_peered":"2024-04-25T01:16:29.774396+0000","last_clean":"2024-04-25T01:16:29.774396+0000","last_became_active":"2024-04-25T01:16:22.627424+0000","last_became_peered":"2024-04-25T01:16:22.627424+0000","last_unstale":"2024-04-25T01:16:29.774396+0000","last_undegraded":"2024-04-25T01:16:29.774396+0000","last_fullsized":"2024-04-25T01:16:29.774396+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:58:06.081060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774278+0000","last_change":"2024-04-25T01:16:22.625859+0000","last_active":"2024-04-25T01:16:29.774278+0000","last_peered":"2024-04-25T01:16:29.774278+0000","last_clean":"2024-04-25T01:16:29.774278+0000","last_became_active":"2024-04-25T01:16:22.625512+0000","last_became_peered":"2024-04-25T01:16:22.625512+0000","last_unstale":"2024-04-25T01:16:29.774278+0000","last_undegraded":"2024-04-25T01:16:29.774278+0000","last_fullsized":"2024-04-25T01:16:29.774278+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:09.176572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.971952+0000","last_change":"2024-04-25T01:16:22.622906+0000","last_active":"2024-04-25T01:16:27.971952+0000","last_peered":"2024-04-25T01:16:27.971952+0000","last_clean":"2024-04-25T01:16:27.971952+0000","last_became_active":"2024-04-25T01:16:22.622619+0000","last_became_peered":"2024-04-25T01:16:22.622619+0000","last_unstale":"2024-04-25T01:16:27.971952+0000","last_undegraded":"2024-04-25T01:16:27.971952+0000","last_fullsized":"2024-04-25T01:16:27.971952+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:20.635412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694511+0000","last_change":"2024-04-25T01:16:22.623806+0000","last_active":"2024-04-25T01:16:27.694511+0000","last_peered":"2024-04-25T01:16:27.694511+0000","last_clean":"2024-04-25T01:16:27.694511+0000","last_became_active":"2024-04-25T01:16:22.623165+0000","last_became_peered":"2024-04-25T01:16:22.623165+0000","last_unstale":"2024-04-25T01:16:27.694511+0000","last_undegraded":"2024-04-25T01:16:27.694511+0000","last_fullsized":"2024-04-25T01:16:27.694511+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:01:54.012020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.773234+0000","last_change":"2024-04-25T01:16:22.621330+0000","last_active":"2024-04-25T01:16:29.773234+0000","last_peered":"2024-04-25T01:16:29.773234+0000","last_clean":"2024-04-25T01:16:29.773234+0000","last_became_active":"2024-04-25T01:16:22.620993+0000","last_became_peered":"2024-04-25T01:16:22.620993+0000","last_unstale":"2024-04-25T01:16:29.773234+0000","last_undegraded":"2024-04-25T01:16:29.773234+0000","last_fullsized":"2024-04-25T01:16:29.773234+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:53.625025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.875525+0000","last_change":"2024-04-25T01:16:22.623636+0000","last_active":"2024-04-25T01:16:27.875525+0000","last_peered":"2024-04-25T01:16:27.875525+0000","last_clean":"2024-04-25T01:16:27.875525+0000","last_became_active":"2024-04-25T01:16:22.623365+0000","last_became_peered":"2024-04-25T01:16:22.623365+0000","last_unstale":"2024-04-25T01:16:27.875525+0000","last_undegraded":"2024-04-25T01:16:27.875525+0000","last_fullsized":"2024-04-25T01:16:27.875525+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:49:05.483350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.773871+0000","last_change":"2024-04-25T01:16:22.616635+0000","last_active":"2024-04-25T01:16:29.773871+0000","last_peered":"2024-04-25T01:16:29.773871+0000","last_clean":"2024-04-25T01:16:29.773871+0000","last_became_active":"2024-04-25T01:16:22.616342+0000","last_became_peered":"2024-04-25T01:16:22.616342+0000","last_unstale":"2024-04-25T01:16:29.773871+0000","last_undegraded":"2024-04-25T01:16:29.773871+0000","last_fullsized":"2024-04-25T01:16:29.773871+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:09:07.843928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.875763+0000","last_change":"2024-04-25T01:16:22.623875+0000","last_active":"2024-04-25T01:16:27.875763+0000","last_peered":"2024-04-25T01:16:27.875763+0000","last_clean":"2024-04-25T01:16:27.875763+0000","last_became_active":"2024-04-25T01:16:22.623524+0000","last_became_peered":"2024-04-25T01:16:22.623524+0000","last_unstale":"2024-04-25T01:16:27.875763+0000","last_undegraded":"2024-04-25T01:16:27.875763+0000","last_fullsized":"2024-04-25T01:16:27.875763+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:07:32.166451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.698853+0000","last_change":"2024-04-25T01:16:22.620609+0000","last_active":"2024-04-25T01:16:27.698853+0000","last_peered":"2024-04-25T01:16:27.698853+0000","last_clean":"2024-04-25T01:16:27.698853+0000","last_became_active":"2024-04-25T01:16:22.620261+0000","last_became_peered":"2024-04-25T01:16:22.620261+0000","last_unstale":"2024-04-25T01:16:27.698853+0000","last_undegraded":"2024-04-25T01:16:27.698853+0000","last_fullsized":"2024-04-25T01:16:27.698853+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:01.416632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699491+0000","last_change":"2024-04-25T01:16:22.633173+0000","last_active":"2024-04-25T01:16:27.699491+0000","last_peered":"2024-04-25T01:16:27.699491+0000","last_clean":"2024-04-25T01:16:27.699491+0000","last_became_active":"2024-04-25T01:16:22.632857+0000","last_became_peered":"2024-04-25T01:16:22.632857+0000","last_unstale":"2024-04-25T01:16:27.699491+0000","last_undegraded":"2024-04-25T01:16:27.699491+0000","last_fullsized":"2024-04-25T01:16:27.699491+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:14.238333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774391+0000","last_change":"2024-04-25T01:16:22.628348+0000","last_active":"2024-04-25T01:16:29.774391+0000","last_peered":"2024-04-25T01:16:29.774391+0000","last_clean":"2024-04-25T01:16:29.774391+0000","last_became_active":"2024-04-25T01:16:22.627597+0000","last_became_peered":"2024-04-25T01:16:22.627597+0000","last_unstale":"2024-04-25T01:16:29.774391+0000","last_undegraded":"2024-04-25T01:16:29.774391+0000","last_fullsized":"2024-04-25T01:16:29.774391+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:23:15.759529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542153,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":108288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542152,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":108164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542153,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":876544,"data_stored":698624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542152,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":294912,"data_stored":112678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542152,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":108746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542154,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":286720,"data_stored":112006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542152,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":872448,"data_stored":698602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542153,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":108220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:16:52.336 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:16:52.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.592+0000 7fb64b800640 1 Processor -- start 2024-04-25T01:16:52.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.592+0000 7fb64b800640 1 -- start start 2024-04-25T01:16:52.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.594+0000 7fb64b800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb644145590 0x7fb6440b4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:52.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.594+0000 7fb64b800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb64415f9b0 0x7fb64415fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:52.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.594+0000 7fb64b800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6440b4240 con 0x7fb6440ce110 2024-04-25T01:16:52.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.594+0000 7fb64b800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fb644160830 con 0x7fb644145590 2024-04-25T01:16:52.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.594+0000 7fb64b800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb644160350 con 0x7fb64415f9b0 2024-04-25T01:16:52.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.594+0000 7fb648e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb644145590 0x7fb6440b4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:52.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.594+0000 7fb648e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb644145590 0x7fb6440b4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33494/0 (socket says 172.21.15.90:33494) 2024-04-25T01:16:52.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.594+0000 7fb648e00640 1 -- 172.21.15.90:0/1616150126 learned_addr learned my addr 172.21.15.90:0/1616150126 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:52.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.594+0000 7fb63be00640 1 --2- 172.21.15.90:0/1616150126 >> v2:172.21.15.103:3300/0 conn(0x7fb64415f9b0 0x7fb64415fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:52.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.594+0000 7fb648e00640 1 -- 172.21.15.90:0/1616150126 >> v2:172.21.15.103:3300/0 conn(0x7fb64415f9b0 msgr2=0x7fb64415fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:52.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.595+0000 7fb648e00640 1 --2- 172.21.15.90:0/1616150126 >> v2:172.21.15.103:3300/0 conn(0x7fb64415f9b0 0x7fb64415fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:52.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.595+0000 7fb648e00640 1 -- 172.21.15.90:0/1616150126 >> v1:172.21.15.90:6789/0 conn(0x7fb6440ce110 legacy=0x7fb6440ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:52.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.595+0000 7fb648e00640 1 -- 172.21.15.90:0/1616150126 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6440b4f60 con 0x7fb644145590 2024-04-25T01:16:52.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.595+0000 7fb643e00640 1 -- 172.21.15.90:0/1616150126 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 657150184 0 0) 0x7fb6440b4240 con 0x7fb6440ce110 2024-04-25T01:16:52.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.596+0000 7fb648e00640 1 --2- 172.21.15.90:0/1616150126 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb644145590 0x7fb6440b4a20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb6440b46c0 tx=0x7fb62c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad562dd6346080d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:52.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.596+0000 7fb643e00640 1 -- 172.21.15.90:0/1616150126 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb62c037070 con 0x7fb644145590 2024-04-25T01:16:52.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.596+0000 7fb643e00640 1 -- 172.21.15.90:0/1616150126 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb62c032440 con 0x7fb644145590 2024-04-25T01:16:52.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.596+0000 7fb643e00640 1 -- 172.21.15.90:0/1616150126 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb62c032810 con 0x7fb644145590 2024-04-25T01:16:52.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.596+0000 7fb64b800640 1 -- 172.21.15.90:0/1616150126 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb644145590 msgr2=0x7fb6440b4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:52.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.596+0000 7fb64b800640 1 --2- 172.21.15.90:0/1616150126 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb644145590 0x7fb6440b4a20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb6440b46c0 tx=0x7fb62c02a270 comp rx=0 tx=0).stop 2024-04-25T01:16:52.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.597+0000 7fb64b800640 1 -- 172.21.15.90:0/1616150126 shutdown_connections 2024-04-25T01:16:52.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.597+0000 7fb64b800640 1 --2- 172.21.15.90:0/1616150126 >> v2:172.21.15.103:3300/0 conn(0x7fb64415f9b0 0x7fb64415fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:52.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.597+0000 7fb64b800640 1 --2- 172.21.15.90:0/1616150126 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb644145590 0x7fb6440b4a20 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:52.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.597+0000 7fb64b800640 1 -- 172.21.15.90:0/1616150126 >> 172.21.15.90:0/1616150126 conn(0x7fb6440b1820 msgr2=0x7fb6440b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:52.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.597+0000 7fb64b800640 1 -- 172.21.15.90:0/1616150126 shutdown_connections 2024-04-25T01:16:52.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.597+0000 7fb64b800640 1 -- 172.21.15.90:0/1616150126 wait complete. 2024-04-25T01:16:52.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.598+0000 7fb64b800640 1 Processor -- start 2024-04-25T01:16:52.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.599+0000 7fb64b800640 1 -- start start 2024-04-25T01:16:52.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.599+0000 7fb64b800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6440ce110 0x7fb64413db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:52.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.600+0000 7fb64b800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb644145590 0x7fb644139c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:52.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.600+0000 7fb64b800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb64415f9b0 0x7fb64413a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:52.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.600+0000 7fb64b800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb6440b3c80 con 0x7fb644145590 2024-04-25T01:16:52.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.600+0000 7fb64b800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb64406cde0 con 0x7fb6440ce110 2024-04-25T01:16:52.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.600+0000 7fb64b800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb6440b3f60 con 0x7fb64415f9b0 2024-04-25T01:16:52.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.600+0000 7fb63be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb644145590 0x7fb644139c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:52.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.600+0000 7fb63be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb644145590 0x7fb644139c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33500/0 (socket says 172.21.15.90:33500) 2024-04-25T01:16:52.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.600+0000 7fb63be00640 1 -- 172.21.15.90:0/869639794 learned_addr learned my addr 172.21.15.90:0/869639794 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:52.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.600+0000 7fb648e00640 1 --2- 172.21.15.90:0/869639794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6440ce110 0x7fb64413db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:52.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.600+0000 7fb649800640 1 --2- 172.21.15.90:0/869639794 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb64415f9b0 0x7fb64413a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:52.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.600+0000 7fb63be00640 1 -- 172.21.15.90:0/869639794 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb64415f9b0 msgr2=0x7fb64413a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:52.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.600+0000 7fb63be00640 1 --2- 172.21.15.90:0/869639794 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb64415f9b0 0x7fb64413a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:52.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.600+0000 7fb63be00640 1 -- 172.21.15.90:0/869639794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6440ce110 msgr2=0x7fb64413db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:52.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.600+0000 7fb63be00640 1 --2- 172.21.15.90:0/869639794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6440ce110 0x7fb64413db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:52.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.600+0000 7fb63be00640 1 -- 172.21.15.90:0/869639794 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb64413e0a0 con 0x7fb644145590 2024-04-25T01:16:52.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.601+0000 7fb63be00640 1 --2- 172.21.15.90:0/869639794 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb644145590 0x7fb644139c80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb63400d620 tx=0x7fb63400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:52.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.601+0000 7fb641600640 1 -- 172.21.15.90:0/869639794 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63400b620 con 0x7fb644145590 2024-04-25T01:16:52.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.601+0000 7fb641600640 1 -- 172.21.15.90:0/869639794 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb634019070 con 0x7fb644145590 2024-04-25T01:16:52.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.601+0000 7fb641600640 1 -- 172.21.15.90:0/869639794 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63400ba00 con 0x7fb644145590 2024-04-25T01:16:52.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.601+0000 7fb64b800640 1 -- 172.21.15.90:0/869639794 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb64413a7f0 con 0x7fb644145590 2024-04-25T01:16:52.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.601+0000 7fb64b800640 1 -- 172.21.15.90:0/869639794 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb644071140 con 0x7fb644145590 2024-04-25T01:16:52.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.602+0000 7fb64b800640 1 -- 172.21.15.90:0/869639794 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb644068fb0 con 0x7fb644145590 2024-04-25T01:16:52.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.603+0000 7fb641600640 1 -- 172.21.15.90:0/869639794 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb63400bba0 con 0x7fb644145590 2024-04-25T01:16:52.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.604+0000 7fb641600640 1 --2- 172.21.15.90:0/869639794 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb62006be30 0x7fb62006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:52.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.604+0000 7fb641600640 1 -- 172.21.15.90:0/869639794 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fb63408b650 con 0x7fb644145590 2024-04-25T01:16:52.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.604+0000 7fb648e00640 1 --2- 172.21.15.90:0/869639794 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb62006be30 0x7fb62006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:52.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.605+0000 7fb648e00640 1 --2- 172.21.15.90:0/869639794 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb62006be30 0x7fb62006e2f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb62c034000 tx=0x7fb62c02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:52.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.609+0000 7fb641600640 1 -- 172.21.15.90:0/869639794 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb634059c20 con 0x7fb644145590 2024-04-25T01:16:52.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.789+0000 7fb64b800640 1 -- 172.21.15.90:0/869639794 --> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb64406ea50 con 0x7fb62006be30 2024-04-25T01:16:52.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.813+0000 7fb641600640 1 -- 172.21.15.90:0/869639794 <== mgr.4104 v2:172.21.15.90:6832/2198086733 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332534 (secure 0 0 0) 0x7fb64406ea50 con 0x7fb62006be30 2024-04-25T01:16:52.799 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:16:52.804 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-25T01:16:52.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.821+0000 7fb64b800640 1 -- 172.21.15.90:0/869639794 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb62006be30 msgr2=0x7fb62006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:52.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.821+0000 7fb64b800640 1 --2- 172.21.15.90:0/869639794 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb62006be30 0x7fb62006e2f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb62c034000 tx=0x7fb62c02e000 comp rx=0 tx=0).stop 2024-04-25T01:16:52.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.821+0000 7fb64b800640 1 -- 172.21.15.90:0/869639794 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb644145590 msgr2=0x7fb644139c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:52.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.821+0000 7fb64b800640 1 --2- 172.21.15.90:0/869639794 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb644145590 0x7fb644139c80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb63400d620 tx=0x7fb63400dae0 comp rx=0 tx=0).stop 2024-04-25T01:16:52.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.821+0000 7fb64b800640 1 -- 172.21.15.90:0/869639794 shutdown_connections 2024-04-25T01:16:52.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.821+0000 7fb64b800640 1 --2- 172.21.15.90:0/869639794 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb62006be30 0x7fb62006e2f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:52.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.821+0000 7fb64b800640 1 --2- 172.21.15.90:0/869639794 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb64415f9b0 0x7fb64413a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:52.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.821+0000 7fb64b800640 1 --2- 172.21.15.90:0/869639794 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb644145590 0x7fb644139c80 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:52.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.821+0000 7fb64b800640 1 --2- 172.21.15.90:0/869639794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6440ce110 0x7fb64413db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:52.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.821+0000 7fb64b800640 1 -- 172.21.15.90:0/869639794 >> 172.21.15.90:0/869639794 conn(0x7fb6440b1820 msgr2=0x7fb6440adea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:52.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.821+0000 7fb64b800640 1 -- 172.21.15.90:0/869639794 shutdown_connections 2024-04-25T01:16:52.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:52.821+0000 7fb64b800640 1 -- 172.21.15.90:0/869639794 wait complete. 2024-04-25T01:16:52.824 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-04-25T01:16:51.833739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219536,"kb_used_data":3368,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755184,"statfs":{"total":773094113280,"available":772869308416,"internally_reserved":0,"allocated":3448832,"data_stored":2055328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1781,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004702"},"pg_stats":[{"pgid":"2.1f","version":"21'4","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.698700+0000","last_change":"2024-04-25T01:16:21.622879+0000","last_active":"2024-04-25T01:16:27.698700+0000","last_peered":"2024-04-25T01:16:27.698700+0000","last_clean":"2024-04-25T01:16:27.698700+0000","last_became_active":"2024-04-25T01:16:21.622586+0000","last_became_peered":"2024-04-25T01:16:21.622586+0000","last_unstale":"2024-04-25T01:16:27.698700+0000","last_undegraded":"2024-04-25T01:16:27.698700+0000","last_fullsized":"2024-04-25T01:16:27.698700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:50.882927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699904+0000","last_change":"2024-04-25T01:16:22.615827+0000","last_active":"2024-04-25T01:16:27.699904+0000","last_peered":"2024-04-25T01:16:27.699904+0000","last_clean":"2024-04-25T01:16:27.699904+0000","last_became_active":"2024-04-25T01:16:22.615497+0000","last_became_peered":"2024-04-25T01:16:22.615497+0000","last_unstale":"2024-04-25T01:16:27.699904+0000","last_undegraded":"2024-04-25T01:16:27.699904+0000","last_fullsized":"2024-04-25T01:16:27.699904+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:03:53.876481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.223962+0000","last_change":"2024-04-25T01:16:26.672437+0000","last_active":"2024-04-25T01:16:28.223962+0000","last_peered":"2024-04-25T01:16:28.223962+0000","last_clean":"2024-04-25T01:16:28.223962+0000","last_became_active":"2024-04-25T01:16:26.671999+0000","last_became_peered":"2024-04-25T01:16:26.671999+0000","last_unstale":"2024-04-25T01:16:28.223962+0000","last_undegraded":"2024-04-25T01:16:28.223962+0000","last_fullsized":"2024-04-25T01:16:28.223962+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:56.586900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4,1],"acting":[7,6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.700145+0000","last_change":"2024-04-25T01:16:21.609447+0000","last_active":"2024-04-25T01:16:27.700145+0000","last_peered":"2024-04-25T01:16:27.700145+0000","last_clean":"2024-04-25T01:16:27.700145+0000","last_became_active":"2024-04-25T01:16:21.608490+0000","last_became_peered":"2024-04-25T01:16:21.608490+0000","last_unstale":"2024-04-25T01:16:27.700145+0000","last_undegraded":"2024-04-25T01:16:27.700145+0000","last_fullsized":"2024-04-25T01:16:27.700145+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:15.007939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699632+0000","last_change":"2024-04-25T01:16:22.613891+0000","last_active":"2024-04-25T01:16:27.699632+0000","last_peered":"2024-04-25T01:16:27.699632+0000","last_clean":"2024-04-25T01:16:27.699632+0000","last_became_active":"2024-04-25T01:16:22.613471+0000","last_became_peered":"2024-04-25T01:16:22.613471+0000","last_unstale":"2024-04-25T01:16:27.699632+0000","last_undegraded":"2024-04-25T01:16:27.699632+0000","last_fullsized":"2024-04-25T01:16:27.699632+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:37:00.182534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.874736+0000","last_change":"2024-04-25T01:16:26.653705+0000","last_active":"2024-04-25T01:16:27.874736+0000","last_peered":"2024-04-25T01:16:27.874736+0000","last_clean":"2024-04-25T01:16:27.874736+0000","last_became_active":"2024-04-25T01:16:26.653348+0000","last_became_peered":"2024-04-25T01:16:26.653348+0000","last_unstale":"2024-04-25T01:16:27.874736+0000","last_undegraded":"2024-04-25T01:16:27.874736+0000","last_fullsized":"2024-04-25T01:16:27.874736+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:58:37.794558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4,6],"acting":[2,7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.772478+0000","last_change":"2024-04-25T01:16:21.611644+0000","last_active":"2024-04-25T01:16:29.772478+0000","last_peered":"2024-04-25T01:16:29.772478+0000","last_clean":"2024-04-25T01:16:29.772478+0000","last_became_active":"2024-04-25T01:16:21.611346+0000","last_became_peered":"2024-04-25T01:16:21.611346+0000","last_unstale":"2024-04-25T01:16:29.772478+0000","last_undegraded":"2024-04-25T01:16:29.772478+0000","last_fullsized":"2024-04-25T01:16:29.772478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:44.675184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.971313+0000","last_change":"2024-04-25T01:16:22.627961+0000","last_active":"2024-04-25T01:16:27.971313+0000","last_peered":"2024-04-25T01:16:27.971313+0000","last_clean":"2024-04-25T01:16:27.971313+0000","last_became_active":"2024-04-25T01:16:22.626767+0000","last_became_peered":"2024-04-25T01:16:22.626767+0000","last_unstale":"2024-04-25T01:16:27.971313+0000","last_undegraded":"2024-04-25T01:16:27.971313+0000","last_fullsized":"2024-04-25T01:16:27.971313+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:49:05.483350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.875764+0000","last_change":"2024-04-25T01:16:26.657547+0000","last_active":"2024-04-25T01:16:27.875764+0000","last_peered":"2024-04-25T01:16:27.875764+0000","last_clean":"2024-04-25T01:16:27.875764+0000","last_became_active":"2024-04-25T01:16:26.656643+0000","last_became_peered":"2024-04-25T01:16:26.656643+0000","last_unstale":"2024-04-25T01:16:27.875764+0000","last_undegraded":"2024-04-25T01:16:27.875764+0000","last_fullsized":"2024-04-25T01:16:27.875764+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:05.563072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3,6],"acting":[2,5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.772473+0000","last_change":"2024-04-25T01:16:21.617309+0000","last_active":"2024-04-25T01:16:29.772473+0000","last_peered":"2024-04-25T01:16:29.772473+0000","last_clean":"2024-04-25T01:16:29.772473+0000","last_became_active":"2024-04-25T01:16:21.617021+0000","last_became_peered":"2024-04-25T01:16:21.617021+0000","last_unstale":"2024-04-25T01:16:29.772473+0000","last_undegraded":"2024-04-25T01:16:29.772473+0000","last_fullsized":"2024-04-25T01:16:29.772473+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:06.940097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694375+0000","last_change":"2024-04-25T01:16:22.639034+0000","last_active":"2024-04-25T01:16:27.694375+0000","last_peered":"2024-04-25T01:16:27.694375+0000","last_clean":"2024-04-25T01:16:27.694375+0000","last_became_active":"2024-04-25T01:16:22.638661+0000","last_became_peered":"2024-04-25T01:16:22.638661+0000","last_unstale":"2024-04-25T01:16:27.694375+0000","last_undegraded":"2024-04-25T01:16:27.694375+0000","last_fullsized":"2024-04-25T01:16:27.694375+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:28.044482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.875219+0000","last_change":"2024-04-25T01:16:26.656419+0000","last_active":"2024-04-25T01:16:27.875219+0000","last_peered":"2024-04-25T01:16:27.875219+0000","last_clean":"2024-04-25T01:16:27.875219+0000","last_became_active":"2024-04-25T01:16:26.655743+0000","last_became_peered":"2024-04-25T01:16:26.655743+0000","last_unstale":"2024-04-25T01:16:27.875219+0000","last_undegraded":"2024-04-25T01:16:27.875219+0000","last_fullsized":"2024-04-25T01:16:27.875219+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:41:04.510055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5,7],"acting":[2,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694711+0000","last_change":"2024-04-25T01:16:22.621126+0000","last_active":"2024-04-25T01:16:27.694711+0000","last_peered":"2024-04-25T01:16:27.694711+0000","last_clean":"2024-04-25T01:16:27.694711+0000","last_became_active":"2024-04-25T01:16:22.620692+0000","last_became_peered":"2024-04-25T01:16:22.620692+0000","last_unstale":"2024-04-25T01:16:27.694711+0000","last_undegraded":"2024-04-25T01:16:27.694711+0000","last_fullsized":"2024-04-25T01:16:27.694711+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:28.261961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694834+0000","last_change":"2024-04-25T01:16:21.608564+0000","last_active":"2024-04-25T01:16:27.694834+0000","last_peered":"2024-04-25T01:16:27.694834+0000","last_clean":"2024-04-25T01:16:27.694834+0000","last_became_active":"2024-04-25T01:16:21.608227+0000","last_became_peered":"2024-04-25T01:16:21.608227+0000","last_unstale":"2024-04-25T01:16:27.694834+0000","last_undegraded":"2024-04-25T01:16:27.694834+0000","last_fullsized":"2024-04-25T01:16:27.694834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:38:08.656990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694160+0000","last_change":"2024-04-25T01:16:26.656167+0000","last_active":"2024-04-25T01:16:27.694160+0000","last_peered":"2024-04-25T01:16:27.694160+0000","last_clean":"2024-04-25T01:16:27.694160+0000","last_became_active":"2024-04-25T01:16:26.655643+0000","last_became_peered":"2024-04-25T01:16:26.655643+0000","last_unstale":"2024-04-25T01:16:27.694160+0000","last_undegraded":"2024-04-25T01:16:27.694160+0000","last_fullsized":"2024-04-25T01:16:27.694160+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:17:57.227839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,5],"acting":[3,1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.224130+0000","last_change":"2024-04-25T01:16:22.630130+0000","last_active":"2024-04-25T01:16:28.224130+0000","last_peered":"2024-04-25T01:16:28.224130+0000","last_clean":"2024-04-25T01:16:28.224130+0000","last_became_active":"2024-04-25T01:16:22.629829+0000","last_became_peered":"2024-04-25T01:16:22.629829+0000","last_unstale":"2024-04-25T01:16:28.224130+0000","last_undegraded":"2024-04-25T01:16:28.224130+0000","last_fullsized":"2024-04-25T01:16:28.224130+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:48:42.505862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.086752+0000","last_change":"2024-04-25T01:16:21.615603+0000","last_active":"2024-04-25T01:16:30.086752+0000","last_peered":"2024-04-25T01:16:30.086752+0000","last_clean":"2024-04-25T01:16:30.086752+0000","last_became_active":"2024-04-25T01:16:21.615217+0000","last_became_peered":"2024-04-25T01:16:21.615217+0000","last_unstale":"2024-04-25T01:16:30.086752+0000","last_undegraded":"2024-04-25T01:16:30.086752+0000","last_fullsized":"2024-04-25T01:16:30.086752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:10:00.414303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.773845+0000","last_change":"2024-04-25T01:16:26.677623+0000","last_active":"2024-04-25T01:16:29.773845+0000","last_peered":"2024-04-25T01:16:29.773845+0000","last_clean":"2024-04-25T01:16:29.773845+0000","last_became_active":"2024-04-25T01:16:26.677212+0000","last_became_peered":"2024-04-25T01:16:26.677212+0000","last_unstale":"2024-04-25T01:16:29.773845+0000","last_undegraded":"2024-04-25T01:16:29.773845+0000","last_fullsized":"2024-04-25T01:16:29.773845+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:58.601828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6,1],"acting":[0,4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694468+0000","last_change":"2024-04-25T01:16:22.617565+0000","last_active":"2024-04-25T01:16:27.694468+0000","last_peered":"2024-04-25T01:16:27.694468+0000","last_clean":"2024-04-25T01:16:27.694468+0000","last_became_active":"2024-04-25T01:16:22.617247+0000","last_became_peered":"2024-04-25T01:16:22.617247+0000","last_unstale":"2024-04-25T01:16:27.694468+0000","last_undegraded":"2024-04-25T01:16:27.694468+0000","last_fullsized":"2024-04-25T01:16:27.694468+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:00:17.366512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695722+0000","last_change":"2024-04-25T01:16:21.620221+0000","last_active":"2024-04-25T01:16:27.695722+0000","last_peered":"2024-04-25T01:16:27.695722+0000","last_clean":"2024-04-25T01:16:27.695722+0000","last_became_active":"2024-04-25T01:16:21.619923+0000","last_became_peered":"2024-04-25T01:16:21.619923+0000","last_unstale":"2024-04-25T01:16:27.695722+0000","last_undegraded":"2024-04-25T01:16:27.695722+0000","last_fullsized":"2024-04-25T01:16:27.695722+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:06.940097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.225195+0000","last_change":"2024-04-25T01:16:26.674824+0000","last_active":"2024-04-25T01:16:28.225195+0000","last_peered":"2024-04-25T01:16:28.225195+0000","last_clean":"2024-04-25T01:16:28.225195+0000","last_became_active":"2024-04-25T01:16:26.672608+0000","last_became_peered":"2024-04-25T01:16:26.672608+0000","last_unstale":"2024-04-25T01:16:28.225195+0000","last_undegraded":"2024-04-25T01:16:28.225195+0000","last_fullsized":"2024-04-25T01:16:28.225195+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:46:37.320887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2,3],"acting":[7,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.693656+0000","last_change":"2024-04-25T01:16:22.624853+0000","last_active":"2024-04-25T01:16:27.693656+0000","last_peered":"2024-04-25T01:16:27.693656+0000","last_clean":"2024-04-25T01:16:27.693656+0000","last_became_active":"2024-04-25T01:16:22.624275+0000","last_became_peered":"2024-04-25T01:16:22.624275+0000","last_unstale":"2024-04-25T01:16:27.693656+0000","last_undegraded":"2024-04-25T01:16:27.693656+0000","last_fullsized":"2024-04-25T01:16:27.693656+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:09.176572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.772092+0000","last_change":"2024-04-25T01:16:21.616953+0000","last_active":"2024-04-25T01:16:29.772092+0000","last_peered":"2024-04-25T01:16:29.772092+0000","last_clean":"2024-04-25T01:16:29.772092+0000","last_became_active":"2024-04-25T01:16:21.615656+0000","last_became_peered":"2024-04-25T01:16:21.615656+0000","last_unstale":"2024-04-25T01:16:29.772092+0000","last_undegraded":"2024-04-25T01:16:29.772092+0000","last_fullsized":"2024-04-25T01:16:29.772092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:48:41.491267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.876501+0000","last_change":"2024-04-25T01:16:26.661383+0000","last_active":"2024-04-25T01:16:27.876501+0000","last_peered":"2024-04-25T01:16:27.876501+0000","last_clean":"2024-04-25T01:16:27.876501+0000","last_became_active":"2024-04-25T01:16:26.658181+0000","last_became_peered":"2024-04-25T01:16:26.658181+0000","last_unstale":"2024-04-25T01:16:27.876501+0000","last_undegraded":"2024-04-25T01:16:27.876501+0000","last_fullsized":"2024-04-25T01:16:27.876501+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:23:19.810360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,7],"acting":[2,1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.697025+0000","last_change":"2024-04-25T01:16:22.618495+0000","last_active":"2024-04-25T01:16:27.697025+0000","last_peered":"2024-04-25T01:16:27.697025+0000","last_clean":"2024-04-25T01:16:27.697025+0000","last_became_active":"2024-04-25T01:16:22.618155+0000","last_became_peered":"2024-04-25T01:16:22.618155+0000","last_unstale":"2024-04-25T01:16:27.697025+0000","last_undegraded":"2024-04-25T01:16:27.697025+0000","last_fullsized":"2024-04-25T01:16:27.697025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:53.625025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695692+0000","last_change":"2024-04-25T01:16:21.618714+0000","last_active":"2024-04-25T01:16:27.695692+0000","last_peered":"2024-04-25T01:16:27.695692+0000","last_clean":"2024-04-25T01:16:27.695692+0000","last_became_active":"2024-04-25T01:16:21.618439+0000","last_became_peered":"2024-04-25T01:16:21.618439+0000","last_unstale":"2024-04-25T01:16:27.695692+0000","last_undegraded":"2024-04-25T01:16:27.695692+0000","last_fullsized":"2024-04-25T01:16:27.695692+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:17:15.320611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.876136+0000","last_change":"2024-04-25T01:16:26.649207+0000","last_active":"2024-04-25T01:16:27.876136+0000","last_peered":"2024-04-25T01:16:27.876136+0000","last_clean":"2024-04-25T01:16:27.876136+0000","last_became_active":"2024-04-25T01:16:26.648837+0000","last_became_peered":"2024-04-25T01:16:26.648837+0000","last_unstale":"2024-04-25T01:16:27.876136+0000","last_undegraded":"2024-04-25T01:16:27.876136+0000","last_fullsized":"2024-04-25T01:16:27.876136+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:13.227403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,1],"acting":[2,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.698475+0000","last_change":"2024-04-25T01:16:22.627735+0000","last_active":"2024-04-25T01:16:27.698475+0000","last_peered":"2024-04-25T01:16:27.698475+0000","last_clean":"2024-04-25T01:16:27.698475+0000","last_became_active":"2024-04-25T01:16:22.627430+0000","last_became_peered":"2024-04-25T01:16:22.627430+0000","last_unstale":"2024-04-25T01:16:27.698475+0000","last_undegraded":"2024-04-25T01:16:27.698475+0000","last_fullsized":"2024-04-25T01:16:27.698475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:06:31.257662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.697307+0000","last_change":"2024-04-25T01:16:21.620348+0000","last_active":"2024-04-25T01:16:27.697307+0000","last_peered":"2024-04-25T01:16:27.697307+0000","last_clean":"2024-04-25T01:16:27.697307+0000","last_became_active":"2024-04-25T01:16:21.620049+0000","last_became_peered":"2024-04-25T01:16:21.620049+0000","last_unstale":"2024-04-25T01:16:27.697307+0000","last_undegraded":"2024-04-25T01:16:27.697307+0000","last_fullsized":"2024-04-25T01:16:27.697307+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:17:59.668250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694701+0000","last_change":"2024-04-25T01:16:26.672455+0000","last_active":"2024-04-25T01:16:27.694701+0000","last_peered":"2024-04-25T01:16:27.694701+0000","last_clean":"2024-04-25T01:16:27.694701+0000","last_became_active":"2024-04-25T01:16:26.671952+0000","last_became_peered":"2024-04-25T01:16:26.671952+0000","last_unstale":"2024-04-25T01:16:27.694701+0000","last_undegraded":"2024-04-25T01:16:27.694701+0000","last_fullsized":"2024-04-25T01:16:27.694701+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:13.227403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6,3],"acting":[4,1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.088865+0000","last_change":"2024-04-25T01:16:22.628050+0000","last_active":"2024-04-25T01:16:30.088865+0000","last_peered":"2024-04-25T01:16:30.088865+0000","last_clean":"2024-04-25T01:16:30.088865+0000","last_became_active":"2024-04-25T01:16:22.626875+0000","last_became_peered":"2024-04-25T01:16:22.626875+0000","last_unstale":"2024-04-25T01:16:30.088865+0000","last_undegraded":"2024-04-25T01:16:30.088865+0000","last_fullsized":"2024-04-25T01:16:30.088865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:24:16.000792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695500+0000","last_change":"2024-04-25T01:16:21.621352+0000","last_active":"2024-04-25T01:16:27.695500+0000","last_peered":"2024-04-25T01:16:27.695500+0000","last_clean":"2024-04-25T01:16:27.695500+0000","last_became_active":"2024-04-25T01:16:21.620116+0000","last_became_peered":"2024-04-25T01:16:21.620116+0000","last_unstale":"2024-04-25T01:16:27.695500+0000","last_undegraded":"2024-04-25T01:16:27.695500+0000","last_fullsized":"2024-04-25T01:16:27.695500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:07:31.151855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.087941+0000","last_change":"2024-04-25T01:16:26.663371+0000","last_active":"2024-04-25T01:16:30.087941+0000","last_peered":"2024-04-25T01:16:30.087941+0000","last_clean":"2024-04-25T01:16:30.087941+0000","last_became_active":"2024-04-25T01:16:26.662972+0000","last_became_peered":"2024-04-25T01:16:26.662972+0000","last_unstale":"2024-04-25T01:16:30.087941+0000","last_undegraded":"2024-04-25T01:16:30.087941+0000","last_fullsized":"2024-04-25T01:16:30.087941+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:18.289163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2,7],"acting":[6,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.875472+0000","last_change":"2024-04-25T01:16:22.628735+0000","last_active":"2024-04-25T01:16:27.875472+0000","last_peered":"2024-04-25T01:16:27.875472+0000","last_clean":"2024-04-25T01:16:27.875472+0000","last_became_active":"2024-04-25T01:16:22.628429+0000","last_became_peered":"2024-04-25T01:16:22.628429+0000","last_unstale":"2024-04-25T01:16:27.875472+0000","last_undegraded":"2024-04-25T01:16:27.875472+0000","last_fullsized":"2024-04-25T01:16:27.875472+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:20.635412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.697942+0000","last_change":"2024-04-25T01:16:21.622977+0000","last_active":"2024-04-25T01:16:27.697942+0000","last_peered":"2024-04-25T01:16:27.697942+0000","last_clean":"2024-04-25T01:16:27.697942+0000","last_became_active":"2024-04-25T01:16:21.622298+0000","last_became_peered":"2024-04-25T01:16:21.622298+0000","last_unstale":"2024-04-25T01:16:27.697942+0000","last_undegraded":"2024-04-25T01:16:27.697942+0000","last_fullsized":"2024-04-25T01:16:27.697942+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:34.732380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.087427+0000","last_change":"2024-04-25T01:16:26.664207+0000","last_active":"2024-04-25T01:16:30.087427+0000","last_peered":"2024-04-25T01:16:30.087427+0000","last_clean":"2024-04-25T01:16:30.087427+0000","last_became_active":"2024-04-25T01:16:26.663749+0000","last_became_peered":"2024-04-25T01:16:26.663749+0000","last_unstale":"2024-04-25T01:16:30.087427+0000","last_undegraded":"2024-04-25T01:16:30.087427+0000","last_fullsized":"2024-04-25T01:16:30.087427+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:05.467463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5,7],"acting":[6,3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.773873+0000","last_change":"2024-04-25T01:16:22.635293+0000","last_active":"2024-04-25T01:16:29.773873+0000","last_peered":"2024-04-25T01:16:29.773873+0000","last_clean":"2024-04-25T01:16:29.773873+0000","last_became_active":"2024-04-25T01:16:22.635026+0000","last_became_peered":"2024-04-25T01:16:22.635026+0000","last_unstale":"2024-04-25T01:16:29.773873+0000","last_undegraded":"2024-04-25T01:16:29.773873+0000","last_fullsized":"2024-04-25T01:16:29.773873+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:46:16.608442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696109+0000","last_change":"2024-04-25T01:16:21.623621+0000","last_active":"2024-04-25T01:16:27.696109+0000","last_peered":"2024-04-25T01:16:27.696109+0000","last_clean":"2024-04-25T01:16:27.696109+0000","last_became_active":"2024-04-25T01:16:21.623351+0000","last_became_peered":"2024-04-25T01:16:21.623351+0000","last_unstale":"2024-04-25T01:16:27.696109+0000","last_undegraded":"2024-04-25T01:16:27.696109+0000","last_fullsized":"2024-04-25T01:16:27.696109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:49:04.468754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.088614+0000","last_change":"2024-04-25T01:16:26.656694+0000","last_active":"2024-04-25T01:16:30.088614+0000","last_peered":"2024-04-25T01:16:30.088614+0000","last_clean":"2024-04-25T01:16:30.088614+0000","last_became_active":"2024-04-25T01:16:26.656201+0000","last_became_peered":"2024-04-25T01:16:26.656201+0000","last_unstale":"2024-04-25T01:16:30.088614+0000","last_undegraded":"2024-04-25T01:16:30.088614+0000","last_fullsized":"2024-04-25T01:16:30.088614+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:39:18.953803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0,3],"acting":[6,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694802+0000","last_change":"2024-04-25T01:16:22.615076+0000","last_active":"2024-04-25T01:16:27.694802+0000","last_peered":"2024-04-25T01:16:27.694802+0000","last_clean":"2024-04-25T01:16:27.694802+0000","last_became_active":"2024-04-25T01:16:22.613139+0000","last_became_peered":"2024-04-25T01:16:22.613139+0000","last_unstale":"2024-04-25T01:16:27.694802+0000","last_undegraded":"2024-04-25T01:16:27.694802+0000","last_fullsized":"2024-04-25T01:16:27.694802+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:22:36.726844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.224133+0000","last_change":"2024-04-25T01:16:21.614492+0000","last_active":"2024-04-25T01:16:28.224133+0000","last_peered":"2024-04-25T01:16:28.224133+0000","last_clean":"2024-04-25T01:16:28.224133+0000","last_became_active":"2024-04-25T01:16:21.613490+0000","last_became_peered":"2024-04-25T01:16:21.613490+0000","last_unstale":"2024-04-25T01:16:28.224133+0000","last_undegraded":"2024-04-25T01:16:28.224133+0000","last_fullsized":"2024-04-25T01:16:28.224133+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:41:30.225989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.088458+0000","last_change":"2024-04-25T01:16:26.656061+0000","last_active":"2024-04-25T01:16:30.088458+0000","last_peered":"2024-04-25T01:16:30.088458+0000","last_clean":"2024-04-25T01:16:30.088458+0000","last_became_active":"2024-04-25T01:16:26.655189+0000","last_became_peered":"2024-04-25T01:16:26.655189+0000","last_unstale":"2024-04-25T01:16:30.088458+0000","last_undegraded":"2024-04-25T01:16:30.088458+0000","last_fullsized":"2024-04-25T01:16:30.088458+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:37:04.233365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,7],"acting":[6,0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.087313+0000","last_change":"2024-04-25T01:16:22.634160+0000","last_active":"2024-04-25T01:16:30.087313+0000","last_peered":"2024-04-25T01:16:30.087313+0000","last_clean":"2024-04-25T01:16:30.087313+0000","last_became_active":"2024-04-25T01:16:22.633856+0000","last_became_peered":"2024-04-25T01:16:22.633856+0000","last_unstale":"2024-04-25T01:16:30.087313+0000","last_undegraded":"2024-04-25T01:16:30.087313+0000","last_fullsized":"2024-04-25T01:16:30.087313+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:46:05.184032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695122+0000","last_change":"2024-04-25T01:16:21.617465+0000","last_active":"2024-04-25T01:16:27.695122+0000","last_peered":"2024-04-25T01:16:27.695122+0000","last_clean":"2024-04-25T01:16:27.695122+0000","last_became_active":"2024-04-25T01:16:21.617201+0000","last_became_peered":"2024-04-25T01:16:21.617201+0000","last_unstale":"2024-04-25T01:16:27.695122+0000","last_undegraded":"2024-04-25T01:16:27.695122+0000","last_fullsized":"2024-04-25T01:16:27.695122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:48:41.491267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699450+0000","last_change":"2024-04-25T01:16:26.666150+0000","last_active":"2024-04-25T01:16:27.699450+0000","last_peered":"2024-04-25T01:16:27.699450+0000","last_clean":"2024-04-25T01:16:27.699450+0000","last_became_active":"2024-04-25T01:16:26.665620+0000","last_became_peered":"2024-04-25T01:16:26.665620+0000","last_unstale":"2024-04-25T01:16:27.699450+0000","last_undegraded":"2024-04-25T01:16:27.699450+0000","last_fullsized":"2024-04-25T01:16:27.699450+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:51:23.702657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6,1],"acting":[5,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695397+0000","last_change":"2024-04-25T01:16:22.637692+0000","last_active":"2024-04-25T01:16:27.695397+0000","last_peered":"2024-04-25T01:16:27.695397+0000","last_clean":"2024-04-25T01:16:27.695397+0000","last_became_active":"2024-04-25T01:16:22.635225+0000","last_became_peered":"2024-04-25T01:16:22.635225+0000","last_unstale":"2024-04-25T01:16:27.695397+0000","last_undegraded":"2024-04-25T01:16:27.695397+0000","last_fullsized":"2024-04-25T01:16:27.695397+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:11.514022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695136+0000","last_change":"2024-04-25T01:16:21.608226+0000","last_active":"2024-04-25T01:16:27.695136+0000","last_peered":"2024-04-25T01:16:27.695136+0000","last_clean":"2024-04-25T01:16:27.695136+0000","last_became_active":"2024-04-25T01:16:21.607851+0000","last_became_peered":"2024-04-25T01:16:21.607851+0000","last_unstale":"2024-04-25T01:16:27.695136+0000","last_undegraded":"2024-04-25T01:16:27.695136+0000","last_fullsized":"2024-04-25T01:16:27.695136+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:51.521474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":178,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.970532+0000","last_change":"2024-04-25T01:16:17.466569+0000","last_active":"2024-04-25T01:16:27.970532+0000","last_peered":"2024-04-25T01:16:27.970532+0000","last_clean":"2024-04-25T01:16:27.970532+0000","last_became_active":"2024-04-25T01:16:17.465687+0000","last_became_peered":"2024-04-25T01:16:17.465687+0000","last_unstale":"2024-04-25T01:16:27.970532+0000","last_undegraded":"2024-04-25T01:16:27.970532+0000","last_fullsized":"2024-04-25T01:16:27.970532+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:16.440559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:16.440559+0000","last_clean_scrub_stamp":"2024-04-25T01:16:16.440559+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:18:17.493555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695469+0000","last_change":"2024-04-25T01:16:26.646040+0000","last_active":"2024-04-25T01:16:27.695469+0000","last_peered":"2024-04-25T01:16:27.695469+0000","last_clean":"2024-04-25T01:16:27.695469+0000","last_became_active":"2024-04-25T01:16:26.645500+0000","last_became_peered":"2024-04-25T01:16:26.645500+0000","last_unstale":"2024-04-25T01:16:27.695469+0000","last_undegraded":"2024-04-25T01:16:27.695469+0000","last_fullsized":"2024-04-25T01:16:27.695469+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:01:53.032237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2,0],"acting":[3,1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694653+0000","last_change":"2024-04-25T01:16:22.631511+0000","last_active":"2024-04-25T01:16:27.694653+0000","last_peered":"2024-04-25T01:16:27.694653+0000","last_clean":"2024-04-25T01:16:27.694653+0000","last_became_active":"2024-04-25T01:16:22.631169+0000","last_became_peered":"2024-04-25T01:16:22.631169+0000","last_unstale":"2024-04-25T01:16:27.694653+0000","last_undegraded":"2024-04-25T01:16:27.694653+0000","last_fullsized":"2024-04-25T01:16:27.694653+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:54.550997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.089006+0000","last_change":"2024-04-25T01:16:21.621474+0000","last_active":"2024-04-25T01:16:30.089006+0000","last_peered":"2024-04-25T01:16:30.089006+0000","last_clean":"2024-04-25T01:16:30.089006+0000","last_became_active":"2024-04-25T01:16:21.621139+0000","last_became_peered":"2024-04-25T01:16:21.621139+0000","last_unstale":"2024-04-25T01:16:30.089006+0000","last_undegraded":"2024-04-25T01:16:30.089006+0000","last_fullsized":"2024-04-25T01:16:30.089006+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:35:01.136892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.875567+0000","last_change":"2024-04-25T01:16:26.669916+0000","last_active":"2024-04-25T01:16:27.875567+0000","last_peered":"2024-04-25T01:16:27.875567+0000","last_clean":"2024-04-25T01:16:27.875567+0000","last_became_active":"2024-04-25T01:16:26.668054+0000","last_became_peered":"2024-04-25T01:16:26.668054+0000","last_unstale":"2024-04-25T01:16:27.875567+0000","last_undegraded":"2024-04-25T01:16:27.875567+0000","last_fullsized":"2024-04-25T01:16:27.875567+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:58:10.131890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7,4],"acting":[2,3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774586+0000","last_change":"2024-04-25T01:16:22.617183+0000","last_active":"2024-04-25T01:16:29.774586+0000","last_peered":"2024-04-25T01:16:29.774586+0000","last_clean":"2024-04-25T01:16:29.774586+0000","last_became_active":"2024-04-25T01:16:22.615748+0000","last_became_peered":"2024-04-25T01:16:22.615748+0000","last_unstale":"2024-04-25T01:16:29.774586+0000","last_undegraded":"2024-04-25T01:16:29.774586+0000","last_fullsized":"2024-04-25T01:16:29.774586+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:41:00.459225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699060+0000","last_change":"2024-04-25T01:16:21.617685+0000","last_active":"2024-04-25T01:16:27.699060+0000","last_peered":"2024-04-25T01:16:27.699060+0000","last_clean":"2024-04-25T01:16:27.699060+0000","last_became_active":"2024-04-25T01:16:21.617209+0000","last_became_peered":"2024-04-25T01:16:21.617209+0000","last_unstale":"2024-04-25T01:16:27.699060+0000","last_undegraded":"2024-04-25T01:16:27.699060+0000","last_fullsized":"2024-04-25T01:16:27.699060+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:34:39.714953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696564+0000","last_change":"2024-04-25T01:16:26.661702+0000","last_active":"2024-04-25T01:16:27.696564+0000","last_peered":"2024-04-25T01:16:27.696564+0000","last_clean":"2024-04-25T01:16:27.696564+0000","last_became_active":"2024-04-25T01:16:26.660914+0000","last_became_peered":"2024-04-25T01:16:26.660914+0000","last_unstale":"2024-04-25T01:16:27.696564+0000","last_undegraded":"2024-04-25T01:16:27.696564+0000","last_fullsized":"2024-04-25T01:16:27.696564+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:04:40.070035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1,3],"acting":[5,4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774509+0000","last_change":"2024-04-25T01:16:22.626033+0000","last_active":"2024-04-25T01:16:29.774509+0000","last_peered":"2024-04-25T01:16:29.774509+0000","last_clean":"2024-04-25T01:16:29.774509+0000","last_became_active":"2024-04-25T01:16:22.625646+0000","last_became_peered":"2024-04-25T01:16:22.625646+0000","last_unstale":"2024-04-25T01:16:29.774509+0000","last_undegraded":"2024-04-25T01:16:29.774509+0000","last_fullsized":"2024-04-25T01:16:29.774509+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:28.261961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.697212+0000","last_change":"2024-04-25T01:16:21.614628+0000","last_active":"2024-04-25T01:16:27.697212+0000","last_peered":"2024-04-25T01:16:27.697212+0000","last_clean":"2024-04-25T01:16:27.697212+0000","last_became_active":"2024-04-25T01:16:21.614110+0000","last_became_peered":"2024-04-25T01:16:21.614110+0000","last_unstale":"2024-04-25T01:16:27.697212+0000","last_undegraded":"2024-04-25T01:16:27.697212+0000","last_fullsized":"2024-04-25T01:16:27.697212+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:44.675184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.693414+0000","last_change":"2024-04-25T01:16:26.652814+0000","last_active":"2024-04-25T01:16:27.693414+0000","last_peered":"2024-04-25T01:16:27.693414+0000","last_clean":"2024-04-25T01:16:27.693414+0000","last_became_active":"2024-04-25T01:16:26.652030+0000","last_became_peered":"2024-04-25T01:16:26.652030+0000","last_unstale":"2024-04-25T01:16:27.693414+0000","last_undegraded":"2024-04-25T01:16:27.693414+0000","last_fullsized":"2024-04-25T01:16:27.693414+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:22.783625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.876518+0000","last_change":"2024-04-25T01:16:22.617280+0000","last_active":"2024-04-25T01:16:27.876518+0000","last_peered":"2024-04-25T01:16:27.876518+0000","last_clean":"2024-04-25T01:16:27.876518+0000","last_became_active":"2024-04-25T01:16:22.616985+0000","last_became_peered":"2024-04-25T01:16:22.616985+0000","last_unstale":"2024-04-25T01:16:27.876518+0000","last_undegraded":"2024-04-25T01:16:27.876518+0000","last_fullsized":"2024-04-25T01:16:27.876518+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:17:16.335206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.772458+0000","last_change":"2024-04-25T01:16:21.612374+0000","last_active":"2024-04-25T01:16:29.772458+0000","last_peered":"2024-04-25T01:16:29.772458+0000","last_clean":"2024-04-25T01:16:29.772458+0000","last_became_active":"2024-04-25T01:16:21.610765+0000","last_became_peered":"2024-04-25T01:16:21.610765+0000","last_unstale":"2024-04-25T01:16:29.772458+0000","last_undegraded":"2024-04-25T01:16:29.772458+0000","last_fullsized":"2024-04-25T01:16:29.772458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:17:59.668250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.971020+0000","last_change":"2024-04-25T01:16:26.655048+0000","last_active":"2024-04-25T01:16:27.971020+0000","last_peered":"2024-04-25T01:16:27.971020+0000","last_clean":"2024-04-25T01:16:27.971020+0000","last_became_active":"2024-04-25T01:16:26.654534+0000","last_became_peered":"2024-04-25T01:16:26.654534+0000","last_unstale":"2024-04-25T01:16:27.971020+0000","last_undegraded":"2024-04-25T01:16:27.971020+0000","last_fullsized":"2024-04-25T01:16:27.971020+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:17:20.386037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,2],"acting":[1,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.693580+0000","last_change":"2024-04-25T01:16:22.621853+0000","last_active":"2024-04-25T01:16:27.693580+0000","last_peered":"2024-04-25T01:16:27.693580+0000","last_clean":"2024-04-25T01:16:27.693580+0000","last_became_active":"2024-04-25T01:16:22.621537+0000","last_became_peered":"2024-04-25T01:16:22.621537+0000","last_unstale":"2024-04-25T01:16:27.693580+0000","last_undegraded":"2024-04-25T01:16:27.693580+0000","last_fullsized":"2024-04-25T01:16:27.693580+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:01.512241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694812+0000","last_change":"2024-04-25T01:16:21.607981+0000","last_active":"2024-04-25T01:16:27.694812+0000","last_peered":"2024-04-25T01:16:27.694812+0000","last_clean":"2024-04-25T01:16:27.694812+0000","last_became_active":"2024-04-25T01:16:21.607619+0000","last_became_peered":"2024-04-25T01:16:21.607619+0000","last_unstale":"2024-04-25T01:16:27.694812+0000","last_undegraded":"2024-04-25T01:16:27.694812+0000","last_fullsized":"2024-04-25T01:16:27.694812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:48:41.491267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699998+0000","last_change":"2024-04-25T01:16:26.664351+0000","last_active":"2024-04-25T01:16:27.699998+0000","last_peered":"2024-04-25T01:16:27.699998+0000","last_clean":"2024-04-25T01:16:27.699998+0000","last_became_active":"2024-04-25T01:16:26.663469+0000","last_became_peered":"2024-04-25T01:16:26.663469+0000","last_unstale":"2024-04-25T01:16:27.699998+0000","last_undegraded":"2024-04-25T01:16:27.699998+0000","last_fullsized":"2024-04-25T01:16:27.699998+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:10.152413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1,6],"acting":[5,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.970542+0000","last_change":"2024-04-25T01:16:22.609467+0000","last_active":"2024-04-25T01:16:27.970542+0000","last_peered":"2024-04-25T01:16:27.970542+0000","last_clean":"2024-04-25T01:16:27.970542+0000","last_became_active":"2024-04-25T01:16:22.609021+0000","last_became_peered":"2024-04-25T01:16:22.609021+0000","last_unstale":"2024-04-25T01:16:27.970542+0000","last_undegraded":"2024-04-25T01:16:27.970542+0000","last_fullsized":"2024-04-25T01:16:27.970542+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:45.689780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.772174+0000","last_change":"2024-04-25T01:16:21.618665+0000","last_active":"2024-04-25T01:16:29.772174+0000","last_peered":"2024-04-25T01:16:29.772174+0000","last_clean":"2024-04-25T01:16:29.772174+0000","last_became_active":"2024-04-25T01:16:21.618387+0000","last_became_peered":"2024-04-25T01:16:21.618387+0000","last_unstale":"2024-04-25T01:16:29.772174+0000","last_undegraded":"2024-04-25T01:16:29.772174+0000","last_fullsized":"2024-04-25T01:16:29.772174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:50.882927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699704+0000","last_change":"2024-04-25T01:16:26.676113+0000","last_active":"2024-04-25T01:16:27.699704+0000","last_peered":"2024-04-25T01:16:27.699704+0000","last_clean":"2024-04-25T01:16:27.699704+0000","last_became_active":"2024-04-25T01:16:26.675447+0000","last_became_peered":"2024-04-25T01:16:26.675447+0000","last_unstale":"2024-04-25T01:16:27.699704+0000","last_undegraded":"2024-04-25T01:16:27.699704+0000","last_fullsized":"2024-04-25T01:16:27.699704+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:07:54.641757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6,4],"acting":[5,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694284+0000","last_change":"2024-04-25T01:16:22.633079+0000","last_active":"2024-04-25T01:16:27.694284+0000","last_peered":"2024-04-25T01:16:27.694284+0000","last_clean":"2024-04-25T01:16:27.694284+0000","last_became_active":"2024-04-25T01:16:22.632813+0000","last_became_peered":"2024-04-25T01:16:22.632813+0000","last_unstale":"2024-04-25T01:16:27.694284+0000","last_undegraded":"2024-04-25T01:16:27.694284+0000","last_fullsized":"2024-04-25T01:16:27.694284+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:45:08.158420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695843+0000","last_change":"2024-04-25T01:16:21.610362+0000","last_active":"2024-04-25T01:16:27.695843+0000","last_peered":"2024-04-25T01:16:27.695843+0000","last_clean":"2024-04-25T01:16:27.695843+0000","last_became_active":"2024-04-25T01:16:21.608581+0000","last_became_peered":"2024-04-25T01:16:21.608581+0000","last_unstale":"2024-04-25T01:16:27.695843+0000","last_undegraded":"2024-04-25T01:16:27.695843+0000","last_fullsized":"2024-04-25T01:16:27.695843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:17:59.668250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.088195+0000","last_change":"2024-04-25T01:16:26.678631+0000","last_active":"2024-04-25T01:16:30.088195+0000","last_peered":"2024-04-25T01:16:30.088195+0000","last_clean":"2024-04-25T01:16:30.088195+0000","last_became_active":"2024-04-25T01:16:26.676876+0000","last_became_peered":"2024-04-25T01:16:26.676876+0000","last_unstale":"2024-04-25T01:16:30.088195+0000","last_undegraded":"2024-04-25T01:16:30.088195+0000","last_fullsized":"2024-04-25T01:16:30.088195+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:03:57.927312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0,7],"acting":[6,4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.087763+0000","last_change":"2024-04-25T01:16:22.627367+0000","last_active":"2024-04-25T01:16:30.087763+0000","last_peered":"2024-04-25T01:16:30.087763+0000","last_clean":"2024-04-25T01:16:30.087763+0000","last_became_active":"2024-04-25T01:16:22.626338+0000","last_became_peered":"2024-04-25T01:16:22.626338+0000","last_unstale":"2024-04-25T01:16:30.087763+0000","last_undegraded":"2024-04-25T01:16:30.087763+0000","last_fullsized":"2024-04-25T01:16:30.087763+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:50:15.794188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.224430+0000","last_change":"2024-04-25T01:16:21.613871+0000","last_active":"2024-04-25T01:16:28.224430+0000","last_peered":"2024-04-25T01:16:28.224430+0000","last_clean":"2024-04-25T01:16:28.224430+0000","last_became_active":"2024-04-25T01:16:21.613496+0000","last_became_peered":"2024-04-25T01:16:21.613496+0000","last_unstale":"2024-04-25T01:16:28.224430+0000","last_undegraded":"2024-04-25T01:16:28.224430+0000","last_fullsized":"2024-04-25T01:16:28.224430+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:37:47.297259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694895+0000","last_change":"2024-04-25T01:16:26.653921+0000","last_active":"2024-04-25T01:16:27.694895+0000","last_peered":"2024-04-25T01:16:27.694895+0000","last_clean":"2024-04-25T01:16:27.694895+0000","last_became_active":"2024-04-25T01:16:26.653438+0000","last_became_peered":"2024-04-25T01:16:26.653438+0000","last_unstale":"2024-04-25T01:16:27.694895+0000","last_undegraded":"2024-04-25T01:16:27.694895+0000","last_fullsized":"2024-04-25T01:16:27.694895+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:58:40.113666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5,6],"acting":[3,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774620+0000","last_change":"2024-04-25T01:16:22.631703+0000","last_active":"2024-04-25T01:16:29.774620+0000","last_peered":"2024-04-25T01:16:29.774620+0000","last_clean":"2024-04-25T01:16:29.774620+0000","last_became_active":"2024-04-25T01:16:22.631211+0000","last_became_peered":"2024-04-25T01:16:22.631211+0000","last_unstale":"2024-04-25T01:16:29.774620+0000","last_undegraded":"2024-04-25T01:16:29.774620+0000","last_fullsized":"2024-04-25T01:16:29.774620+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:01.512241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"26'10","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:41.501751+0000","last_change":"2024-04-25T01:16:21.614845+0000","last_active":"2024-04-25T01:16:41.501751+0000","last_peered":"2024-04-25T01:16:41.501751+0000","last_clean":"2024-04-25T01:16:41.501751+0000","last_became_active":"2024-04-25T01:16:21.614378+0000","last_became_peered":"2024-04-25T01:16:21.614378+0000","last_unstale":"2024-04-25T01:16:41.501751+0000","last_undegraded":"2024-04-25T01:16:41.501751+0000","last_fullsized":"2024-04-25T01:16:41.501751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:41:30.225989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695086+0000","last_change":"2024-04-25T01:16:26.676618+0000","last_active":"2024-04-25T01:16:27.695086+0000","last_peered":"2024-04-25T01:16:27.695086+0000","last_clean":"2024-04-25T01:16:27.695086+0000","last_became_active":"2024-04-25T01:16:26.667226+0000","last_became_peered":"2024-04-25T01:16:26.667226+0000","last_unstale":"2024-04-25T01:16:27.695086+0000","last_undegraded":"2024-04-25T01:16:27.695086+0000","last_fullsized":"2024-04-25T01:16:27.695086+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:23:19.810360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5,0],"acting":[4,3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.693968+0000","last_change":"2024-04-25T01:16:22.624757+0000","last_active":"2024-04-25T01:16:27.693968+0000","last_peered":"2024-04-25T01:16:27.693968+0000","last_clean":"2024-04-25T01:16:27.693968+0000","last_became_active":"2024-04-25T01:16:22.619992+0000","last_became_peered":"2024-04-25T01:16:22.619992+0000","last_unstale":"2024-04-25T01:16:27.693968+0000","last_undegraded":"2024-04-25T01:16:27.693968+0000","last_fullsized":"2024-04-25T01:16:27.693968+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:26:11.534238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.225000+0000","last_change":"2024-04-25T01:16:21.615700+0000","last_active":"2024-04-25T01:16:28.225000+0000","last_peered":"2024-04-25T01:16:28.225000+0000","last_clean":"2024-04-25T01:16:28.225000+0000","last_became_active":"2024-04-25T01:16:21.615207+0000","last_became_peered":"2024-04-25T01:16:21.615207+0000","last_unstale":"2024-04-25T01:16:28.225000+0000","last_undegraded":"2024-04-25T01:16:28.225000+0000","last_fullsized":"2024-04-25T01:16:28.225000+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:13.115669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.875349+0000","last_change":"2024-04-25T01:16:26.666133+0000","last_active":"2024-04-25T01:16:27.875349+0000","last_peered":"2024-04-25T01:16:27.875349+0000","last_clean":"2024-04-25T01:16:27.875349+0000","last_became_active":"2024-04-25T01:16:26.665359+0000","last_became_peered":"2024-04-25T01:16:26.665359+0000","last_unstale":"2024-04-25T01:16:27.875349+0000","last_undegraded":"2024-04-25T01:16:27.875349+0000","last_fullsized":"2024-04-25T01:16:27.875349+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:46:20.659273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1,7],"acting":[2,3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699852+0000","last_change":"2024-04-25T01:16:22.626495+0000","last_active":"2024-04-25T01:16:27.699852+0000","last_peered":"2024-04-25T01:16:27.699852+0000","last_clean":"2024-04-25T01:16:27.699852+0000","last_became_active":"2024-04-25T01:16:22.625913+0000","last_became_peered":"2024-04-25T01:16:22.625913+0000","last_unstale":"2024-04-25T01:16:27.699852+0000","last_undegraded":"2024-04-25T01:16:27.699852+0000","last_fullsized":"2024-04-25T01:16:27.699852+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:36.984987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.772343+0000","last_change":"2024-04-25T01:16:21.611655+0000","last_active":"2024-04-25T01:16:29.772343+0000","last_peered":"2024-04-25T01:16:29.772343+0000","last_clean":"2024-04-25T01:16:29.772343+0000","last_became_active":"2024-04-25T01:16:21.611388+0000","last_became_peered":"2024-04-25T01:16:21.611388+0000","last_unstale":"2024-04-25T01:16:29.772343+0000","last_undegraded":"2024-04-25T01:16:29.772343+0000","last_fullsized":"2024-04-25T01:16:29.772343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:39:20.330039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.223876+0000","last_change":"2024-04-25T01:16:26.668770+0000","last_active":"2024-04-25T01:16:28.223876+0000","last_peered":"2024-04-25T01:16:28.223876+0000","last_clean":"2024-04-25T01:16:28.223876+0000","last_became_active":"2024-04-25T01:16:26.668309+0000","last_became_peered":"2024-04-25T01:16:26.668309+0000","last_unstale":"2024-04-25T01:16:28.223876+0000","last_undegraded":"2024-04-25T01:16:28.223876+0000","last_fullsized":"2024-04-25T01:16:28.223876+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:17:20.386037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3,2],"acting":[7,1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.876399+0000","last_change":"2024-04-25T01:16:22.632717+0000","last_active":"2024-04-25T01:16:27.876399+0000","last_peered":"2024-04-25T01:16:27.876399+0000","last_clean":"2024-04-25T01:16:27.876399+0000","last_became_active":"2024-04-25T01:16:22.630174+0000","last_became_peered":"2024-04-25T01:16:22.630174+0000","last_unstale":"2024-04-25T01:16:27.876399+0000","last_undegraded":"2024-04-25T01:16:27.876399+0000","last_fullsized":"2024-04-25T01:16:27.876399+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:52.536070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.971972+0000","last_change":"2024-04-25T01:16:21.619013+0000","last_active":"2024-04-25T01:16:27.971972+0000","last_peered":"2024-04-25T01:16:27.971972+0000","last_clean":"2024-04-25T01:16:27.971972+0000","last_became_active":"2024-04-25T01:16:21.615903+0000","last_became_peered":"2024-04-25T01:16:21.615903+0000","last_unstale":"2024-04-25T01:16:27.971972+0000","last_undegraded":"2024-04-25T01:16:27.971972+0000","last_fullsized":"2024-04-25T01:16:27.971972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:37:47.297259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695313+0000","last_change":"2024-04-25T01:16:26.673940+0000","last_active":"2024-04-25T01:16:27.695313+0000","last_peered":"2024-04-25T01:16:27.695313+0000","last_clean":"2024-04-25T01:16:27.695313+0000","last_became_active":"2024-04-25T01:16:26.667111+0000","last_became_peered":"2024-04-25T01:16:26.667111+0000","last_unstale":"2024-04-25T01:16:27.695313+0000","last_undegraded":"2024-04-25T01:16:27.695313+0000","last_fullsized":"2024-04-25T01:16:27.695313+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:32.312792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3,7],"acting":[4,0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696886+0000","last_change":"2024-04-25T01:16:26.679002+0000","last_active":"2024-04-25T01:16:27.696886+0000","last_peered":"2024-04-25T01:16:27.696886+0000","last_clean":"2024-04-25T01:16:27.696886+0000","last_became_active":"2024-04-25T01:16:26.678398+0000","last_became_peered":"2024-04-25T01:16:26.678398+0000","last_unstale":"2024-04-25T01:16:27.696886+0000","last_undegraded":"2024-04-25T01:16:27.696886+0000","last_fullsized":"2024-04-25T01:16:27.696886+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:45:12.209250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5,3],"acting":[4,1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695447+0000","last_change":"2024-04-25T01:16:22.619871+0000","last_active":"2024-04-25T01:16:27.695447+0000","last_peered":"2024-04-25T01:16:27.695447+0000","last_clean":"2024-04-25T01:16:27.695447+0000","last_became_active":"2024-04-25T01:16:22.619553+0000","last_became_peered":"2024-04-25T01:16:22.619553+0000","last_unstale":"2024-04-25T01:16:27.695447+0000","last_undegraded":"2024-04-25T01:16:27.695447+0000","last_fullsized":"2024-04-25T01:16:27.695447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:46:33.270057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694865+0000","last_change":"2024-04-25T01:16:21.625865+0000","last_active":"2024-04-25T01:16:27.694865+0000","last_peered":"2024-04-25T01:16:27.694865+0000","last_clean":"2024-04-25T01:16:27.694865+0000","last_became_active":"2024-04-25T01:16:21.625566+0000","last_became_peered":"2024-04-25T01:16:21.625566+0000","last_unstale":"2024-04-25T01:16:27.694865+0000","last_undegraded":"2024-04-25T01:16:27.694865+0000","last_fullsized":"2024-04-25T01:16:27.694865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:44.675184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695078+0000","last_change":"2024-04-25T01:16:26.650705+0000","last_active":"2024-04-25T01:16:27.695078+0000","last_peered":"2024-04-25T01:16:27.695078+0000","last_clean":"2024-04-25T01:16:27.695078+0000","last_became_active":"2024-04-25T01:16:26.650332+0000","last_became_peered":"2024-04-25T01:16:26.650332+0000","last_unstale":"2024-04-25T01:16:27.695078+0000","last_undegraded":"2024-04-25T01:16:27.695078+0000","last_fullsized":"2024-04-25T01:16:27.695078+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:47:34.491567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,1],"acting":[3,2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694898+0000","last_change":"2024-04-25T01:16:22.622105+0000","last_active":"2024-04-25T01:16:27.694898+0000","last_peered":"2024-04-25T01:16:27.694898+0000","last_clean":"2024-04-25T01:16:27.694898+0000","last_became_active":"2024-04-25T01:16:22.621798+0000","last_became_peered":"2024-04-25T01:16:22.621798+0000","last_unstale":"2024-04-25T01:16:27.694898+0000","last_undegraded":"2024-04-25T01:16:27.694898+0000","last_fullsized":"2024-04-25T01:16:27.694898+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:47.769440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.087421+0000","last_change":"2024-04-25T01:16:21.621471+0000","last_active":"2024-04-25T01:16:30.087421+0000","last_peered":"2024-04-25T01:16:30.087421+0000","last_clean":"2024-04-25T01:16:30.087421+0000","last_became_active":"2024-04-25T01:16:21.621210+0000","last_became_peered":"2024-04-25T01:16:21.621210+0000","last_unstale":"2024-04-25T01:16:30.087421+0000","last_undegraded":"2024-04-25T01:16:30.087421+0000","last_fullsized":"2024-04-25T01:16:30.087421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:34.732380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695224+0000","last_change":"2024-04-25T01:16:21.619390+0000","last_active":"2024-04-25T01:16:27.695224+0000","last_peered":"2024-04-25T01:16:27.695224+0000","last_clean":"2024-04-25T01:16:27.695224+0000","last_became_active":"2024-04-25T01:16:21.619110+0000","last_became_peered":"2024-04-25T01:16:21.619110+0000","last_unstale":"2024-04-25T01:16:27.695224+0000","last_undegraded":"2024-04-25T01:16:27.695224+0000","last_fullsized":"2024-04-25T01:16:27.695224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:50.882927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696696+0000","last_change":"2024-04-25T01:16:22.638518+0000","last_active":"2024-04-25T01:16:27.696696+0000","last_peered":"2024-04-25T01:16:27.696696+0000","last_clean":"2024-04-25T01:16:27.696696+0000","last_became_active":"2024-04-25T01:16:22.638204+0000","last_became_peered":"2024-04-25T01:16:22.638204+0000","last_unstale":"2024-04-25T01:16:27.696696+0000","last_undegraded":"2024-04-25T01:16:27.696696+0000","last_fullsized":"2024-04-25T01:16:27.696696+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:06:58.102648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.697794+0000","last_change":"2024-04-25T01:16:26.673686+0000","last_active":"2024-04-25T01:16:27.697794+0000","last_peered":"2024-04-25T01:16:27.697794+0000","last_clean":"2024-04-25T01:16:27.697794+0000","last_became_active":"2024-04-25T01:16:26.673316+0000","last_became_peered":"2024-04-25T01:16:26.673316+0000","last_unstale":"2024-04-25T01:16:27.697794+0000","last_undegraded":"2024-04-25T01:16:27.697794+0000","last_fullsized":"2024-04-25T01:16:27.697794+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:26:15.585069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,7],"acting":[4,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696890+0000","last_change":"2024-04-25T01:16:21.623660+0000","last_active":"2024-04-25T01:16:27.696890+0000","last_peered":"2024-04-25T01:16:27.696890+0000","last_clean":"2024-04-25T01:16:27.696890+0000","last_became_active":"2024-04-25T01:16:21.623239+0000","last_became_peered":"2024-04-25T01:16:21.623239+0000","last_unstale":"2024-04-25T01:16:27.696890+0000","last_undegraded":"2024-04-25T01:16:27.696890+0000","last_fullsized":"2024-04-25T01:16:27.696890+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:10:00.414303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.970886+0000","last_change":"2024-04-25T01:16:22.606098+0000","last_active":"2024-04-25T01:16:27.970886+0000","last_peered":"2024-04-25T01:16:27.970886+0000","last_clean":"2024-04-25T01:16:27.970886+0000","last_became_active":"2024-04-25T01:16:22.605632+0000","last_became_peered":"2024-04-25T01:16:22.605632+0000","last_unstale":"2024-04-25T01:16:27.970886+0000","last_undegraded":"2024-04-25T01:16:27.970886+0000","last_fullsized":"2024-04-25T01:16:27.970886+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:18:00.682845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.224854+0000","last_change":"2024-04-25T01:16:26.667852+0000","last_active":"2024-04-25T01:16:28.224854+0000","last_peered":"2024-04-25T01:16:28.224854+0000","last_clean":"2024-04-25T01:16:28.224854+0000","last_became_active":"2024-04-25T01:16:26.667070+0000","last_became_peered":"2024-04-25T01:16:26.667070+0000","last_unstale":"2024-04-25T01:16:28.224854+0000","last_undegraded":"2024-04-25T01:16:28.224854+0000","last_fullsized":"2024-04-25T01:16:28.224854+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:35:37.472927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,0],"acting":[7,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.874985+0000","last_change":"2024-04-25T01:16:22.621505+0000","last_active":"2024-04-25T01:16:27.874985+0000","last_peered":"2024-04-25T01:16:27.874985+0000","last_clean":"2024-04-25T01:16:27.874985+0000","last_became_active":"2024-04-25T01:16:22.621239+0000","last_became_peered":"2024-04-25T01:16:22.621239+0000","last_unstale":"2024-04-25T01:16:27.874985+0000","last_undegraded":"2024-04-25T01:16:27.874985+0000","last_fullsized":"2024-04-25T01:16:27.874985+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:18:00.682845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.223851+0000","last_change":"2024-04-25T01:16:22.619015+0000","last_active":"2024-04-25T01:16:28.223851+0000","last_peered":"2024-04-25T01:16:28.223851+0000","last_clean":"2024-04-25T01:16:28.223851+0000","last_became_active":"2024-04-25T01:16:22.618739+0000","last_became_peered":"2024-04-25T01:16:22.618739+0000","last_unstale":"2024-04-25T01:16:28.223851+0000","last_undegraded":"2024-04-25T01:16:28.223851+0000","last_fullsized":"2024-04-25T01:16:28.223851+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:07.954692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694435+0000","last_change":"2024-04-25T01:16:22.621371+0000","last_active":"2024-04-25T01:16:27.694435+0000","last_peered":"2024-04-25T01:16:27.694435+0000","last_clean":"2024-04-25T01:16:27.694435+0000","last_became_active":"2024-04-25T01:16:22.621094+0000","last_became_peered":"2024-04-25T01:16:22.621094+0000","last_unstale":"2024-04-25T01:16:27.694435+0000","last_undegraded":"2024-04-25T01:16:27.694435+0000","last_fullsized":"2024-04-25T01:16:27.694435+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:23:15.759529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696710+0000","last_change":"2024-04-25T01:16:22.613471+0000","last_active":"2024-04-25T01:16:27.696710+0000","last_peered":"2024-04-25T01:16:27.696710+0000","last_clean":"2024-04-25T01:16:27.696710+0000","last_became_active":"2024-04-25T01:16:22.613036+0000","last_became_peered":"2024-04-25T01:16:22.613036+0000","last_unstale":"2024-04-25T01:16:27.696710+0000","last_undegraded":"2024-04-25T01:16:27.696710+0000","last_fullsized":"2024-04-25T01:16:27.696710+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:24:38.413314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.088877+0000","last_change":"2024-04-25T01:16:22.634780+0000","last_active":"2024-04-25T01:16:30.088877+0000","last_peered":"2024-04-25T01:16:30.088877+0000","last_clean":"2024-04-25T01:16:30.088877+0000","last_became_active":"2024-04-25T01:16:22.634412+0000","last_became_peered":"2024-04-25T01:16:22.634412+0000","last_unstale":"2024-04-25T01:16:30.088877+0000","last_undegraded":"2024-04-25T01:16:30.088877+0000","last_fullsized":"2024-04-25T01:16:30.088877+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:18:36.768908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.088852+0000","last_change":"2024-04-25T01:16:22.629001+0000","last_active":"2024-04-25T01:16:30.088852+0000","last_peered":"2024-04-25T01:16:30.088852+0000","last_clean":"2024-04-25T01:16:30.088852+0000","last_became_active":"2024-04-25T01:16:22.626649+0000","last_became_peered":"2024-04-25T01:16:22.626649+0000","last_unstale":"2024-04-25T01:16:30.088852+0000","last_undegraded":"2024-04-25T01:16:30.088852+0000","last_fullsized":"2024-04-25T01:16:30.088852+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:27:44.210712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.876273+0000","last_change":"2024-04-25T01:16:22.633405+0000","last_active":"2024-04-25T01:16:27.876273+0000","last_peered":"2024-04-25T01:16:27.876273+0000","last_clean":"2024-04-25T01:16:27.876273+0000","last_became_active":"2024-04-25T01:16:22.632539+0000","last_became_peered":"2024-04-25T01:16:22.632539+0000","last_unstale":"2024-04-25T01:16:27.876273+0000","last_undegraded":"2024-04-25T01:16:27.876273+0000","last_fullsized":"2024-04-25T01:16:27.876273+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:38:09.671586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699448+0000","last_change":"2024-04-25T01:16:22.615235+0000","last_active":"2024-04-25T01:16:27.699448+0000","last_peered":"2024-04-25T01:16:27.699448+0000","last_clean":"2024-04-25T01:16:27.699448+0000","last_became_active":"2024-04-25T01:16:22.614864+0000","last_became_peered":"2024-04-25T01:16:22.614864+0000","last_unstale":"2024-04-25T01:16:27.699448+0000","last_undegraded":"2024-04-25T01:16:27.699448+0000","last_fullsized":"2024-04-25T01:16:27.699448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:39:14.902973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.693959+0000","last_change":"2024-04-25T01:16:22.615333+0000","last_active":"2024-04-25T01:16:27.693959+0000","last_peered":"2024-04-25T01:16:27.693959+0000","last_clean":"2024-04-25T01:16:27.693959+0000","last_became_active":"2024-04-25T01:16:22.614510+0000","last_became_peered":"2024-04-25T01:16:22.614510+0000","last_unstale":"2024-04-25T01:16:27.693959+0000","last_undegraded":"2024-04-25T01:16:27.693959+0000","last_fullsized":"2024-04-25T01:16:27.693959+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:58:06.081060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696037+0000","last_change":"2024-04-25T01:16:22.615852+0000","last_active":"2024-04-25T01:16:27.696037+0000","last_peered":"2024-04-25T01:16:27.696037+0000","last_clean":"2024-04-25T01:16:27.696037+0000","last_became_active":"2024-04-25T01:16:22.615517+0000","last_became_peered":"2024-04-25T01:16:22.615517+0000","last_unstale":"2024-04-25T01:16:27.696037+0000","last_undegraded":"2024-04-25T01:16:27.696037+0000","last_fullsized":"2024-04-25T01:16:27.696037+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:31.459176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.970579+0000","last_change":"2024-04-25T01:16:22.627717+0000","last_active":"2024-04-25T01:16:27.970579+0000","last_peered":"2024-04-25T01:16:27.970579+0000","last_clean":"2024-04-25T01:16:27.970579+0000","last_became_active":"2024-04-25T01:16:22.627359+0000","last_became_peered":"2024-04-25T01:16:22.627359+0000","last_unstale":"2024-04-25T01:16:27.970579+0000","last_undegraded":"2024-04-25T01:16:27.970579+0000","last_fullsized":"2024-04-25T01:16:27.970579+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:51.897522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696423+0000","last_change":"2024-04-25T01:16:22.634253+0000","last_active":"2024-04-25T01:16:27.696423+0000","last_peered":"2024-04-25T01:16:27.696423+0000","last_clean":"2024-04-25T01:16:27.696423+0000","last_became_active":"2024-04-25T01:16:22.632643+0000","last_became_peered":"2024-04-25T01:16:22.632643+0000","last_unstale":"2024-04-25T01:16:27.696423+0000","last_undegraded":"2024-04-25T01:16:27.696423+0000","last_fullsized":"2024-04-25T01:16:27.696423+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:14:10.348493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774424+0000","last_change":"2024-04-25T01:16:22.627377+0000","last_active":"2024-04-25T01:16:29.774424+0000","last_peered":"2024-04-25T01:16:29.774424+0000","last_clean":"2024-04-25T01:16:29.774424+0000","last_became_active":"2024-04-25T01:16:22.626141+0000","last_became_peered":"2024-04-25T01:16:22.626141+0000","last_unstale":"2024-04-25T01:16:29.774424+0000","last_undegraded":"2024-04-25T01:16:29.774424+0000","last_fullsized":"2024-04-25T01:16:29.774424+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:58:33.743727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695050+0000","last_change":"2024-04-25T01:16:22.622416+0000","last_active":"2024-04-25T01:16:27.695050+0000","last_peered":"2024-04-25T01:16:27.695050+0000","last_clean":"2024-04-25T01:16:27.695050+0000","last_became_active":"2024-04-25T01:16:22.620963+0000","last_became_peered":"2024-04-25T01:16:22.620963+0000","last_unstale":"2024-04-25T01:16:27.695050+0000","last_undegraded":"2024-04-25T01:16:27.695050+0000","last_fullsized":"2024-04-25T01:16:27.695050+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:58:32.477986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.225310+0000","last_change":"2024-04-25T01:16:22.614902+0000","last_active":"2024-04-25T01:16:28.225310+0000","last_peered":"2024-04-25T01:16:28.225310+0000","last_clean":"2024-04-25T01:16:28.225310+0000","last_became_active":"2024-04-25T01:16:22.614621+0000","last_became_peered":"2024-04-25T01:16:22.614621+0000","last_unstale":"2024-04-25T01:16:28.225310+0000","last_undegraded":"2024-04-25T01:16:28.225310+0000","last_fullsized":"2024-04-25T01:16:28.225310+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:45.689780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695622+0000","last_change":"2024-04-25T01:16:22.637378+0000","last_active":"2024-04-25T01:16:27.695622+0000","last_peered":"2024-04-25T01:16:27.695622+0000","last_clean":"2024-04-25T01:16:27.695622+0000","last_became_active":"2024-04-25T01:16:22.636599+0000","last_became_peered":"2024-04-25T01:16:22.636599+0000","last_unstale":"2024-04-25T01:16:27.695622+0000","last_undegraded":"2024-04-25T01:16:27.695622+0000","last_fullsized":"2024-04-25T01:16:27.695622+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:44:52.172147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.693572+0000","last_change":"2024-04-25T01:16:22.621416+0000","last_active":"2024-04-25T01:16:27.693572+0000","last_peered":"2024-04-25T01:16:27.693572+0000","last_clean":"2024-04-25T01:16:27.693572+0000","last_became_active":"2024-04-25T01:16:22.619180+0000","last_became_peered":"2024-04-25T01:16:22.619180+0000","last_unstale":"2024-04-25T01:16:27.693572+0000","last_undegraded":"2024-04-25T01:16:27.693572+0000","last_fullsized":"2024-04-25T01:16:27.693572+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:46:16.608442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695613+0000","last_change":"2024-04-25T01:16:22.623812+0000","last_active":"2024-04-25T01:16:27.695613+0000","last_peered":"2024-04-25T01:16:27.695613+0000","last_clean":"2024-04-25T01:16:27.695613+0000","last_became_active":"2024-04-25T01:16:22.623468+0000","last_became_peered":"2024-04-25T01:16:22.623468+0000","last_unstale":"2024-04-25T01:16:27.695613+0000","last_undegraded":"2024-04-25T01:16:27.695613+0000","last_fullsized":"2024-04-25T01:16:27.695613+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:07:25.313465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.087421+0000","last_change":"2024-04-25T01:16:22.622280+0000","last_active":"2024-04-25T01:16:30.087421+0000","last_peered":"2024-04-25T01:16:30.087421+0000","last_clean":"2024-04-25T01:16:30.087421+0000","last_became_active":"2024-04-25T01:16:22.621901+0000","last_became_peered":"2024-04-25T01:16:22.621901+0000","last_unstale":"2024-04-25T01:16:30.087421+0000","last_undegraded":"2024-04-25T01:16:30.087421+0000","last_fullsized":"2024-04-25T01:16:30.087421+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:09:42.404928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.087951+0000","last_change":"2024-04-25T01:16:22.635778+0000","last_active":"2024-04-25T01:16:30.087951+0000","last_peered":"2024-04-25T01:16:30.087951+0000","last_clean":"2024-04-25T01:16:30.087951+0000","last_became_active":"2024-04-25T01:16:22.635492+0000","last_became_peered":"2024-04-25T01:16:22.635492+0000","last_unstale":"2024-04-25T01:16:30.087951+0000","last_undegraded":"2024-04-25T01:16:30.087951+0000","last_fullsized":"2024-04-25T01:16:30.087951+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:16:27.803006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774190+0000","last_change":"2024-04-25T01:16:22.627185+0000","last_active":"2024-04-25T01:16:29.774190+0000","last_peered":"2024-04-25T01:16:29.774190+0000","last_clean":"2024-04-25T01:16:29.774190+0000","last_became_active":"2024-04-25T01:16:22.626147+0000","last_became_peered":"2024-04-25T01:16:22.626147+0000","last_unstale":"2024-04-25T01:16:29.774190+0000","last_undegraded":"2024-04-25T01:16:29.774190+0000","last_fullsized":"2024-04-25T01:16:29.774190+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:06:31.257662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774396+0000","last_change":"2024-04-25T01:16:22.627705+0000","last_active":"2024-04-25T01:16:29.774396+0000","last_peered":"2024-04-25T01:16:29.774396+0000","last_clean":"2024-04-25T01:16:29.774396+0000","last_became_active":"2024-04-25T01:16:22.627424+0000","last_became_peered":"2024-04-25T01:16:22.627424+0000","last_unstale":"2024-04-25T01:16:29.774396+0000","last_undegraded":"2024-04-25T01:16:29.774396+0000","last_fullsized":"2024-04-25T01:16:29.774396+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:58:06.081060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774278+0000","last_change":"2024-04-25T01:16:22.625859+0000","last_active":"2024-04-25T01:16:29.774278+0000","last_peered":"2024-04-25T01:16:29.774278+0000","last_clean":"2024-04-25T01:16:29.774278+0000","last_became_active":"2024-04-25T01:16:22.625512+0000","last_became_peered":"2024-04-25T01:16:22.625512+0000","last_unstale":"2024-04-25T01:16:29.774278+0000","last_undegraded":"2024-04-25T01:16:29.774278+0000","last_fullsized":"2024-04-25T01:16:29.774278+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:09.176572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.971952+0000","last_change":"2024-04-25T01:16:22.622906+0000","last_active":"2024-04-25T01:16:27.971952+0000","last_peered":"2024-04-25T01:16:27.971952+0000","last_clean":"2024-04-25T01:16:27.971952+0000","last_became_active":"2024-04-25T01:16:22.622619+0000","last_became_peered":"2024-04-25T01:16:22.622619+0000","last_unstale":"2024-04-25T01:16:27.971952+0000","last_undegraded":"2024-04-25T01:16:27.971952+0000","last_fullsized":"2024-04-25T01:16:27.971952+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:20.635412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694511+0000","last_change":"2024-04-25T01:16:22.623806+0000","last_active":"2024-04-25T01:16:27.694511+0000","last_peered":"2024-04-25T01:16:27.694511+0000","last_clean":"2024-04-25T01:16:27.694511+0000","last_became_active":"2024-04-25T01:16:22.623165+0000","last_became_peered":"2024-04-25T01:16:22.623165+0000","last_unstale":"2024-04-25T01:16:27.694511+0000","last_undegraded":"2024-04-25T01:16:27.694511+0000","last_fullsized":"2024-04-25T01:16:27.694511+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:01:54.012020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.773234+0000","last_change":"2024-04-25T01:16:22.621330+0000","last_active":"2024-04-25T01:16:29.773234+0000","last_peered":"2024-04-25T01:16:29.773234+0000","last_clean":"2024-04-25T01:16:29.773234+0000","last_became_active":"2024-04-25T01:16:22.620993+0000","last_became_peered":"2024-04-25T01:16:22.620993+0000","last_unstale":"2024-04-25T01:16:29.773234+0000","last_undegraded":"2024-04-25T01:16:29.773234+0000","last_fullsized":"2024-04-25T01:16:29.773234+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:53.625025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.875525+0000","last_change":"2024-04-25T01:16:22.623636+0000","last_active":"2024-04-25T01:16:27.875525+0000","last_peered":"2024-04-25T01:16:27.875525+0000","last_clean":"2024-04-25T01:16:27.875525+0000","last_became_active":"2024-04-25T01:16:22.623365+0000","last_became_peered":"2024-04-25T01:16:22.623365+0000","last_unstale":"2024-04-25T01:16:27.875525+0000","last_undegraded":"2024-04-25T01:16:27.875525+0000","last_fullsized":"2024-04-25T01:16:27.875525+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:49:05.483350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.773871+0000","last_change":"2024-04-25T01:16:22.616635+0000","last_active":"2024-04-25T01:16:29.773871+0000","last_peered":"2024-04-25T01:16:29.773871+0000","last_clean":"2024-04-25T01:16:29.773871+0000","last_became_active":"2024-04-25T01:16:22.616342+0000","last_became_peered":"2024-04-25T01:16:22.616342+0000","last_unstale":"2024-04-25T01:16:29.773871+0000","last_undegraded":"2024-04-25T01:16:29.773871+0000","last_fullsized":"2024-04-25T01:16:29.773871+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:09:07.843928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.875763+0000","last_change":"2024-04-25T01:16:22.623875+0000","last_active":"2024-04-25T01:16:27.875763+0000","last_peered":"2024-04-25T01:16:27.875763+0000","last_clean":"2024-04-25T01:16:27.875763+0000","last_became_active":"2024-04-25T01:16:22.623524+0000","last_became_peered":"2024-04-25T01:16:22.623524+0000","last_unstale":"2024-04-25T01:16:27.875763+0000","last_undegraded":"2024-04-25T01:16:27.875763+0000","last_fullsized":"2024-04-25T01:16:27.875763+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:07:32.166451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.698853+0000","last_change":"2024-04-25T01:16:22.620609+0000","last_active":"2024-04-25T01:16:27.698853+0000","last_peered":"2024-04-25T01:16:27.698853+0000","last_clean":"2024-04-25T01:16:27.698853+0000","last_became_active":"2024-04-25T01:16:22.620261+0000","last_became_peered":"2024-04-25T01:16:22.620261+0000","last_unstale":"2024-04-25T01:16:27.698853+0000","last_undegraded":"2024-04-25T01:16:27.698853+0000","last_fullsized":"2024-04-25T01:16:27.698853+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:01.416632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699491+0000","last_change":"2024-04-25T01:16:22.633173+0000","last_active":"2024-04-25T01:16:27.699491+0000","last_peered":"2024-04-25T01:16:27.699491+0000","last_clean":"2024-04-25T01:16:27.699491+0000","last_became_active":"2024-04-25T01:16:22.632857+0000","last_became_peered":"2024-04-25T01:16:22.632857+0000","last_unstale":"2024-04-25T01:16:27.699491+0000","last_undegraded":"2024-04-25T01:16:27.699491+0000","last_fullsized":"2024-04-25T01:16:27.699491+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:14.238333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774391+0000","last_change":"2024-04-25T01:16:22.628348+0000","last_active":"2024-04-25T01:16:29.774391+0000","last_peered":"2024-04-25T01:16:29.774391+0000","last_clean":"2024-04-25T01:16:29.774391+0000","last_became_active":"2024-04-25T01:16:22.627597+0000","last_became_peered":"2024-04-25T01:16:22.627597+0000","last_unstale":"2024-04-25T01:16:29.774391+0000","last_undegraded":"2024-04-25T01:16:29.774391+0000","last_fullsized":"2024-04-25T01:16:29.774391+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:23:15.759529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542153,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":108288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542152,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":108164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542153,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":876544,"data_stored":698624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542152,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":294912,"data_stored":112678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542152,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":108746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542154,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":286720,"data_stored":112006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542152,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":872448,"data_stored":698602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542153,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":108220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:16:52.829 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T01:16:52.830 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-25T01:16:52.830 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-25T01:16:52.830 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-25T01:16:52.831 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-25T01:16:53.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.082+0000 7fb6bc000640 1 Processor -- start 2024-04-25T01:16:53.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.082+0000 7fb6bc000640 1 -- start start 2024-04-25T01:16:53.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.083+0000 7fb6bc000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb6b4145590 0x7fb6b40b4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:53.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.083+0000 7fb6bc000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6b415f9b0 0x7fb6b415fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:53.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.084+0000 7fb6bc000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb6b40ce110 0x7fb6b40ce5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:53.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.084+0000 7fb6bc000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fb6b4160830 con 0x7fb6b40ce110 2024-04-25T01:16:53.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.084+0000 7fb6bc000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb6b4160350 con 0x7fb6b4145590 2024-04-25T01:16:53.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.084+0000 7fb6bc000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb6b406cde0 con 0x7fb6b415f9b0 2024-04-25T01:16:53.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.084+0000 7fb6ba000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb6b40ce110 0x7fb6b40ce5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:53.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.084+0000 7fb6ba000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb6b40ce110 0x7fb6b40ce5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33516/0 (socket says 172.21.15.90:33516) 2024-04-25T01:16:53.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.084+0000 7fb6ba000640 1 -- 172.21.15.90:0/3664754426 learned_addr learned my addr 172.21.15.90:0/3664754426 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:53.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.084+0000 7fb6b9600640 1 --2- 172.21.15.90:0/3664754426 >> v2:172.21.15.103:3300/0 conn(0x7fb6b4145590 0x7fb6b40b4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:53.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.084+0000 7fb6b8c00640 1 --2- 172.21.15.90:0/3664754426 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6b415f9b0 0x7fb6b415fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:53.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.085+0000 7fb6ba000640 1 -- 172.21.15.90:0/3664754426 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6b415f9b0 msgr2=0x7fb6b415fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:53.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.085+0000 7fb6ba000640 1 --2- 172.21.15.90:0/3664754426 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6b415f9b0 0x7fb6b415fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:53.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.085+0000 7fb6ba000640 1 -- 172.21.15.90:0/3664754426 >> v2:172.21.15.103:3300/0 conn(0x7fb6b4145590 msgr2=0x7fb6b40b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:53.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.085+0000 7fb6ba000640 1 --2- 172.21.15.90:0/3664754426 >> v2:172.21.15.103:3300/0 conn(0x7fb6b4145590 0x7fb6b40b4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:53.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.085+0000 7fb6ba000640 1 -- 172.21.15.90:0/3664754426 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6b40b4240 con 0x7fb6b40ce110 2024-04-25T01:16:53.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.086+0000 7fb6ba000640 1 --2- 172.21.15.90:0/3664754426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6b40ce110 0x7fb6b40ce5a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb6b406c150 tx=0x7fb6ac02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f86276baaf3bd3f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:53.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.087+0000 7fb6b3e00640 1 -- 172.21.15.90:0/3664754426 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ac036070 con 0x7fb6b40ce110 2024-04-25T01:16:53.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.087+0000 7fb6b3e00640 1 -- 172.21.15.90:0/3664754426 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6ac031600 con 0x7fb6b40ce110 2024-04-25T01:16:53.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.087+0000 7fb6b3e00640 1 -- 172.21.15.90:0/3664754426 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ac0319d0 con 0x7fb6b40ce110 2024-04-25T01:16:53.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.087+0000 7fb6bc000640 1 -- 172.21.15.90:0/3664754426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6b40ce110 msgr2=0x7fb6b40ce5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:53.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.087+0000 7fb6bc000640 1 --2- 172.21.15.90:0/3664754426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6b40ce110 0x7fb6b40ce5a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb6b406c150 tx=0x7fb6ac02a6a0 comp rx=0 tx=0).stop 2024-04-25T01:16:53.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.087+0000 7fb6bc000640 1 -- 172.21.15.90:0/3664754426 shutdown_connections 2024-04-25T01:16:53.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.087+0000 7fb6bc000640 1 --2- 172.21.15.90:0/3664754426 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6b415f9b0 0x7fb6b415fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:53.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.087+0000 7fb6bc000640 1 --2- 172.21.15.90:0/3664754426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6b40ce110 0x7fb6b40ce5a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:53.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.087+0000 7fb6bc000640 1 --2- 172.21.15.90:0/3664754426 >> v2:172.21.15.103:3300/0 conn(0x7fb6b4145590 0x7fb6b40b4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:53.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.087+0000 7fb6bc000640 1 -- 172.21.15.90:0/3664754426 >> 172.21.15.90:0/3664754426 conn(0x7fb6b40b1820 msgr2=0x7fb6b40b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:53.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.087+0000 7fb6bc000640 1 -- 172.21.15.90:0/3664754426 shutdown_connections 2024-04-25T01:16:53.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.087+0000 7fb6bc000640 1 -- 172.21.15.90:0/3664754426 wait complete. 2024-04-25T01:16:53.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.089+0000 7fb6bc000640 1 Processor -- start 2024-04-25T01:16:53.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.089+0000 7fb6bc000640 1 -- start start 2024-04-25T01:16:53.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.090+0000 7fb6bc000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6b40ce110 0x7fb6b415d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:53.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.090+0000 7fb6bc000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6b4145590 0x7fb6b413ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:53.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.090+0000 7fb6bc000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6b415f9b0 0x7fb6b40af890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:53.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.090+0000 7fb6bc000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b40b3c80 con 0x7fb6b40ce110 2024-04-25T01:16:53.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.090+0000 7fb6bc000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b40b39a0 con 0x7fb6b4145590 2024-04-25T01:16:53.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.090+0000 7fb6bc000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb6b40b3f60 con 0x7fb6b415f9b0 2024-04-25T01:16:53.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.090+0000 7fb6b9600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6b40ce110 0x7fb6b415d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:53.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.090+0000 7fb6b9600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6b40ce110 0x7fb6b415d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33520/0 (socket says 172.21.15.90:33520) 2024-04-25T01:16:53.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.090+0000 7fb6b9600640 1 -- 172.21.15.90:0/1671948042 learned_addr learned my addr 172.21.15.90:0/1671948042 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:53.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.091+0000 7fb6b9600640 1 -- 172.21.15.90:0/1671948042 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6b415f9b0 msgr2=0x7fb6b40af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:53.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.091+0000 7fb6b9600640 1 --2- 172.21.15.90:0/1671948042 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6b415f9b0 0x7fb6b40af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:53.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.091+0000 7fb6b9600640 1 -- 172.21.15.90:0/1671948042 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6b4145590 msgr2=0x7fb6b413ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:53.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.091+0000 7fb6b8c00640 1 --2- 172.21.15.90:0/1671948042 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6b4145590 0x7fb6b413ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:53.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.091+0000 7fb6b9600640 1 --2- 172.21.15.90:0/1671948042 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6b4145590 0x7fb6b413ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:53.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.091+0000 7fb6b9600640 1 -- 172.21.15.90:0/1671948042 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6b415d570 con 0x7fb6b40ce110 2024-04-25T01:16:53.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.091+0000 7fb6b9600640 1 --2- 172.21.15.90:0/1671948042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6b40ce110 0x7fb6b415d030 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb6a800e890 tx=0x7fb6a800ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:53.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.091+0000 7fb6b1600640 1 -- 172.21.15.90:0/1671948042 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a8017070 con 0x7fb6b40ce110 2024-04-25T01:16:53.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.091+0000 7fb6b1600640 1 -- 172.21.15.90:0/1671948042 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6a800c940 con 0x7fb6b40ce110 2024-04-25T01:16:53.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.091+0000 7fb6b1600640 1 -- 172.21.15.90:0/1671948042 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a800cd10 con 0x7fb6b40ce110 2024-04-25T01:16:53.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.091+0000 7fb6bc000640 1 -- 172.21.15.90:0/1671948042 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6b413f1a0 con 0x7fb6b40ce110 2024-04-25T01:16:53.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.092+0000 7fb6bc000640 1 -- 172.21.15.90:0/1671948042 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6b41f7cf0 con 0x7fb6b40ce110 2024-04-25T01:16:53.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.093+0000 7fb6bc000640 1 -- 172.21.15.90:0/1671948042 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6b4068fb0 con 0x7fb6b40ce110 2024-04-25T01:16:53.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.094+0000 7fb6b1600640 1 -- 172.21.15.90:0/1671948042 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb6a801a070 con 0x7fb6b40ce110 2024-04-25T01:16:53.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.095+0000 7fb6b1600640 1 --2- 172.21.15.90:0/1671948042 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb69406be30 0x7fb69406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:53.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.095+0000 7fb6b1600640 1 -- 172.21.15.90:0/1671948042 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fb6a808b1e0 con 0x7fb6b40ce110 2024-04-25T01:16:53.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.095+0000 7fb6b8c00640 1 --2- 172.21.15.90:0/1671948042 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb69406be30 0x7fb69406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:53.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.096+0000 7fb6b8c00640 1 --2- 172.21.15.90:0/1671948042 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb69406be30 0x7fb69406e2f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb6a0006fd0 tx=0x7fb6a0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:53.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.099+0000 7fb6b1600640 1 -- 172.21.15.90:0/1671948042 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb6b415d570 con 0x7fb6b40ce110 2024-04-25T01:16:53.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.351+0000 7fb6bc000640 1 -- 172.21.15.90:0/1671948042 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fb6b406ea50 con 0x7fb6b40ce110 2024-04-25T01:16:53.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.355+0000 7fb6b1600640 1 -- 172.21.15.90:0/1671948042 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+172938 (secure 0 0 0) 0x7fb6a8059850 con 0x7fb6b40ce110 2024-04-25T01:16:53.341 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:16:53.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.359+0000 7fb6bc000640 1 -- 172.21.15.90:0/1671948042 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb69406be30 msgr2=0x7fb69406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:53.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.359+0000 7fb6bc000640 1 --2- 172.21.15.90:0/1671948042 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb69406be30 0x7fb69406e2f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb6a0006fd0 tx=0x7fb6a0008040 comp rx=0 tx=0).stop 2024-04-25T01:16:53.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.359+0000 7fb6bc000640 1 -- 172.21.15.90:0/1671948042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6b40ce110 msgr2=0x7fb6b415d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:53.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.359+0000 7fb6bc000640 1 --2- 172.21.15.90:0/1671948042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6b40ce110 0x7fb6b415d030 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb6a800e890 tx=0x7fb6a800ed50 comp rx=0 tx=0).stop 2024-04-25T01:16:53.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.359+0000 7fb6bc000640 1 -- 172.21.15.90:0/1671948042 shutdown_connections 2024-04-25T01:16:53.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.359+0000 7fb6bc000640 1 --2- 172.21.15.90:0/1671948042 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb69406be30 0x7fb69406e2f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:53.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.359+0000 7fb6bc000640 1 --2- 172.21.15.90:0/1671948042 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6b415f9b0 0x7fb6b40af890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:53.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.359+0000 7fb6bc000640 1 --2- 172.21.15.90:0/1671948042 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6b4145590 0x7fb6b413ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:53.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.359+0000 7fb6bc000640 1 --2- 172.21.15.90:0/1671948042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6b40ce110 0x7fb6b415d030 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:53.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.359+0000 7fb6bc000640 1 -- 172.21.15.90:0/1671948042 >> 172.21.15.90:0/1671948042 conn(0x7fb6b40b1820 msgr2=0x7fb6b40aecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:53.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.360+0000 7fb6bc000640 1 -- 172.21.15.90:0/1671948042 shutdown_connections 2024-04-25T01:16:53.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.360+0000 7fb6bc000640 1 -- 172.21.15.90:0/1671948042 wait complete. 2024-04-25T01:16:53.361 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":5,"flags":0,"active_gid":4104,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2198086733},{"type":"v1","addr":"172.21.15.90:6833","nonce":2198086733}]},"active_addr":"172.21.15.90:6833/2198086733","active_change":"2024-04-25T01:16:07.798926+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4108,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":2179974398}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":2625288594}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":491912209}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":2002952832}]}]} 2024-04-25T01:16:53.365 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-25T01:16:53.365 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-25T01:16:53.365 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:16:53.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.576+0000 7fefb4a00640 1 Processor -- start 2024-04-25T01:16:53.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.576+0000 7fefb4a00640 1 -- start start 2024-04-25T01:16:53.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.578+0000 7fefb4a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fefb015ca10 0x7fefb015cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:53.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.578+0000 7fefb4a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefb015c220 con 0x7fefb00ae730 2024-04-25T01:16:53.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.578+0000 7fefb4a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefb00aec80 con 0x7fefb006b120 2024-04-25T01:16:53.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.578+0000 7fefb4a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fefb00cef40 con 0x7fefb015ca10 2024-04-25T01:16:53.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.578+0000 7fefaf200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fefb015ca10 0x7fefb015cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:53.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.578+0000 7fefaf200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fefb015ca10 0x7fefb015cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33528/0 (socket says 172.21.15.90:33528) 2024-04-25T01:16:53.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.578+0000 7fefaf200640 1 -- 172.21.15.90:0/2563510779 learned_addr learned my addr 172.21.15.90:0/2563510779 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:53.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.579+0000 7fefaf200640 1 -- 172.21.15.90:0/2563510779 >> v1:172.21.15.103:6789/0 conn(0x7fefb006b120 legacy=0x7fefb006b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:53.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.579+0000 7fefaf200640 1 -- 172.21.15.90:0/2563510779 >> v1:172.21.15.90:6789/0 conn(0x7fefb00ae730 legacy=0x7fefb00cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:53.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.579+0000 7fefaf200640 1 -- 172.21.15.90:0/2563510779 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefb00aef60 con 0x7fefb015ca10 2024-04-25T01:16:53.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.580+0000 7fefaf200640 1 --2- 172.21.15.90:0/2563510779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefb015ca10 0x7fefb015cea0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fefa8009e50 tx=0x7fefa802a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc5e07ff31492ed4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:53.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.580+0000 7fefade00640 1 -- 172.21.15.90:0/2563510779 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa8036070 con 0x7fefb015ca10 2024-04-25T01:16:53.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.580+0000 7fefade00640 1 -- 172.21.15.90:0/2563510779 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefa8031600 con 0x7fefb015ca10 2024-04-25T01:16:53.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.580+0000 7fefade00640 1 -- 172.21.15.90:0/2563510779 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa80319d0 con 0x7fefb015ca10 2024-04-25T01:16:53.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.580+0000 7fefb4a00640 1 -- 172.21.15.90:0/2563510779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefb015ca10 msgr2=0x7fefb015cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:53.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.580+0000 7fefb4a00640 1 --2- 172.21.15.90:0/2563510779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefb015ca10 0x7fefb015cea0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fefa8009e50 tx=0x7fefa802a780 comp rx=0 tx=0).stop 2024-04-25T01:16:53.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.581+0000 7fefb4a00640 1 -- 172.21.15.90:0/2563510779 shutdown_connections 2024-04-25T01:16:53.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.581+0000 7fefb4a00640 1 --2- 172.21.15.90:0/2563510779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefb015ca10 0x7fefb015cea0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:53.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.581+0000 7fefb4a00640 1 -- 172.21.15.90:0/2563510779 >> 172.21.15.90:0/2563510779 conn(0x7fefb00b1660 msgr2=0x7fefb00b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:53.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.581+0000 7fefb4a00640 1 -- 172.21.15.90:0/2563510779 shutdown_connections 2024-04-25T01:16:53.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.581+0000 7fefb4a00640 1 -- 172.21.15.90:0/2563510779 wait complete. 2024-04-25T01:16:53.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.582+0000 7fefb4a00640 1 Processor -- start 2024-04-25T01:16:53.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.582+0000 7fefb4a00640 1 -- start start 2024-04-25T01:16:53.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.583+0000 7fefb4a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefb006b120 0x7fefb01453f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:53.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.583+0000 7fefb4a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefb00ae730 0x7fefb00cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:53.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.583+0000 7fefb4a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fefb015ca10 0x7fefb00ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:53.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.583+0000 7fefb4a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fefb0153080 con 0x7fefb00ae730 2024-04-25T01:16:53.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.583+0000 7fefb4a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fefb015c580 con 0x7fefb006b120 2024-04-25T01:16:53.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.583+0000 7fefb4a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fefb0145930 con 0x7fefb015ca10 2024-04-25T01:16:53.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.584+0000 7fefa7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefb00ae730 0x7fefb00cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:53.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.584+0000 7fefa7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefb00ae730 0x7fefb00cc7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33542/0 (socket says 172.21.15.90:33542) 2024-04-25T01:16:53.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.584+0000 7fefa7e00640 1 -- 172.21.15.90:0/1252205634 learned_addr learned my addr 172.21.15.90:0/1252205634 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:53.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.584+0000 7fefae800640 1 --2- 172.21.15.90:0/1252205634 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefb006b120 0x7fefb01453f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:53.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.584+0000 7fefaf200640 1 --2- 172.21.15.90:0/1252205634 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fefb015ca10 0x7fefb00ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:53.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.584+0000 7fefa7e00640 1 -- 172.21.15.90:0/1252205634 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fefb015ca10 msgr2=0x7fefb00ccd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:53.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.584+0000 7fefa7e00640 1 --2- 172.21.15.90:0/1252205634 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fefb015ca10 0x7fefb00ccd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:53.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.584+0000 7fefa7e00640 1 -- 172.21.15.90:0/1252205634 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefb006b120 msgr2=0x7fefb01453f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:53.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.584+0000 7fefa7e00640 1 --2- 172.21.15.90:0/1252205634 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefb006b120 0x7fefb01453f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:53.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.584+0000 7fefa7e00640 1 -- 172.21.15.90:0/1252205634 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefb0158ba0 con 0x7fefb00ae730 2024-04-25T01:16:53.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.584+0000 7fefa7e00640 1 --2- 172.21.15.90:0/1252205634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefb00ae730 0x7fefb00cc7e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fef9800c400 tx=0x7fef9800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:53.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.585+0000 7fefa6a00640 1 -- 172.21.15.90:0/1252205634 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef980052d0 con 0x7fefb00ae730 2024-04-25T01:16:53.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.585+0000 7fefa6a00640 1 -- 172.21.15.90:0/1252205634 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef98005490 con 0x7fefb00ae730 2024-04-25T01:16:53.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.585+0000 7fefa6a00640 1 -- 172.21.15.90:0/1252205634 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef98007260 con 0x7fefb00ae730 2024-04-25T01:16:53.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.585+0000 7fefb4a00640 1 -- 172.21.15.90:0/1252205634 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefb0158e40 con 0x7fefb00ae730 2024-04-25T01:16:53.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.585+0000 7fefb4a00640 1 -- 172.21.15.90:0/1252205634 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefb0159400 con 0x7fefb00ae730 2024-04-25T01:16:53.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.586+0000 7fefb4a00640 1 -- 172.21.15.90:0/1252205634 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefb0075350 con 0x7fefb00ae730 2024-04-25T01:16:53.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.587+0000 7fefa6a00640 1 -- 172.21.15.90:0/1252205634 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fef98007400 con 0x7fefb00ae730 2024-04-25T01:16:53.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.588+0000 7fefa6a00640 1 --2- 172.21.15.90:0/1252205634 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fef8c06be60 0x7fef8c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:53.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.588+0000 7fefa6a00640 1 -- 172.21.15.90:0/1252205634 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fef9808a6a0 con 0x7fefb00ae730 2024-04-25T01:16:53.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.588+0000 7fefae800640 1 --2- 172.21.15.90:0/1252205634 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fef8c06be60 0x7fef8c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:53.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.589+0000 7fefae800640 1 --2- 172.21.15.90:0/1252205634 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fef8c06be60 0x7fef8c06e320 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fefa0004ea0 tx=0x7fefa0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:53.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.592+0000 7fefa6a00640 1 -- 172.21.15.90:0/1252205634 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fef98058d00 con 0x7fefb00ae730 2024-04-25T01:16:53.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.795+0000 7fefb4a00640 1 -- 172.21.15.90:0/1252205634 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fefb006ccd0 con 0x7fefb00ae730 2024-04-25T01:16:53.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.797+0000 7fefa6a00640 1 -- 172.21.15.90:0/1252205634 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18482 (secure 0 0 0) 0x7fef98058600 con 0x7fefb00ae730 2024-04-25T01:16:53.782 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:16:53.782 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":26,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:16:27.092855+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:16:20.094512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:16:21.078268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:16:24.670642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:16:53.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.800+0000 7fefb4a00640 1 -- 172.21.15.90:0/1252205634 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fef8c06be60 msgr2=0x7fef8c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:53.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.800+0000 7fefb4a00640 1 --2- 172.21.15.90:0/1252205634 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fef8c06be60 0x7fef8c06e320 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fefa0004ea0 tx=0x7fefa0009040 comp rx=0 tx=0).stop 2024-04-25T01:16:53.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.800+0000 7fefb4a00640 1 -- 172.21.15.90:0/1252205634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefb00ae730 msgr2=0x7fefb00cc7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:53.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.800+0000 7fefb4a00640 1 --2- 172.21.15.90:0/1252205634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefb00ae730 0x7fefb00cc7e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fef9800c400 tx=0x7fef9800c8c0 comp rx=0 tx=0).stop 2024-04-25T01:16:53.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.800+0000 7fefb4a00640 1 -- 172.21.15.90:0/1252205634 shutdown_connections 2024-04-25T01:16:53.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.800+0000 7fefb4a00640 1 --2- 172.21.15.90:0/1252205634 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fef8c06be60 0x7fef8c06e320 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:53.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.800+0000 7fefb4a00640 1 --2- 172.21.15.90:0/1252205634 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fefb015ca10 0x7fefb00ccd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:53.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.800+0000 7fefb4a00640 1 --2- 172.21.15.90:0/1252205634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefb00ae730 0x7fefb00cc7e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:53.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.800+0000 7fefb4a00640 1 --2- 172.21.15.90:0/1252205634 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefb006b120 0x7fefb01453f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:53.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.800+0000 7fefb4a00640 1 -- 172.21.15.90:0/1252205634 >> 172.21.15.90:0/1252205634 conn(0x7fefb00b1660 msgr2=0x7fefb0152600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:53.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.800+0000 7fefb4a00640 1 -- 172.21.15.90:0/1252205634 shutdown_connections 2024-04-25T01:16:53.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:53.800+0000 7fefb4a00640 1 -- 172.21.15.90:0/1252205634 wait complete. 2024-04-25T01:16:53.804 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-25T01:16:53.804 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:16:54.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.019+0000 7f5870a00640 1 Processor -- start 2024-04-25T01:16:54.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.019+0000 7f5870a00640 1 -- start start 2024-04-25T01:16:54.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.020+0000 7f5870a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f586c145590 0x7f586c0b4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.020+0000 7f5870a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f586c06e280 0x7f586c15fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.020+0000 7f5870a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f586c0b4240 con 0x7f586c15f9b0 2024-04-25T01:16:54.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.020+0000 7f5870a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f586c160830 con 0x7f586c145590 2024-04-25T01:16:54.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.020+0000 7f5870a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f586c160350 con 0x7f586c06e280 2024-04-25T01:16:54.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.021+0000 7f5869e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f586c145590 0x7f586c0b4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.021+0000 7f5861400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f586c15f9b0 0x7f586c0ce110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60762/0 (socket says 172.21.15.90:60762) 2024-04-25T01:16:54.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.021+0000 7f5869e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f586c145590 0x7f586c0b4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33552/0 (socket says 172.21.15.90:33552) 2024-04-25T01:16:54.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.021+0000 7f5861400640 1 -- 172.21.15.90:0/119398282 learned_addr learned my addr 172.21.15.90:0/119398282 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:54.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.021+0000 7f586a800640 1 --2- 172.21.15.90:0/119398282 >> v2:172.21.15.103:3300/0 conn(0x7f586c06e280 0x7f586c15fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.021+0000 7f5869e00640 1 -- 172.21.15.90:0/119398282 >> v2:172.21.15.103:3300/0 conn(0x7f586c06e280 msgr2=0x7f586c15fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:54.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.021+0000 7f5869e00640 1 --2- 172.21.15.90:0/119398282 >> v2:172.21.15.103:3300/0 conn(0x7f586c06e280 0x7f586c15fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.021+0000 7f5869e00640 1 -- 172.21.15.90:0/119398282 >> v1:172.21.15.90:6789/0 conn(0x7f586c15f9b0 legacy=0x7f586c0ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.021+0000 7f5869e00640 1 -- 172.21.15.90:0/119398282 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f586c0b4f60 con 0x7f586c145590 2024-04-25T01:16:54.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.021+0000 7f5869400640 1 -- 172.21.15.90:0/119398282 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1416742252 0 0) 0x7f586c0b4240 con 0x7f586c15f9b0 2024-04-25T01:16:54.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.021+0000 7f586a800640 1 --2- 172.21.15.90:0/119398282 >> v2:172.21.15.103:3300/0 conn(0x7f586c06e280 0x7f586c15fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:54.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.022+0000 7f5869e00640 1 --2- 172.21.15.90:0/119398282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f586c145590 0x7f586c0b4a20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f586c068b60 tx=0x7f585c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3473b3d95e7b0279 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.023+0000 7f5869400640 1 -- 172.21.15.90:0/119398282 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585c037070 con 0x7f586c145590 2024-04-25T01:16:54.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.023+0000 7f5869400640 1 -- 172.21.15.90:0/119398282 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f585c032440 con 0x7f586c145590 2024-04-25T01:16:54.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.023+0000 7f5869400640 1 -- 172.21.15.90:0/119398282 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585c032810 con 0x7f586c145590 2024-04-25T01:16:54.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.023+0000 7f5870a00640 1 -- 172.21.15.90:0/119398282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f586c145590 msgr2=0x7f586c0b4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.023+0000 7f5870a00640 1 --2- 172.21.15.90:0/119398282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f586c145590 0x7f586c0b4a20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f586c068b60 tx=0x7f585c02a1a0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.023+0000 7f5870a00640 1 -- 172.21.15.90:0/119398282 shutdown_connections 2024-04-25T01:16:54.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.023+0000 7f5870a00640 1 --2- 172.21.15.90:0/119398282 >> v2:172.21.15.103:3300/0 conn(0x7f586c06e280 0x7f586c15fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.023+0000 7f5870a00640 1 --2- 172.21.15.90:0/119398282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f586c145590 0x7f586c0b4a20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.023+0000 7f5870a00640 1 -- 172.21.15.90:0/119398282 >> 172.21.15.90:0/119398282 conn(0x7f586c0b1820 msgr2=0x7f586c0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:54.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.023+0000 7f5870a00640 1 -- 172.21.15.90:0/119398282 shutdown_connections 2024-04-25T01:16:54.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.023+0000 7f5870a00640 1 -- 172.21.15.90:0/119398282 wait complete. 2024-04-25T01:16:54.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.025+0000 7f5870a00640 1 Processor -- start 2024-04-25T01:16:54.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.025+0000 7f5870a00640 1 -- start start 2024-04-25T01:16:54.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.026+0000 7f5870a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f586c06e280 0x7f586c15ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.026+0000 7f5870a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f586c145590 0x7f586c0722b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.026+0000 7f5870a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f586c15f9b0 0x7f586c0727f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.026+0000 7f5870a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f586c0b3c80 con 0x7f586c145590 2024-04-25T01:16:54.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.026+0000 7f5870a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f586c06cde0 con 0x7f586c06e280 2024-04-25T01:16:54.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.026+0000 7f5870a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f586c0b3f60 con 0x7f586c15f9b0 2024-04-25T01:16:54.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.026+0000 7f5861400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f586c145590 0x7f586c0722b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.027+0000 7f5861400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f586c145590 0x7f586c0722b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33566/0 (socket says 172.21.15.90:33566) 2024-04-25T01:16:54.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.027+0000 7f5861400640 1 -- 172.21.15.90:0/3402769695 learned_addr learned my addr 172.21.15.90:0/3402769695 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:54.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.027+0000 7f5869e00640 1 --2- 172.21.15.90:0/3402769695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f586c06e280 0x7f586c15ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.027+0000 7f586a800640 1 --2- 172.21.15.90:0/3402769695 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f586c15f9b0 0x7f586c0727f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.027+0000 7f5861400640 1 -- 172.21.15.90:0/3402769695 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f586c15f9b0 msgr2=0x7f586c0727f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.027+0000 7f5861400640 1 --2- 172.21.15.90:0/3402769695 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f586c15f9b0 0x7f586c0727f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.027+0000 7f5861400640 1 -- 172.21.15.90:0/3402769695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f586c06e280 msgr2=0x7f586c15ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.027+0000 7f5861400640 1 --2- 172.21.15.90:0/3402769695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f586c06e280 0x7f586c15ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.027+0000 7f5861400640 1 -- 172.21.15.90:0/3402769695 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f586c15d460 con 0x7f586c145590 2024-04-25T01:16:54.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.027+0000 7f5861400640 1 --2- 172.21.15.90:0/3402769695 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f586c145590 0x7f586c0722b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f585400c430 tx=0x7f585400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.027+0000 7f5862a00640 1 -- 172.21.15.90:0/3402769695 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58540074e0 con 0x7f586c145590 2024-04-25T01:16:54.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.028+0000 7f5862a00640 1 -- 172.21.15.90:0/3402769695 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58540047e0 con 0x7f586c145590 2024-04-25T01:16:54.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.028+0000 7f5862a00640 1 -- 172.21.15.90:0/3402769695 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5854004bd0 con 0x7f586c145590 2024-04-25T01:16:54.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.028+0000 7f5870a00640 1 -- 172.21.15.90:0/3402769695 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f586c144010 con 0x7f586c145590 2024-04-25T01:16:54.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.028+0000 7f5870a00640 1 -- 172.21.15.90:0/3402769695 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f586c0af290 con 0x7f586c145590 2024-04-25T01:16:54.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.029+0000 7f5870a00640 1 -- 172.21.15.90:0/3402769695 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f586c068fb0 con 0x7f586c145590 2024-04-25T01:16:54.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.030+0000 7f5862a00640 1 -- 172.21.15.90:0/3402769695 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5854011070 con 0x7f586c145590 2024-04-25T01:16:54.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.030+0000 7f5862a00640 1 --2- 172.21.15.90:0/3402769695 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f584806be30 0x7f584806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.031+0000 7f5862a00640 1 -- 172.21.15.90:0/3402769695 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f585408aeb0 con 0x7f586c145590 2024-04-25T01:16:54.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.031+0000 7f5869e00640 1 --2- 172.21.15.90:0/3402769695 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f584806be30 0x7f584806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.031+0000 7f5869e00640 1 --2- 172.21.15.90:0/3402769695 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f584806be30 0x7f584806e2f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f585c034000 tx=0x7f585c02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.035+0000 7f5862a00640 1 -- 172.21.15.90:0/3402769695 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f586c15d460 con 0x7f586c145590 2024-04-25T01:16:54.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.233+0000 7f5870a00640 1 -- 172.21.15.90:0/3402769695 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f586c06ea50 con 0x7f586c145590 2024-04-25T01:16:54.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.236+0000 7f5862a00640 1 -- 172.21.15.90:0/3402769695 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18482 (secure 0 0 0) 0x7f5854059480 con 0x7f586c145590 2024-04-25T01:16:54.221 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:16:54.221 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":26,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:16:27.092855+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:16:20.094512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:16:21.078268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:16:24.670642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:16:54.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.239+0000 7f5870a00640 1 -- 172.21.15.90:0/3402769695 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f584806be30 msgr2=0x7f584806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.239+0000 7f5870a00640 1 --2- 172.21.15.90:0/3402769695 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f584806be30 0x7f584806e2f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f585c034000 tx=0x7f585c02e000 comp rx=0 tx=0).stop 2024-04-25T01:16:54.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.239+0000 7f5870a00640 1 -- 172.21.15.90:0/3402769695 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f586c145590 msgr2=0x7f586c0722b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.239+0000 7f5870a00640 1 --2- 172.21.15.90:0/3402769695 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f586c145590 0x7f586c0722b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f585400c430 tx=0x7f585400c8f0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.239+0000 7f5870a00640 1 -- 172.21.15.90:0/3402769695 shutdown_connections 2024-04-25T01:16:54.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.239+0000 7f5870a00640 1 --2- 172.21.15.90:0/3402769695 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f584806be30 0x7f584806e2f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.239+0000 7f5870a00640 1 --2- 172.21.15.90:0/3402769695 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f586c15f9b0 0x7f586c0727f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.239+0000 7f5870a00640 1 --2- 172.21.15.90:0/3402769695 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f586c145590 0x7f586c0722b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.239+0000 7f5870a00640 1 --2- 172.21.15.90:0/3402769695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f586c06e280 0x7f586c15ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.239+0000 7f5870a00640 1 -- 172.21.15.90:0/3402769695 >> 172.21.15.90:0/3402769695 conn(0x7f586c0b1820 msgr2=0x7f586c0b0890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.239+0000 7f5870a00640 1 -- 172.21.15.90:0/3402769695 shutdown_connections 2024-04-25T01:16:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.239+0000 7f5870a00640 1 -- 172.21.15.90:0/3402769695 wait complete. 2024-04-25T01:16:54.243 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-25T01:16:54.244 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-25T01:16:54.244 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-25T01:16:54.245 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-25T01:16:54.245 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-25T01:16:54.245 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-25T01:16:54.246 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-25T01:16:54.246 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-25T01:16:54.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.539+0000 7f563dc00640 1 Processor -- start 2024-04-25T01:16:54.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.540+0000 7f563dc00640 1 -- start start 2024-04-25T01:16:54.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.543+0000 7f563dc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5638157ad0 0x7f56380af060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.543+0000 7f563dc00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f563806b830 0x7f563815b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.543+0000 7f563dc00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5638157eb0 con 0x7f56380aea50 2024-04-25T01:16:54.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.543+0000 7f563dc00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f563815ac30 con 0x7f563806b830 2024-04-25T01:16:54.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.543+0000 7f563dc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f56380cdce0 con 0x7f5638157ad0 2024-04-25T01:16:54.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.544+0000 7f563d200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f563806b830 0x7f563815b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.544+0000 7f5637400640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f56380aea50 0x7f563806b120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.90:58026/0 (socket says 172.21.15.90:58026) 2024-04-25T01:16:54.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.544+0000 7f5637e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5638157ad0 0x7f56380af060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.544+0000 7f5637e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5638157ad0 0x7f56380af060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.90:36756/0 (socket says 172.21.15.90:36756) 2024-04-25T01:16:54.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.544+0000 7f5637e00640 1 -- 172.21.15.90:0/697257616 learned_addr learned my addr 172.21.15.90:0/697257616 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:54.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.545+0000 7f5637e00640 1 -- 172.21.15.90:0/697257616 >> v2:172.21.15.103:3300/0 conn(0x7f563806b830 msgr2=0x7f563815b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:54.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.545+0000 7f5637e00640 1 --2- 172.21.15.90:0/697257616 >> v2:172.21.15.103:3300/0 conn(0x7f563806b830 0x7f563815b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.545+0000 7f5637e00640 1 -- 172.21.15.90:0/697257616 >> v1:172.21.15.103:6789/0 conn(0x7f56380aea50 legacy=0x7f563806b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.545+0000 7fa610e00640 1 Processor -- start 2024-04-25T01:16:54.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.545+0000 7f5636a00640 1 -- 172.21.15.90:0/697257616 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 25319262 0 0) 0x7f5638157eb0 con 0x7f56380aea50 2024-04-25T01:16:54.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.546+0000 7f5637e00640 1 -- 172.21.15.90:0/697257616 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56380cdfc0 con 0x7f5638157ad0 2024-04-25T01:16:54.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.547+0000 7f5637e00640 1 --2- 172.21.15.90:0/697257616 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5638157ad0 0x7f56380af060 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f56200091e0 tx=0x7f562002c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d62d7decb09a19b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.548+0000 7f5636a00640 1 -- 172.21.15.90:0/697257616 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5620038070 con 0x7f5638157ad0 2024-04-25T01:16:54.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.548+0000 7f5636a00640 1 -- 172.21.15.90:0/697257616 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5620031710 con 0x7f5638157ad0 2024-04-25T01:16:54.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.548+0000 7f5636a00640 1 -- 172.21.15.90:0/697257616 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5620031ae0 con 0x7f5638157ad0 2024-04-25T01:16:54.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.546+0000 7fa610e00640 1 -- start start 2024-04-25T01:16:54.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.547+0000 7fd7b3e00640 1 Processor -- start 2024-04-25T01:16:54.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.548+0000 7fa610e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa60403c0a0 0x7fa6040e7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.548+0000 7fa610e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa604053cb0 0x7fa604054120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.548+0000 7fa610e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa604054660 con 0x7fa6040e3160 2024-04-25T01:16:54.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.548+0000 7fa610e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa604007b70 con 0x7fa604053cb0 2024-04-25T01:16:54.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.548+0000 7fa610e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa604058810 con 0x7fa60403c0a0 2024-04-25T01:16:54.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.548+0000 7fd7b3e00640 1 -- start start 2024-04-25T01:16:54.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.549+0000 7fa60b400640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fa6040e3160 0x7fa604058050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.90:58034/0 (socket says 172.21.15.90:58034) 2024-04-25T01:16:54.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.549+0000 7f563dc00640 1 -- 172.21.15.90:0/697257616 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5638157ad0 msgr2=0x7f56380af060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.549+0000 7f563dc00640 1 --2- 172.21.15.90:0/697257616 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5638157ad0 0x7f56380af060 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f56200091e0 tx=0x7f562002c480 comp rx=0 tx=0).stop 2024-04-25T01:16:54.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.549+0000 7fa60b400640 1 -- 172.21.15.90:0/4137601782 learned_addr learned my addr 172.21.15.90:0/4137601782 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:54.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.549+0000 7fa60aa00640 1 --2- 172.21.15.90:0/4137601782 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa60403c0a0 0x7fa6040e7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.549+0000 7f563dc00640 1 -- 172.21.15.90:0/697257616 shutdown_connections 2024-04-25T01:16:54.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.549+0000 7f563dc00640 1 --2- 172.21.15.90:0/697257616 >> v2:172.21.15.103:3300/0 conn(0x7f563806b830 0x7f563815b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.549+0000 7f563dc00640 1 --2- 172.21.15.90:0/697257616 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5638157ad0 0x7f56380af060 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.549+0000 7f563dc00640 1 -- 172.21.15.90:0/697257616 >> 172.21.15.90:0/697257616 conn(0x7f56380b1660 msgr2=0x7f56380b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:54.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.549+0000 7fe535400640 1 Processor -- start 2024-04-25T01:16:54.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.550+0000 7f563dc00640 1 -- 172.21.15.90:0/697257616 shutdown_connections 2024-04-25T01:16:54.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.550+0000 7fe535400640 1 -- start start 2024-04-25T01:16:54.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.550+0000 7f563dc00640 1 -- 172.21.15.90:0/697257616 wait complete. 2024-04-25T01:16:54.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.550+0000 7fd7b3e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd7b40b50e0 0x7fd7b4161010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.550+0000 7fd7b3e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7b40d0ca0 0x7fd7b4161550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.550+0000 7fa60be00640 1 --2- 172.21.15.90:0/4137601782 >> v2:172.21.15.103:3300/0 conn(0x7fa604053cb0 0x7fa604054120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.551+0000 7fd7b2a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd7b40d08c0 0x7fd7b40ce990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60776/0 (socket says 172.21.15.90:60776) 2024-04-25T01:16:54.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.551+0000 7fd7b2000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd7b40b50e0 0x7fd7b4161010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.551+0000 7fa60aa00640 1 -- 172.21.15.90:0/4137601782 >> v2:172.21.15.103:3300/0 conn(0x7fa604053cb0 msgr2=0x7fa604054120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:54.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.551+0000 7fd7b2000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd7b40b50e0 0x7fd7b4161010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33580/0 (socket says 172.21.15.90:33580) 2024-04-25T01:16:54.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.552+0000 7fe535400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe53015c8a0 0x7fe5301633d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.552+0000 7fe535400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5300af400 con 0x7fe5300ae730 2024-04-25T01:16:54.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.552+0000 7fe535400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5300aeb90 con 0x7fe53006b120 2024-04-25T01:16:54.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.552+0000 7fe535400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe5300cc800 con 0x7fe53015c8a0 2024-04-25T01:16:54.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.552+0000 7fd7b3e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7b40cf3e0 con 0x7fd7b40d08c0 2024-04-25T01:16:54.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.552+0000 7fd7b3e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fd7b40d2e80 con 0x7fd7b40b50e0 2024-04-25T01:16:54.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.552+0000 7fd7b3e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd7b40cf6c0 con 0x7fd7b40d0ca0 2024-04-25T01:16:54.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.552+0000 7fd7b2a00640 1 -- 172.21.15.90:0/2246931769 learned_addr learned my addr 172.21.15.90:0/2246931769 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:54.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.552+0000 7f563dc00640 1 Processor -- start 2024-04-25T01:16:54.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.551+0000 7fa60aa00640 1 --2- 172.21.15.90:0/4137601782 >> v2:172.21.15.103:3300/0 conn(0x7fa604053cb0 0x7fa604054120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.552+0000 7f563dc00640 1 -- start start 2024-04-25T01:16:54.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.551+0000 7fa60aa00640 1 -- 172.21.15.90:0/4137601782 >> v1:172.21.15.103:6789/0 conn(0x7fa6040e3160 legacy=0x7fa604058050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.551+0000 7fa60aa00640 1 -- 172.21.15.90:0/4137601782 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa604058af0 con 0x7fa60403c0a0 2024-04-25T01:16:54.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.553+0000 7fe52fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe53015c8a0 0x7fe5301633d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.553+0000 7fe52fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe53015c8a0 0x7fe5301633d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.90:36784/0 (socket says 172.21.15.90:36784) 2024-04-25T01:16:54.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.553+0000 7fd7b3400640 1 --2- 172.21.15.90:0/2246931769 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7b40d0ca0 0x7fd7b4161550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.553+0000 7fe534a00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fe53006b120 0x7fe53006b540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.90:58036/0 (socket says 172.21.15.90:58036) 2024-04-25T01:16:54.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.553+0000 7fe534a00640 1 -- 172.21.15.90:0/493802017 learned_addr learned my addr 172.21.15.90:0/493802017 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:54.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.553+0000 7fe52f400640 1 --1- 172.21.15.90:0/493802017 >> v1:172.21.15.90:6789/0 conn(0x7fe5300ae730 0x7fe5300ccc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60792/0 (socket says 172.21.15.90:60792) 2024-04-25T01:16:54.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.553+0000 7fd7b2000640 1 -- 172.21.15.90:0/2246931769 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7b40d0ca0 msgr2=0x7fd7b4161550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:54.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.554+0000 7f563dc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f563806b830 0x7f5638159230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.554+0000 7f563dc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56380aea50 0x7f5638159770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.554+0000 7f563dc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5638157ad0 0x7f5638151860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.554+0000 7f563dc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f56380cd870 con 0x7f5638157ad0 2024-04-25T01:16:54.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.554+0000 7fe52ea00640 1 -- 172.21.15.90:0/493802017 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2920144574 0 0) 0x7fe5300aeb90 con 0x7fe53006b120 2024-04-25T01:16:54.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.554+0000 7f563dc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f563815a8c0 con 0x7f56380aea50 2024-04-25T01:16:54.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.554+0000 7f563dc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f56380cd9f0 con 0x7f563806b830 2024-04-25T01:16:54.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.553+0000 7fd7b2000640 1 --2- 172.21.15.90:0/2246931769 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7b40d0ca0 0x7fd7b4161550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.553+0000 7fd7b2000640 1 -- 172.21.15.90:0/2246931769 >> v1:172.21.15.90:6789/0 conn(0x7fd7b40d08c0 legacy=0x7fd7b40ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.554+0000 7fd7b2000640 1 -- 172.21.15.90:0/2246931769 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7b40cf9a0 con 0x7fd7b40b50e0 2024-04-25T01:16:54.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.554+0000 7fe52ea00640 1 -- 172.21.15.90:0/493802017 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe514003520 con 0x7fe53006b120 2024-04-25T01:16:54.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.554+0000 7fe52ea00640 1 -- 172.21.15.90:0/493802017 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 445058178 0 0) 0x7fe5300af400 con 0x7fe5300ae730 2024-04-25T01:16:54.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.554+0000 7fe52ea00640 1 -- 172.21.15.90:0/493802017 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5300aeb90 con 0x7fe5300ae730 2024-04-25T01:16:54.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.555+0000 7f5637400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56380aea50 0x7f5638159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.555+0000 7fe52fe00640 1 -- 172.21.15.90:0/493802017 >> v1:172.21.15.103:6789/0 conn(0x7fe53006b120 legacy=0x7fe53006b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.555+0000 7f563d200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5638157ad0 0x7f5638151860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.555+0000 7f563d200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5638157ad0 0x7f5638151860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33582/0 (socket says 172.21.15.90:33582) 2024-04-25T01:16:54.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.555+0000 7f563d200640 1 -- 172.21.15.90:0/26083863 learned_addr learned my addr 172.21.15.90:0/26083863 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:54.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.555+0000 7fe52fe00640 1 -- 172.21.15.90:0/493802017 >> v1:172.21.15.90:6789/0 conn(0x7fe5300ae730 legacy=0x7fe5300ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.555+0000 7fe52fe00640 1 -- 172.21.15.90:0/493802017 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5300aee40 con 0x7fe53015c8a0 2024-04-25T01:16:54.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.555+0000 7fa60aa00640 1 --2- 172.21.15.90:0/4137601782 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa60403c0a0 0x7fa6040e7780 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa5f80097a0 tx=0x7fa5f802a0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b06c7d1a8364405 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.555+0000 7fa60a000640 1 -- 172.21.15.90:0/4137601782 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8037070 con 0x7fa60403c0a0 2024-04-25T01:16:54.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.556+0000 7fa60a000640 1 -- 172.21.15.90:0/4137601782 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5f802ae20 con 0x7fa60403c0a0 2024-04-25T01:16:54.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.556+0000 7fa60a000640 1 -- 172.21.15.90:0/4137601782 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8032660 con 0x7fa60403c0a0 2024-04-25T01:16:54.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.556+0000 7f5637400640 1 -- 172.21.15.90:0/26083863 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f563806b830 msgr2=0x7f5638159230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.556+0000 7f5637400640 1 --2- 172.21.15.90:0/26083863 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f563806b830 0x7f5638159230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.556+0000 7f5637400640 1 -- 172.21.15.90:0/26083863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5638157ad0 msgr2=0x7f5638151860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.556+0000 7fd7b2000640 1 --2- 172.21.15.90:0/2246931769 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7b40b50e0 0x7fd7b4161010 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd79c0097a0 tx=0x7fd79c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=232281bd505ddef8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.557+0000 7f5637400640 1 --2- 172.21.15.90:0/26083863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5638157ad0 0x7f5638151860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.557+0000 7f5637400640 1 -- 172.21.15.90:0/26083863 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56381525d0 con 0x7f56380aea50 2024-04-25T01:16:54.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.557+0000 7fe52fe00640 1 --2- 172.21.15.90:0/493802017 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe53015c8a0 0x7fe5301633d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe5200097d0 tx=0x7fe52002a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7b0b2280440c12c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.557+0000 7f563d200640 1 --2- 172.21.15.90:0/26083863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5638157ad0 0x7f5638151860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:16:54.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.557+0000 7f5637400640 1 --2- 172.21.15.90:0/26083863 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56380aea50 0x7f5638159770 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f562800c3e0 tx=0x7f562800c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.558+0000 7fa610e00640 1 -- 172.21.15.90:0/4137601782 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa60403c0a0 msgr2=0x7fa6040e7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.558+0000 7fa610e00640 1 --2- 172.21.15.90:0/4137601782 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa60403c0a0 0x7fa6040e7780 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa5f80097a0 tx=0x7fa5f802a0d0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.558+0000 7fd7b1600640 1 -- 172.21.15.90:0/2246931769 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c037070 con 0x7fd7b40b50e0 2024-04-25T01:16:54.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.558+0000 7fd7b1600640 1 -- 172.21.15.90:0/2246931769 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd79c032440 con 0x7fd7b40b50e0 2024-04-25T01:16:54.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.558+0000 7fd7b1600640 1 -- 172.21.15.90:0/2246931769 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c032810 con 0x7fd7b40b50e0 2024-04-25T01:16:54.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.558+0000 7fe52ea00640 1 -- 172.21.15.90:0/493802017 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe520037070 con 0x7fe53015c8a0 2024-04-25T01:16:54.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.558+0000 7fa610e00640 1 -- 172.21.15.90:0/4137601782 shutdown_connections 2024-04-25T01:16:54.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.558+0000 7fa610e00640 1 --2- 172.21.15.90:0/4137601782 >> v2:172.21.15.103:3300/0 conn(0x7fa604053cb0 0x7fa604054120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.558+0000 7fa610e00640 1 --2- 172.21.15.90:0/4137601782 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa60403c0a0 0x7fa6040e7780 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.558+0000 7fa610e00640 1 -- 172.21.15.90:0/4137601782 >> 172.21.15.90:0/4137601782 conn(0x7fa604058ee0 msgr2=0x7fa6040592c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:54.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.558+0000 7fe52ea00640 1 -- 172.21.15.90:0/493802017 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe520032440 con 0x7fe53015c8a0 2024-04-25T01:16:54.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.558+0000 7fe52ea00640 1 -- 172.21.15.90:0/493802017 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe520032810 con 0x7fe53015c8a0 2024-04-25T01:16:54.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.558+0000 7fa610e00640 1 -- 172.21.15.90:0/4137601782 shutdown_connections 2024-04-25T01:16:54.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.558+0000 7f562fe00640 1 -- 172.21.15.90:0/26083863 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56280074e0 con 0x7f56380aea50 2024-04-25T01:16:54.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.558+0000 7f563dc00640 1 -- 172.21.15.90:0/26083863 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56381528b0 con 0x7f56380aea50 2024-04-25T01:16:54.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.558+0000 7f562fe00640 1 -- 172.21.15.90:0/26083863 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5628007680 con 0x7f56380aea50 2024-04-25T01:16:54.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.558+0000 7f562fe00640 1 -- 172.21.15.90:0/26083863 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5628004a20 con 0x7f56380aea50 2024-04-25T01:16:54.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.559+0000 7f563dc00640 1 -- 172.21.15.90:0/26083863 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5638152e70 con 0x7f56380aea50 2024-04-25T01:16:54.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.559+0000 7fe535400640 1 -- 172.21.15.90:0/493802017 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe53015c8a0 msgr2=0x7fe5301633d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.559+0000 7fe535400640 1 --2- 172.21.15.90:0/493802017 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe53015c8a0 0x7fe5301633d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe5200097d0 tx=0x7fe52002a210 comp rx=0 tx=0).stop 2024-04-25T01:16:54.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.559+0000 7fe535400640 1 -- 172.21.15.90:0/493802017 shutdown_connections 2024-04-25T01:16:54.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.559+0000 7fe535400640 1 --2- 172.21.15.90:0/493802017 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe53015c8a0 0x7fe5301633d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.559+0000 7fe535400640 1 -- 172.21.15.90:0/493802017 >> 172.21.15.90:0/493802017 conn(0x7fe5300b1660 msgr2=0x7fe5300b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:54.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.559+0000 7fd7b3e00640 1 -- 172.21.15.90:0/2246931769 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7b40b50e0 msgr2=0x7fd7b4161010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.559+0000 7fd7b3e00640 1 --2- 172.21.15.90:0/2246931769 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7b40b50e0 0x7fd7b4161010 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd79c0097a0 tx=0x7fd79c02a180 comp rx=0 tx=0).stop 2024-04-25T01:16:54.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.560+0000 7fd7b3e00640 1 -- 172.21.15.90:0/2246931769 shutdown_connections 2024-04-25T01:16:54.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.560+0000 7fd7b3e00640 1 --2- 172.21.15.90:0/2246931769 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7b40d0ca0 0x7fd7b4161550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.560+0000 7fd7b3e00640 1 --2- 172.21.15.90:0/2246931769 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7b40b50e0 0x7fd7b4161010 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.560+0000 7fd7b3e00640 1 -- 172.21.15.90:0/2246931769 >> 172.21.15.90:0/2246931769 conn(0x7fd7b409d070 msgr2=0x7fd7b40d3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:54.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.560+0000 7f6d36e00640 1 Processor -- start 2024-04-25T01:16:54.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.560+0000 7fe535400640 1 -- 172.21.15.90:0/493802017 shutdown_connections 2024-04-25T01:16:54.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.560+0000 7fd7b3e00640 1 -- 172.21.15.90:0/2246931769 shutdown_connections 2024-04-25T01:16:54.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.558+0000 7fa610e00640 1 -- 172.21.15.90:0/4137601782 wait complete. 2024-04-25T01:16:54.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.561+0000 7f563dc00640 1 -- 172.21.15.90:0/26083863 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5638075350 con 0x7f56380aea50 2024-04-25T01:16:54.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.561+0000 7fa610e00640 1 Processor -- start 2024-04-25T01:16:54.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.561+0000 7fa610e00640 1 -- start start 2024-04-25T01:16:54.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.560+0000 7fe535400640 1 -- 172.21.15.90:0/493802017 wait complete. 2024-04-25T01:16:54.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.560+0000 7f6d36e00640 1 -- start start 2024-04-25T01:16:54.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.562+0000 7f6d36e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d300b4a10 0x7f6d300cd060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.563+0000 7f6d36e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6d3006c130 0x7f6d3006c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.563+0000 7f631c800640 1 Processor -- start 2024-04-25T01:16:54.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.563+0000 7f562fe00640 1 -- 172.21.15.90:0/26083863 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5628011070 con 0x7f56380aea50 2024-04-25T01:16:54.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.560+0000 7fd7b3e00640 1 -- 172.21.15.90:0/2246931769 wait complete. 2024-04-25T01:16:54.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.563+0000 7f6d2fe00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f6d3015a570 0x7f6d300b3e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.90:58052/0 (socket says 172.21.15.90:58052) 2024-04-25T01:16:54.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.563+0000 7f631c800640 1 -- start start 2024-04-25T01:16:54.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.563+0000 7fd7b3e00640 1 Processor -- start 2024-04-25T01:16:54.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.563+0000 7fe535400640 1 Processor -- start 2024-04-25T01:16:54.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.563+0000 7f6d2fe00640 1 -- 172.21.15.90:0/3845071324 learned_addr learned my addr 172.21.15.90:0/3845071324 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:54.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.563+0000 7f6d36e00640 1 -- 172.21.15.90:0/3845071324 --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d300b4fd0 con 0x7f6d3015a570 2024-04-25T01:16:54.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.563+0000 7f6d36e00640 1 -- 172.21.15.90:0/3845071324 --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f6d300b52b0 con 0x7f6d3006c130 2024-04-25T01:16:54.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.563+0000 7f6d36e00640 1 -- 172.21.15.90:0/3845071324 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6d3006db70 con 0x7f6d300b4a10 2024-04-25T01:16:54.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.564+0000 7fe535400640 1 -- start start 2024-04-25T01:16:54.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.564+0000 7fd7b3e00640 1 -- start start 2024-04-25T01:16:54.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.563+0000 7f6d34e00640 1 --2- 172.21.15.90:0/3845071324 >> v2:172.21.15.90:3300/0 conn(0x7f6d3006c130 0x7f6d3006c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.564+0000 7fb556000640 1 Processor -- start 2024-04-25T01:16:54.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.564+0000 7fa610e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa60403c0a0 0x7fa604051a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.564+0000 7f6d34e00640 1 -- 172.21.15.90:0/3845071324 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d300b4a10 msgr2=0x7f6d300cd060 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:16:54.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.565+0000 7f6d2f400640 1 --2- 172.21.15.90:0/3845071324 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d300b4a10 0x7f6d300cd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.565+0000 7f6d2ea00640 1 -- 172.21.15.90:0/3845071324 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 727643753 0 0) 0x7f6d300b4fd0 con 0x7f6d3015a570 2024-04-25T01:16:54.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.565+0000 7f6d34e00640 1 --2- 172.21.15.90:0/3845071324 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d300b4a10 0x7f6d300cd060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.565+0000 7f6d34e00640 1 -- 172.21.15.90:0/3845071324 >> v1:172.21.15.103:6789/0 conn(0x7f6d3015a570 legacy=0x7f6d300b3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.565+0000 7f6d34e00640 1 -- 172.21.15.90:0/3845071324 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d3006d520 con 0x7f6d3006c130 2024-04-25T01:16:54.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.564+0000 7fb556000640 1 -- start start 2024-04-25T01:16:54.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.564+0000 7f562fe00640 1 --2- 172.21.15.90:0/26083863 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f561406be30 0x7f561406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.565+0000 7fa60b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa60403c0a0 0x7fa604051a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.565+0000 7fa60b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa60403c0a0 0x7fa604051a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33608/0 (socket says 172.21.15.90:33608) 2024-04-25T01:16:54.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.565+0000 7fa60b400640 1 -- 172.21.15.90:0/3393952628 learned_addr learned my addr 172.21.15.90:0/3393952628 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:54.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.565+0000 7fe97b800640 1 Processor -- start 2024-04-25T01:16:54.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.566+0000 7fa610e00640 1 --2- 172.21.15.90:0/3393952628 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa604053cb0 0x7fa604188a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.566+0000 7f631c800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f631406d670 0x7f6314071c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.566+0000 7fe535400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe53006b120 0x7fe5301593a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.566+0000 7fe97b800640 1 -- start start 2024-04-25T01:16:54.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.566+0000 7fd7b3e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7b40b50e0 0x7fd7b415ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.566+0000 7f5637e00640 1 --2- 172.21.15.90:0/26083863 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f561406be30 0x7f561406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.566+0000 7f562fe00640 1 -- 172.21.15.90:0/26083863 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f5628056d70 con 0x7f56380aea50 2024-04-25T01:16:54.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.566+0000 7f631c800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63140721a0 0x7f631415c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.566+0000 7f562fe00640 1 --2- 172.21.15.90:0/26083863 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x7f56140754b0 0x7f56140778f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.566+0000 7fa610e00640 1 --2- 172.21.15.90:0/3393952628 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa6040e3160 0x7fa6040e56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.566+0000 7fd7b3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7b40d08c0 0x7fd7b40c78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.566+0000 7fd7b3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7b40d0ca0 0x7fd7b40c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.566+0000 7fe535400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe5300ae730 0x7fe5301612f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.566+0000 7fe535400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe53015c8a0 0x7fe53014f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.567+0000 7fe535400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe53015cc80 con 0x7fe53006b120 2024-04-25T01:16:54.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.566+0000 7f631c800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f631415b770 con 0x7f63141405d0 2024-04-25T01:16:54.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.566+0000 7f631c800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f631415bd70 con 0x7f631406d670 2024-04-25T01:16:54.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.566+0000 7f631c800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f631406db00 con 0x7f63140721a0 2024-04-25T01:16:54.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.566+0000 7f6319e00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f63141405d0 0x7f631415b060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.90:58058/0 (socket says 172.21.15.90:58058) 2024-04-25T01:16:54.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.567+0000 7f6319e00640 1 -- 172.21.15.90:0/815773153 learned_addr learned my addr 172.21.15.90:0/815773153 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:54.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.567+0000 7f6319400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f631406d670 0x7f6314071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.567+0000 7fd7b3e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b4161a90 con 0x7fd7b40d08c0 2024-04-25T01:16:54.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.567+0000 7fd7b3e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b40d1950 con 0x7fd7b40d0ca0 2024-04-25T01:16:54.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.567+0000 7fd7b3e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd7b40cf0a0 con 0x7fd7b40b50e0 2024-04-25T01:16:54.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.567+0000 7fe535400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe5300cc490 con 0x7fe53015c8a0 2024-04-25T01:16:54.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.567+0000 7f6d34e00640 1 --2- 172.21.15.90:0/3845071324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d3006c130 0x7f6d3006c5c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6d28009d90 tx=0x7f6d2802a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f09e616719a3a5b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.567+0000 7fe52f400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe5300ae730 0x7fe5301612f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.567+0000 7f562fe00640 1 -- 172.21.15.90:0/26083863 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5614077fc0 con 0x7f56140754b0 2024-04-25T01:16:54.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.567+0000 7f562fe00640 1 -- 172.21.15.90:0/26083863 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5628058aa0 con 0x7f56380aea50 2024-04-25T01:16:54.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.567+0000 7fd7b2a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7b40b50e0 0x7fd7b415ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.567+0000 7fd7b2a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7b40b50e0 0x7fd7b415ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.90:36818/0 (socket says 172.21.15.90:36818) 2024-04-25T01:16:54.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.567+0000 7fd7b2a00640 1 -- 172.21.15.90:0/1475378106 learned_addr learned my addr 172.21.15.90:0/1475378106 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:54.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.567+0000 7fe535400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe5300cdf40 con 0x7fe5300ae730 2024-04-25T01:16:54.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.567+0000 7fa60aa00640 1 --2- 172.21.15.90:0/3393952628 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa604053cb0 0x7fa604188a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.567+0000 7f631a800640 1 --2- 172.21.15.90:0/815773153 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63140721a0 0x7f631415c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.567+0000 7fb556000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb55015e4c0 0x7fb5500d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.567+0000 7f6d2ea00640 1 -- 172.21.15.90:0/3845071324 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d28007740 con 0x7f6d3006c130 2024-04-25T01:16:54.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.567+0000 7fb556000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb5500d4a10 0x7fb5500b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.567+0000 7fb556000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5500d1e90 con 0x7fb5500d2d40 2024-04-25T01:16:54.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.567+0000 7fe52f400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe5300ae730 0x7fe5301612f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.90:36824/0 (socket says 172.21.15.90:36824) 2024-04-25T01:16:54.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.567+0000 7fb556000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb5500b54a0 con 0x7fb5500d4a10 2024-04-25T01:16:54.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.567+0000 7fb556000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb5500d2130 con 0x7fb55015e4c0 2024-04-25T01:16:54.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.567+0000 7fe52f400640 1 -- 172.21.15.90:0/2639351029 learned_addr learned my addr 172.21.15.90:0/2639351029 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:54.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.567+0000 7f6d2ea00640 1 -- 172.21.15.90:0/3845071324 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d280078e0 con 0x7f6d3006c130 2024-04-25T01:16:54.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.568+0000 7f6d2ea00640 1 -- 172.21.15.90:0/3845071324 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d28007740 con 0x7f6d3006c130 2024-04-25T01:16:54.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.568+0000 7fe534a00640 1 --2- 172.21.15.90:0/2639351029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe53015c8a0 0x7fe53014f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.568+0000 7fe52fe00640 1 --2- 172.21.15.90:0/2639351029 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe53006b120 0x7fe5301593a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.568+0000 7fb54ea00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb5500d4a10 0x7fb5500b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.568+0000 7fb54ea00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb5500d4a10 0x7fb5500b3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.90:55756/0 (socket says 172.21.15.90:55756) 2024-04-25T01:16:54.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.568+0000 7fb54ea00640 1 -- 172.21.15.90:0/3278109664 learned_addr learned my addr 172.21.15.90:0/3278109664 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:54.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.568+0000 7f563d200640 1 --2- 172.21.15.90:0/26083863 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x7f56140754b0 0x7f56140778f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.568+0000 7fd7b3400640 1 --2- 172.21.15.90:0/1475378106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7b40d0ca0 0x7fd7b40c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.568+0000 7fd7b2a00640 1 -- 172.21.15.90:0/1475378106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7b40d0ca0 msgr2=0x7fd7b40c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.568+0000 7fd7b2a00640 1 --2- 172.21.15.90:0/1475378106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7b40d0ca0 0x7fd7b40c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.568+0000 7fd7b2a00640 1 -- 172.21.15.90:0/1475378106 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7b40d08c0 msgr2=0x7fd7b40c78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.568+0000 7fd7b2a00640 1 --2- 172.21.15.90:0/1475378106 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7b40d08c0 0x7fd7b40c78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.568+0000 7fd7b2a00640 1 -- 172.21.15.90:0/1475378106 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7b415b9f0 con 0x7fd7b40b50e0 2024-04-25T01:16:54.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.568+0000 7f6d36e00640 1 -- 172.21.15.90:0/3845071324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d3006c130 msgr2=0x7f6d3006c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.568+0000 7f6d36e00640 1 --2- 172.21.15.90:0/3845071324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d3006c130 0x7f6d3006c5c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6d28009d90 tx=0x7f6d2802a610 comp rx=0 tx=0).stop 2024-04-25T01:16:54.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.569+0000 7f5637e00640 1 --2- 172.21.15.90:0/26083863 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f561406be30 0x7f561406e2f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5620035000 tx=0x7f5620030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.569+0000 7f563d200640 1 --2- 172.21.15.90:0/26083863 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x7f56140754b0 0x7f56140778f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.568+0000 7fa610e00640 1 -- 172.21.15.90:0/3393952628 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa6040e3540 con 0x7fa60403c0a0 2024-04-25T01:16:54.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.568+0000 7fa610e00640 1 -- 172.21.15.90:0/3393952628 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa604006640 con 0x7fa604053cb0 2024-04-25T01:16:54.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.568+0000 7fa610e00640 1 -- 172.21.15.90:0/3393952628 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa604006b60 con 0x7fa6040e3160 2024-04-25T01:16:54.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.569+0000 7fa60be00640 1 --2- 172.21.15.90:0/3393952628 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa6040e3160 0x7fa6040e56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.569+0000 7fd7b2a00640 1 --2- 172.21.15.90:0/1475378106 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7b40b50e0 0x7fd7b415ca90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac00cb40 tx=0x7fd7ac00a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.569+0000 7fe52f400640 1 -- 172.21.15.90:0/2639351029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe53015c8a0 msgr2=0x7fe53014f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.569+0000 7fe52f400640 1 --2- 172.21.15.90:0/2639351029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe53015c8a0 0x7fe53014f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.569+0000 7fe52f400640 1 -- 172.21.15.90:0/2639351029 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe53006b120 msgr2=0x7fe5301593a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.569+0000 7fe52f400640 1 --2- 172.21.15.90:0/2639351029 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe53006b120 0x7fe5301593a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.569+0000 7fe52f400640 1 -- 172.21.15.90:0/2639351029 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe530161cd0 con 0x7fe5300ae730 2024-04-25T01:16:54.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.569+0000 7f6d36e00640 1 -- 172.21.15.90:0/3845071324 shutdown_connections 2024-04-25T01:16:54.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.569+0000 7f6d36e00640 1 --2- 172.21.15.90:0/3845071324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d3006c130 0x7f6d3006c5c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.569+0000 7f6d36e00640 1 --2- 172.21.15.90:0/3845071324 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d300b4a10 0x7f6d300cd060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.569+0000 7f6d36e00640 1 -- 172.21.15.90:0/3845071324 >> 172.21.15.90:0/3845071324 conn(0x7f6d30071550 msgr2=0x7f6d30071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:54.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.569+0000 7f6d36e00640 1 -- 172.21.15.90:0/3845071324 shutdown_connections 2024-04-25T01:16:54.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.568+0000 7f6319400640 1 -- 172.21.15.90:0/815773153 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63140721a0 msgr2=0x7f631415c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:54.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.568+0000 7f6319400640 1 --2- 172.21.15.90:0/815773153 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63140721a0 0x7f631415c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.568+0000 7f6319400640 1 -- 172.21.15.90:0/815773153 >> v1:172.21.15.103:6789/0 conn(0x7f63141405d0 legacy=0x7f631415b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.569+0000 7fe978e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe9741405d0 0x7fe97415b060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60804/0 (socket says 172.21.15.90:60804) 2024-04-25T01:16:54.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.569+0000 7fe97b800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe97406d670 0x7fe974071c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.569+0000 7fe978e00640 1 -- 172.21.15.90:0/1841680286 learned_addr learned my addr 172.21.15.90:0/1841680286 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:54.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.569+0000 7fe97b800640 1 --2- 172.21.15.90:0/1841680286 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9740721a0 0x7fe97415c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.568+0000 7fb54f400640 1 --2- 172.21.15.90:0/3278109664 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb55015e4c0 0x7fb5500d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.569+0000 7fb54e000640 1 -- 172.21.15.90:0/3278109664 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3514180424 0 0) 0x7fb5500d1e90 con 0x7fb5500d2d40 2024-04-25T01:16:54.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.569+0000 7fb54e000640 1 -- 172.21.15.90:0/3278109664 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb534003520 con 0x7fb5500d2d40 2024-04-25T01:16:54.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.569+0000 7fe97b800640 1 -- 172.21.15.90:0/1841680286 --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe97415b770 con 0x7fe9741405d0 2024-04-25T01:16:54.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.569+0000 7f6319400640 1 -- 172.21.15.90:0/815773153 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f631406dd50 con 0x7f631406d670 2024-04-25T01:16:54.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.569+0000 7fe973e00640 1 --2- 172.21.15.90:0/1841680286 >> v2:172.21.15.103:3300/0 conn(0x7fe97406d670 0x7fe974071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.570+0000 7fe97b800640 1 -- 172.21.15.90:0/1841680286 --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe97415bd70 con 0x7fe97406d670 2024-04-25T01:16:54.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.570+0000 7fe52f400640 1 --2- 172.21.15.90:0/2639351029 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe5300ae730 0x7fe5301612f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe51c00c480 tx=0x7fe51c00c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.570+0000 7fe97b800640 1 -- 172.21.15.90:0/1841680286 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe97406db00 con 0x7fe9740721a0 2024-04-25T01:16:54.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.570+0000 7fa60be00640 1 -- 172.21.15.90:0/3393952628 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa604053cb0 msgr2=0x7fa604188a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.569+0000 7f6d36e00640 1 -- 172.21.15.90:0/3845071324 wait complete. 2024-04-25T01:16:54.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.570+0000 7fd7aaa00640 1 -- 172.21.15.90:0/1475378106 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac017070 con 0x7fd7b40b50e0 2024-04-25T01:16:54.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.570+0000 7fd7b3e00640 1 -- 172.21.15.90:0/1475378106 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7b415bcd0 con 0x7fd7b40b50e0 2024-04-25T01:16:54.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.570+0000 7fa60be00640 1 --2- 172.21.15.90:0/3393952628 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa604053cb0 0x7fa604188a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.570+0000 7fa60be00640 1 -- 172.21.15.90:0/3393952628 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa60403c0a0 msgr2=0x7fa604051a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.570+0000 7fa60be00640 1 --2- 172.21.15.90:0/3393952628 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa60403c0a0 0x7fa604051a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.570+0000 7fa60be00640 1 -- 172.21.15.90:0/3393952628 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa604052c90 con 0x7fa6040e3160 2024-04-25T01:16:54.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.571+0000 7fd7b3e00640 1 -- 172.21.15.90:0/1475378106 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7b415c290 con 0x7fd7b40b50e0 2024-04-25T01:16:54.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.571+0000 7fe979800640 1 --2- 172.21.15.90:0/1841680286 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9740721a0 0x7fe97415c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.571+0000 7fe973400640 1 -- 172.21.15.90:0/1841680286 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 353129604 0 0) 0x7fe97415b770 con 0x7fe9741405d0 2024-04-25T01:16:54.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.571+0000 7fe973400640 1 -- 172.21.15.90:0/1841680286 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe958003520 con 0x7fe9741405d0 2024-04-25T01:16:54.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.570+0000 7fb54ea00640 1 -- 172.21.15.90:0/3278109664 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb55015e4c0 msgr2=0x7fb5500d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:54.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.570+0000 7fb54ea00640 1 --2- 172.21.15.90:0/3278109664 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb55015e4c0 0x7fb5500d2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.570+0000 7fb54ea00640 1 -- 172.21.15.90:0/3278109664 >> v1:172.21.15.103:6789/0 conn(0x7fb5500d2d40 legacy=0x7fb5500d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.570+0000 7fb54ea00640 1 -- 172.21.15.90:0/3278109664 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5500b5780 con 0x7fb5500d4a10 2024-04-25T01:16:54.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.571+0000 7fd7aaa00640 1 -- 172.21.15.90:0/1475378106 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7ac00aca0 con 0x7fd7b40b50e0 2024-04-25T01:16:54.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.571+0000 7fd7aaa00640 1 -- 172.21.15.90:0/1475378106 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac017070 con 0x7fd7b40b50e0 2024-04-25T01:16:54.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.571+0000 7fa60be00640 1 --2- 172.21.15.90:0/3393952628 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa6040e3160 0x7fa6040e56f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa60c064c60 tx=0x7fa60c073800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.571+0000 7f562fe00640 1 -- 172.21.15.90:0/26083863 <== osd.0 v2:172.21.15.90:6816/2625481430 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5614077fc0 con 0x7f56140754b0 2024-04-25T01:16:54.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.571+0000 7fe527e00640 1 -- 172.21.15.90:0/2639351029 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe51c018070 con 0x7fe5300ae730 2024-04-25T01:16:54.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.571+0000 7fe973e00640 1 -- 172.21.15.90:0/1841680286 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9740721a0 msgr2=0x7fe97415c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:54.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.571+0000 7fe973e00640 1 --2- 172.21.15.90:0/1841680286 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9740721a0 0x7fe97415c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.571+0000 7fe973e00640 1 -- 172.21.15.90:0/1841680286 >> v1:172.21.15.90:6789/0 conn(0x7fe9741405d0 legacy=0x7fe97415b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.571+0000 7fe973e00640 1 -- 172.21.15.90:0/1841680286 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe97406dd50 con 0x7fe97406d670 2024-04-25T01:16:54.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.571+0000 7fa5ff400640 1 -- 172.21.15.90:0/3393952628 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa60c068530 con 0x7fa6040e3160 2024-04-25T01:16:54.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.571+0000 7fa5ff400640 1 -- 172.21.15.90:0/3393952628 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa60c073e30 con 0x7fa6040e3160 2024-04-25T01:16:54.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.571+0000 7f6319400640 1 --2- 172.21.15.90:0/815773153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f631406d670 0x7f6314071c60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f630000a940 tx=0x7f630002b550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b05d7a455099aab0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.571+0000 7fa5ff400640 1 -- 172.21.15.90:0/3393952628 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa60c074540 con 0x7fa6040e3160 2024-04-25T01:16:54.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.572+0000 7fe527e00640 1 -- 172.21.15.90:0/2639351029 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe51c007480 con 0x7fe5300ae730 2024-04-25T01:16:54.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.572+0000 7fe527e00640 1 -- 172.21.15.90:0/2639351029 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe51c0047e0 con 0x7fe5300ae730 2024-04-25T01:16:54.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.572+0000 7f6d36e00640 1 Processor -- start 2024-04-25T01:16:54.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.572+0000 7fe535400640 1 -- 172.21.15.90:0/2639351029 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5301619f0 con 0x7fe5300ae730 2024-04-25T01:16:54.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.572+0000 7fe535400640 1 -- 172.21.15.90:0/2639351029 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe53015d210 con 0x7fe5300ae730 2024-04-25T01:16:54.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.572+0000 7fa610e00640 1 -- 172.21.15.90:0/3393952628 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa604052080 con 0x7fa6040e3160 2024-04-25T01:16:54.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.572+0000 7fa610e00640 1 -- 172.21.15.90:0/3393952628 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa604052640 con 0x7fa6040e3160 2024-04-25T01:16:54.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.572+0000 7fb54ea00640 1 --2- 172.21.15.90:0/3278109664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5500d4a10 0x7fb5500b3ee0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb53c009870 tx=0x7fb53c02a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21f894b685f50666 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.572+0000 7f6318a00640 1 -- 172.21.15.90:0/815773153 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f630000b040 con 0x7f631406d670 2024-04-25T01:16:54.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.572+0000 7f6318a00640 1 -- 172.21.15.90:0/815773153 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6300030710 con 0x7f631406d670 2024-04-25T01:16:54.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.572+0000 7f6318a00640 1 -- 172.21.15.90:0/815773153 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6300030ae0 con 0x7f631406d670 2024-04-25T01:16:54.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.572+0000 7fd7b3e00640 1 -- 172.21.15.90:0/1475378106 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd78c001030 con 0x7fd7b40b50e0 2024-04-25T01:16:54.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.573+0000 7fe535400640 1 -- 172.21.15.90:0/2639351029 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe508001030 con 0x7fe5300ae730 2024-04-25T01:16:54.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.573+0000 7f6d36e00640 1 -- start start 2024-04-25T01:16:54.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.573+0000 7fb54e000640 1 -- 172.21.15.90:0/3278109664 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb53c037070 con 0x7fb5500d4a10 2024-04-25T01:16:54.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.573+0000 7fb54e000640 1 -- 172.21.15.90:0/3278109664 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb53c032440 con 0x7fb5500d4a10 2024-04-25T01:16:54.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.573+0000 7fb54e000640 1 -- 172.21.15.90:0/3278109664 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb53c032810 con 0x7fb5500d4a10 2024-04-25T01:16:54.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.573+0000 7f631c800640 1 -- 172.21.15.90:0/815773153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f631406d670 msgr2=0x7f6314071c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.573+0000 7f631c800640 1 --2- 172.21.15.90:0/815773153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f631406d670 0x7f6314071c60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f630000a940 tx=0x7f630002b550 comp rx=0 tx=0).stop 2024-04-25T01:16:54.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.574+0000 7f631c800640 1 -- 172.21.15.90:0/815773153 shutdown_connections 2024-04-25T01:16:54.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.574+0000 7f631c800640 1 --2- 172.21.15.90:0/815773153 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63140721a0 0x7f631415c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.574+0000 7f631c800640 1 --2- 172.21.15.90:0/815773153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f631406d670 0x7f6314071c60 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.574+0000 7f631c800640 1 -- 172.21.15.90:0/815773153 >> 172.21.15.90:0/815773153 conn(0x7f63140cfb30 msgr2=0x7f63140cff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:54.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.574+0000 7f631c800640 1 -- 172.21.15.90:0/815773153 shutdown_connections 2024-04-25T01:16:54.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.574+0000 7f631c800640 1 -- 172.21.15.90:0/815773153 wait complete. 2024-04-25T01:16:54.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.574+0000 7fe973e00640 1 --2- 172.21.15.90:0/1841680286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe97406d670 0x7fe974071c60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe9600097a0 tx=0x7fe96002a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcc89c3579effaec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.574+0000 7fa610e00640 1 -- 172.21.15.90:0/3393952628 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa604005390 con 0x7fa6040e3160 2024-04-25T01:16:54.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.574+0000 7fb556000640 1 -- 172.21.15.90:0/3278109664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5500d4a10 msgr2=0x7fb5500b3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.574+0000 7fb556000640 1 --2- 172.21.15.90:0/3278109664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5500d4a10 0x7fb5500b3ee0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb53c009870 tx=0x7fb53c02a1c0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.574+0000 7fb556000640 1 -- 172.21.15.90:0/3278109664 shutdown_connections 2024-04-25T01:16:54.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.574+0000 7fb556000640 1 --2- 172.21.15.90:0/3278109664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5500d4a10 0x7fb5500b3ee0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.574+0000 7fb556000640 1 --2- 172.21.15.90:0/3278109664 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb55015e4c0 0x7fb5500d2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.574+0000 7fb556000640 1 -- 172.21.15.90:0/3278109664 >> 172.21.15.90:0/3278109664 conn(0x7fb5500d3ef0 msgr2=0x7fb5500d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:54.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.574+0000 7fb556000640 1 -- 172.21.15.90:0/3278109664 shutdown_connections 2024-04-25T01:16:54.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.574+0000 7fb556000640 1 -- 172.21.15.90:0/3278109664 wait complete. 2024-04-25T01:16:54.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.575+0000 7f6d36e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d3006c130 0x7f6d30166410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.575+0000 7f6d36e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d300b4a10 0x7f6d30164710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.575+0000 7f6d36e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d3015a570 0x7f6d3015bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.576+0000 7f6d2fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d3006c130 0x7f6d30166410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.576+0000 7f6d2fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d3006c130 0x7f6d30166410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33628/0 (socket says 172.21.15.90:33628) 2024-04-25T01:16:54.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.576+0000 7f631c800640 1 Processor -- start 2024-04-25T01:16:54.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.576+0000 7f6d36e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6d30067020 con 0x7f6d3006c130 2024-04-25T01:16:54.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.576+0000 7f6d36e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6d30066ea0 con 0x7f6d300b4a10 2024-04-25T01:16:54.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.576+0000 7f6d36e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6d300671a0 con 0x7f6d3015a570 2024-04-25T01:16:54.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.576+0000 7f6d2fe00640 1 -- 172.21.15.90:0/2839498347 learned_addr learned my addr 172.21.15.90:0/2839498347 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:54.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.576+0000 7f6d2f400640 1 --2- 172.21.15.90:0/2839498347 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d300b4a10 0x7f6d30164710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.576+0000 7fd7aaa00640 1 -- 172.21.15.90:0/1475378106 <== mon.2 v2:172.21.15.103:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd7ac01a070 con 0x7fd7b40b50e0 2024-04-25T01:16:54.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.576+0000 7fe973400640 1 -- 172.21.15.90:0/1841680286 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe960037070 con 0x7fe97406d670 2024-04-25T01:16:54.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.576+0000 7fe973400640 1 -- 172.21.15.90:0/1841680286 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe960032440 con 0x7fe97406d670 2024-04-25T01:16:54.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.576+0000 7fe973400640 1 -- 172.21.15.90:0/1841680286 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe960032810 con 0x7fe97406d670 2024-04-25T01:16:54.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.576+0000 7f631c800640 1 -- start start 2024-04-25T01:16:54.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.577+0000 7f6d34e00640 1 --2- 172.21.15.90:0/2839498347 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d3015a570 0x7f6d3015bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.577+0000 7f6d2fe00640 1 -- 172.21.15.90:0/2839498347 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d3015a570 msgr2=0x7f6d3015bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.577+0000 7f6d2fe00640 1 --2- 172.21.15.90:0/2839498347 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d3015a570 0x7f6d3015bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.577+0000 7f6d2fe00640 1 -- 172.21.15.90:0/2839498347 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d300b4a10 msgr2=0x7f6d30164710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.577+0000 7f6d2fe00640 1 --2- 172.21.15.90:0/2839498347 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d300b4a10 0x7f6d30164710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.577+0000 7f6d2fe00640 1 -- 172.21.15.90:0/2839498347 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d301669f0 con 0x7f6d3006c130 2024-04-25T01:16:54.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.577+0000 7fb556000640 1 Processor -- start 2024-04-25T01:16:54.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.577+0000 7fb556000640 1 -- start start 2024-04-25T01:16:54.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.577+0000 7f631c800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f631406d670 0x7f63141f94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.578+0000 7f631c800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63140721a0 0x7f63141fb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.578+0000 7f631c800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63141405d0 0x7f63140cd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.578+0000 7f631c800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f63141409b0 con 0x7f63141405d0 2024-04-25T01:16:54.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.578+0000 7f631c800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f631415ba00 con 0x7f63140721a0 2024-04-25T01:16:54.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.578+0000 7f631c800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f63140d0650 con 0x7f631406d670 2024-04-25T01:16:54.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.578+0000 7fe97b800640 1 -- 172.21.15.90:0/1841680286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe97406d670 msgr2=0x7fe974071c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.578+0000 7fe97b800640 1 --2- 172.21.15.90:0/1841680286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe97406d670 0x7fe974071c60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe9600097a0 tx=0x7fe96002a180 comp rx=0 tx=0).stop 2024-04-25T01:16:54.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.578+0000 7f6d2fe00640 1 --2- 172.21.15.90:0/2839498347 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d3006c130 0x7f6d30166410 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6d1800c460 tx=0x7f6d1800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.578+0000 7fe97b800640 1 -- 172.21.15.90:0/1841680286 shutdown_connections 2024-04-25T01:16:54.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.578+0000 7fe97b800640 1 --2- 172.21.15.90:0/1841680286 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9740721a0 0x7fe97415c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.578+0000 7fe97b800640 1 --2- 172.21.15.90:0/1841680286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe97406d670 0x7fe974071c60 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.578+0000 7fe97b800640 1 -- 172.21.15.90:0/1841680286 >> 172.21.15.90:0/1841680286 conn(0x7fe9740cfb30 msgr2=0x7fe9740cff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:54.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.578+0000 7fa5ff400640 1 -- 172.21.15.90:0/3393952628 <== mon.2 v2:172.21.15.103:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa60c0746e0 con 0x7fa6040e3160 2024-04-25T01:16:54.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.579+0000 7fd7aaa00640 1 --2- 172.21.15.90:0/1475378106 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd79006c030 0x7fd79006e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.577+0000 7fe527e00640 1 -- 172.21.15.90:0/2639351029 <== mon.2 v2:172.21.15.103:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe51c004980 con 0x7fe5300ae730 2024-04-25T01:16:54.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.578+0000 7fe527e00640 1 --2- 172.21.15.90:0/2639351029 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe51006be30 0x7fe51006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.578+0000 7fe97b800640 1 -- 172.21.15.90:0/1841680286 shutdown_connections 2024-04-25T01:16:54.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.579+0000 7fe97b800640 1 -- 172.21.15.90:0/1841680286 wait complete. 2024-04-25T01:16:54.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.579+0000 7f6319e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f631406d670 0x7f63141f94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.579+0000 7f6d27e00640 1 -- 172.21.15.90:0/2839498347 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d18018070 con 0x7f6d3006c130 2024-04-25T01:16:54.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.579+0000 7f6d27e00640 1 -- 172.21.15.90:0/2839498347 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d18007500 con 0x7f6d3006c130 2024-04-25T01:16:54.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.579+0000 7f6d27e00640 1 -- 172.21.15.90:0/2839498347 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d18004a20 con 0x7f6d3006c130 2024-04-25T01:16:54.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.579+0000 7f6d36e00640 1 -- 172.21.15.90:0/2839498347 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d30166cd0 con 0x7f6d3006c130 2024-04-25T01:16:54.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.579+0000 7fe527e00640 1 -- 172.21.15.90:0/2639351029 <== mon.2 v2:172.21.15.103:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fe51c08a970 con 0x7fe5300ae730 2024-04-25T01:16:54.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.579+0000 7fe52fe00640 1 --2- 172.21.15.90:0/2639351029 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe51006be30 0x7fe51006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.579+0000 7f6d36e00640 1 -- 172.21.15.90:0/2839498347 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d30167290 con 0x7f6d3006c130 2024-04-25T01:16:54.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.579+0000 7fd7aaa00640 1 -- 172.21.15.90:0/1475378106 <== mon.2 v2:172.21.15.103:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fd7ac08b310 con 0x7fd7b40b50e0 2024-04-25T01:16:54.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.579+0000 7fa5ff400640 1 --2- 172.21.15.90:0/3393952628 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa5ec06bf00 0x7fa5ec06e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.579+0000 7fa5ff400640 1 -- 172.21.15.90:0/3393952628 <== mon.2 v2:172.21.15.103:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fa60c107460 con 0x7fa6040e3160 2024-04-25T01:16:54.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.579+0000 7fa5ff400640 1 --2- 172.21.15.90:0/3393952628 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x7fa5ec075680 0x7fa5ec077ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.579+0000 7fd7aaa00640 1 --2- 172.21.15.90:0/1475378106 >> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] conn(0x7fd7900757b0 0x7fd790077bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.579+0000 7fd7b2000640 1 --2- 172.21.15.90:0/1475378106 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd79006c030 0x7fd79006e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.580+0000 7fd7aaa00640 1 -- 172.21.15.90:0/1475378106 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd7ac017e70 con 0x7fd7900757b0 2024-04-25T01:16:54.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.580+0000 7fd7aaa00640 1 -- 172.21.15.90:0/1475378106 <== mon.2 v2:172.21.15.103:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd7ac08b700 con 0x7fd7b40b50e0 2024-04-25T01:16:54.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.579+0000 7f6319e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f631406d670 0x7f63141f94e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.90:36884/0 (socket says 172.21.15.90:36884) 2024-04-25T01:16:54.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.580+0000 7fa5ff400640 1 -- 172.21.15.90:0/3393952628 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa5ec078190 con 0x7fa5ec075680 2024-04-25T01:16:54.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.580+0000 7fa5ff400640 1 -- 172.21.15.90:0/3393952628 <== mon.2 v2:172.21.15.103:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa60c107850 con 0x7fa6040e3160 2024-04-25T01:16:54.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.580+0000 7fe527e00640 1 --2- 172.21.15.90:0/2639351029 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x7fe5100755b0 0x7fe5100779f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.580+0000 7fe527e00640 1 -- 172.21.15.90:0/2639351029 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe51c00ae50 con 0x7fe5100755b0 2024-04-25T01:16:54.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.580+0000 7fd7b3400640 1 --2- 172.21.15.90:0/1475378106 >> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] conn(0x7fd7900757b0 0x7fd790077bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.580+0000 7fe527e00640 1 -- 172.21.15.90:0/2639351029 <== mon.2 v2:172.21.15.103:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe51c08ad60 con 0x7fe5300ae730 2024-04-25T01:16:54.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.580+0000 7fb556000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5500d2d40 0x7fb5500cc080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.580+0000 7fb556000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb5500d4a10 0x7fb5501f8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.580+0000 7fb556000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb55015e4c0 0x7fb55015d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.580+0000 7fb556000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb55015ea20 con 0x7fb5500d2d40 2024-04-25T01:16:54.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.580+0000 7fb556000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb55015e8a0 con 0x7fb55015e4c0 2024-04-25T01:16:54.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.580+0000 7fb556000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb5500d1010 con 0x7fb5500d4a10 2024-04-25T01:16:54.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.579+0000 7f631a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63141405d0 0x7f63140cd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.580+0000 7f631a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63141405d0 0x7f63140cd510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33642/0 (socket says 172.21.15.90:33642) 2024-04-25T01:16:54.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.580+0000 7f631a800640 1 -- 172.21.15.90:0/1715763411 learned_addr learned my addr 172.21.15.90:0/1715763411 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:54.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.579+0000 7f6319400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63140721a0 0x7f63141fb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.581+0000 7fb54fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb55015e4c0 0x7fb55015d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.581+0000 7fb54f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5500d2d40 0x7fb5500cc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.581+0000 7fb54fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb55015e4c0 0x7fb55015d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.90:55774/0 (socket says 172.21.15.90:55774) 2024-04-25T01:16:54.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.581+0000 7fb54fe00640 1 -- 172.21.15.90:0/1696744554 learned_addr learned my addr 172.21.15.90:0/1696744554 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:54.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.581+0000 7fe534a00640 1 --2- 172.21.15.90:0/2639351029 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x7fe5100755b0 0x7fe5100779f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.581+0000 7fa60b400640 1 --2- 172.21.15.90:0/3393952628 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa5ec06bf00 0x7fa5ec06e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.581+0000 7fe52fe00640 1 --2- 172.21.15.90:0/2639351029 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe51006be30 0x7fe51006e2f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe520034000 tx=0x7fe52002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.582+0000 7fd7b2000640 1 --2- 172.21.15.90:0/1475378106 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd79006c030 0x7fd79006e4f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd79c034000 tx=0x7fd79c02c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.582+0000 7fb54ea00640 1 --2- 172.21.15.90:0/1696744554 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb5500d4a10 0x7fb5501f8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.582+0000 7fb54fe00640 1 -- 172.21.15.90:0/1696744554 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb5500d4a10 msgr2=0x7fb5501f8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.582+0000 7fe534a00640 1 --2- 172.21.15.90:0/2639351029 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x7fe5100755b0 0x7fe5100779f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.582+0000 7fb54fe00640 1 --2- 172.21.15.90:0/1696744554 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb5500d4a10 0x7fb5501f8c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.582+0000 7fb54fe00640 1 -- 172.21.15.90:0/1696744554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5500d2d40 msgr2=0x7fb5500cc080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.582+0000 7fb54fe00640 1 --2- 172.21.15.90:0/1696744554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5500d2d40 0x7fb5500cc080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.582+0000 7fb54fe00640 1 -- 172.21.15.90:0/1696744554 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5501fb810 con 0x7fb55015e4c0 2024-04-25T01:16:54.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.582+0000 7fb54f400640 1 --2- 172.21.15.90:0/1696744554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5500d2d40 0x7fb5500cc080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:16:54.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.583+0000 7fe97b800640 1 Processor -- start 2024-04-25T01:16:54.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.583+0000 7fa60aa00640 1 --2- 172.21.15.90:0/3393952628 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x7fa5ec075680 0x7fa5ec077ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.583+0000 7fa60b400640 1 --2- 172.21.15.90:0/3393952628 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa5ec06bf00 0x7fa5ec06e3c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa600005880 tx=0x7fa600009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.583+0000 7fe97b800640 1 -- start start 2024-04-25T01:16:54.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.583+0000 7fb54fe00640 1 --2- 172.21.15.90:0/1696744554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb55015e4c0 0x7fb55015d490 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb54800c6c0 tx=0x7fb54800cb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.582+0000 7f631a800640 1 -- 172.21.15.90:0/1715763411 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f631406d670 msgr2=0x7f63141f94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.582+0000 7f631a800640 1 --2- 172.21.15.90:0/1715763411 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f631406d670 0x7f63141f94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.582+0000 7f631a800640 1 -- 172.21.15.90:0/1715763411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63140721a0 msgr2=0x7f63141fb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.582+0000 7f631a800640 1 --2- 172.21.15.90:0/1715763411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63140721a0 0x7f63141fb0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.582+0000 7f631a800640 1 -- 172.21.15.90:0/1715763411 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63141f8890 con 0x7f63141405d0 2024-04-25T01:16:54.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.583+0000 7f6d36e00640 1 -- 172.21.15.90:0/2839498347 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6d30068fb0 con 0x7f6d3006c130 2024-04-25T01:16:54.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.584+0000 7fb547400640 1 -- 172.21.15.90:0/1696744554 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb548003b70 con 0x7fb55015e4c0 2024-04-25T01:16:54.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.584+0000 7fe527e00640 1 -- 172.21.15.90:0/2639351029 <== osd.3 v2:172.21.15.90:6812/8683104 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe51c00ae50 con 0x7fe5100755b0 2024-04-25T01:16:54.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.584+0000 7fb547400640 1 -- 172.21.15.90:0/1696744554 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb548003d10 con 0x7fb55015e4c0 2024-04-25T01:16:54.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.584+0000 7fb547400640 1 -- 172.21.15.90:0/1696744554 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb548005310 con 0x7fb55015e4c0 2024-04-25T01:16:54.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.584+0000 7fb556000640 1 -- 172.21.15.90:0/1696744554 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5501fbaf0 con 0x7fb55015e4c0 2024-04-25T01:16:54.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.584+0000 7fb556000640 1 -- 172.21.15.90:0/1696744554 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5501fc0b0 con 0x7fb55015e4c0 2024-04-25T01:16:54.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.584+0000 7fd7b3400640 1 --2- 172.21.15.90:0/1475378106 >> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] conn(0x7fd7900757b0 0x7fd790077bf0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.584+0000 7f6d27e00640 1 -- 172.21.15.90:0/2839498347 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6d1801b070 con 0x7f6d3006c130 2024-04-25T01:16:54.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.584+0000 7fe97b800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe97406d670 0x7fe9741368c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.584+0000 7fe97b800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9740721a0 0x7fe974159910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.585+0000 7fe97b800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9741405d0 0x7fe974142530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.585+0000 7f631a800640 1 --2- 172.21.15.90:0/1715763411 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63141405d0 0x7f63140cd510 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f631000e930 tx=0x7f631000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.585+0000 7fe978e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe97406d670 0x7fe9741368c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.586+0000 7f630e000640 1 -- 172.21.15.90:0/1715763411 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6310017070 con 0x7f63141405d0 2024-04-25T01:16:54.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.586+0000 7f630e000640 1 -- 172.21.15.90:0/1715763411 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f631000c980 con 0x7f63141405d0 2024-04-25T01:16:54.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.586+0000 7f630e000640 1 -- 172.21.15.90:0/1715763411 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f631000cd70 con 0x7f63141405d0 2024-04-25T01:16:54.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.585+0000 7fe973e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9740721a0 0x7fe974159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.585+0000 7fe973e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9740721a0 0x7fe974159910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33650/0 (socket says 172.21.15.90:33650) 2024-04-25T01:16:54.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.586+0000 7fe978e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe97406d670 0x7fe9741368c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.90:55782/0 (socket says 172.21.15.90:55782) 2024-04-25T01:16:54.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.586+0000 7fe97b800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe9741409b0 con 0x7fe9740721a0 2024-04-25T01:16:54.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.586+0000 7fe97b800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe97415ba00 con 0x7fe97406d670 2024-04-25T01:16:54.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.586+0000 7fe97b800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe9740d0650 con 0x7fe9741405d0 2024-04-25T01:16:54.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.586+0000 7fb556000640 1 -- 172.21.15.90:0/1696744554 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb5501e7130 con 0x7fb55015e4c0 2024-04-25T01:16:54.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.586+0000 7f6d27e00640 1 --2- 172.21.15.90:0/2839498347 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f6d0c06be80 0x7f6d0c06e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.586+0000 7fe973e00640 1 -- 172.21.15.90:0/702694705 learned_addr learned my addr 172.21.15.90:0/702694705 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:54.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.586+0000 7f631c800640 1 -- 172.21.15.90:0/1715763411 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63141f8b70 con 0x7f63141405d0 2024-04-25T01:16:54.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.584+0000 7fa60aa00640 1 --2- 172.21.15.90:0/3393952628 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x7fa5ec075680 0x7fa5ec077ac0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.586+0000 7f6d27e00640 1 -- 172.21.15.90:0/2839498347 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f6d1808a420 con 0x7f6d3006c130 2024-04-25T01:16:54.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.587+0000 7f6d27e00640 1 --2- 172.21.15.90:0/2839498347 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x7f6d0c075570 0x7f6d0c0779b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.587+0000 7f6d27e00640 1 -- 172.21.15.90:0/2839498347 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6d18017ec0 con 0x7f6d0c075570 2024-04-25T01:16:54.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.587+0000 7f6d27e00640 1 -- 172.21.15.90:0/2839498347 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6d18091050 con 0x7f6d3006c130 2024-04-25T01:16:54.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.587+0000 7f6d2f400640 1 --2- 172.21.15.90:0/2839498347 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f6d0c06be80 0x7f6d0c06e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.587+0000 7fe973e00640 1 -- 172.21.15.90:0/702694705 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9741405d0 msgr2=0x7fe974142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.587+0000 7fe979800640 1 --2- 172.21.15.90:0/702694705 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9741405d0 0x7fe974142530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.586+0000 7fd7aaa00640 1 -- 172.21.15.90:0/1475378106 <== osd.2 v2:172.21.15.90:6808/906792737 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd7ac017e70 con 0x7fd7900757b0 2024-04-25T01:16:54.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.587+0000 7f6d2f400640 1 --2- 172.21.15.90:0/2839498347 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f6d0c06be80 0x7f6d0c06e340 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6d30074990 tx=0x7f6d20008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.588+0000 7f631c800640 1 -- 172.21.15.90:0/1715763411 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63141f9130 con 0x7f63141405d0 2024-04-25T01:16:54.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.588+0000 7fe973e00640 1 --2- 172.21.15.90:0/702694705 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9741405d0 0x7fe974142530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.588+0000 7f6d34e00640 1 --2- 172.21.15.90:0/2839498347 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x7f6d0c075570 0x7f6d0c0779b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.588+0000 7fa5ff400640 1 -- 172.21.15.90:0/3393952628 <== osd.1 v2:172.21.15.90:6800/2685473653 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa5ec078190 con 0x7fa5ec075680 2024-04-25T01:16:54.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.588+0000 7fe973e00640 1 -- 172.21.15.90:0/702694705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe97406d670 msgr2=0x7fe9741368c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.588+0000 7fe973e00640 1 --2- 172.21.15.90:0/702694705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe97406d670 0x7fe9741368c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.588+0000 7fe973e00640 1 -- 172.21.15.90:0/702694705 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe974159e50 con 0x7fe9740721a0 2024-04-25T01:16:54.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.588+0000 7fe979800640 1 --2- 172.21.15.90:0/702694705 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9741405d0 0x7fe974142530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:54.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.588+0000 7fe973e00640 1 --2- 172.21.15.90:0/702694705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9740721a0 0x7fe974159910 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe96002a770 tx=0x7fe9600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.589+0000 7f6d34e00640 1 --2- 172.21.15.90:0/2839498347 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x7f6d0c075570 0x7f6d0c0779b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.590+0000 7fe970c00640 1 -- 172.21.15.90:0/702694705 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe960037050 con 0x7fe9740721a0 2024-04-25T01:16:54.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.590+0000 7fe970c00640 1 -- 172.21.15.90:0/702694705 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe960043070 con 0x7fe9740721a0 2024-04-25T01:16:54.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.590+0000 7fe970c00640 1 -- 172.21.15.90:0/702694705 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96003cb30 con 0x7fe9740721a0 2024-04-25T01:16:54.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.590+0000 7fe97b800640 1 -- 172.21.15.90:0/702694705 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe974135b00 con 0x7fe9740721a0 2024-04-25T01:16:54.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.590+0000 7fe97b800640 1 -- 172.21.15.90:0/702694705 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9741360c0 con 0x7fe9740721a0 2024-04-25T01:16:54.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.590+0000 7f6307400640 1 -- 172.21.15.90:0/1715763411 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f62e8001030 con 0x7f63141405d0 2024-04-25T01:16:54.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.592+0000 7f630e000640 1 -- 172.21.15.90:0/1715763411 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f631001a070 con 0x7f63141405d0 2024-04-25T01:16:54.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.592+0000 7f6d27e00640 1 -- 172.21.15.90:0/2839498347 <== osd.4 v2:172.21.15.103:6821/1971061680 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6d18017ec0 con 0x7f6d0c075570 2024-04-25T01:16:54.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.592+0000 7fb547400640 1 -- 172.21.15.90:0/1696744554 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb5480054b0 con 0x7fb55015e4c0 2024-04-25T01:16:54.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.593+0000 7fe96a000640 1 -- 172.21.15.90:0/702694705 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe948001030 con 0x7fe9740721a0 2024-04-25T01:16:54.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.593+0000 7f630e000640 1 --2- 172.21.15.90:0/1715763411 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f62f406be80 0x7f62f406e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.593+0000 7f630e000640 1 -- 172.21.15.90:0/1715763411 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f631008b3b0 con 0x7f63141405d0 2024-04-25T01:16:54.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.594+0000 7f630e000640 1 --2- 172.21.15.90:0/1715763411 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x7f62f40755f0 0x7f62f4077a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.594+0000 7f630e000640 1 -- 172.21.15.90:0/1715763411 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f631001fe50 con 0x7f62f40755f0 2024-04-25T01:16:54.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.594+0000 7f630e000640 1 -- 172.21.15.90:0/1715763411 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6310092050 con 0x7f63141405d0 2024-04-25T01:16:54.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.594+0000 7f6319e00640 1 --2- 172.21.15.90:0/1715763411 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f62f406be80 0x7f62f406e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.594+0000 7fb547400640 1 --2- 172.21.15.90:0/1696744554 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb53006bf00 0x7fb53006e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.595+0000 7f6319400640 1 --2- 172.21.15.90:0/1715763411 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x7f62f40755f0 0x7f62f4077a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.595+0000 7fb547400640 1 -- 172.21.15.90:0/1696744554 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fb54808b9e0 con 0x7fb55015e4c0 2024-04-25T01:16:54.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.595+0000 7fb547400640 1 --2- 172.21.15.90:0/1696744554 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x7fb530075680 0x7fb530077ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.595+0000 7fb547400640 1 -- 172.21.15.90:0/1696744554 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb54801fec0 con 0x7fb530075680 2024-04-25T01:16:54.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.596+0000 7f6319e00640 1 --2- 172.21.15.90:0/1715763411 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f62f406be80 0x7f62f406e340 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f631415b700 tx=0x7f6308006340 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.595+0000 7fb54f400640 1 --2- 172.21.15.90:0/1696744554 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb53006bf00 0x7fb53006e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.596+0000 7fb547400640 1 -- 172.21.15.90:0/1696744554 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb54808bdd0 con 0x7fb55015e4c0 2024-04-25T01:16:54.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.596+0000 7fb54ea00640 1 --2- 172.21.15.90:0/1696744554 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x7fb530075680 0x7fb530077ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.596+0000 7f6319400640 1 --2- 172.21.15.90:0/1715763411 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x7f62f40755f0 0x7f62f4077a30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.597+0000 7fb54f400640 1 --2- 172.21.15.90:0/1696744554 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb53006bf00 0x7fb53006e3c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb5501e6840 tx=0x7fb540009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.598+0000 7fb54ea00640 1 --2- 172.21.15.90:0/1696744554 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x7fb530075680 0x7fb530077ac0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.599+0000 7f630e000640 1 -- 172.21.15.90:0/1715763411 <== osd.5 v2:172.21.15.103:6801/1348846168 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f631001fe50 con 0x7f62f40755f0 2024-04-25T01:16:54.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.599+0000 7fe970c00640 1 -- 172.21.15.90:0/702694705 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe96003ccd0 con 0x7fe9740721a0 2024-04-25T01:16:54.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.600+0000 7f563dc00640 1 -- 172.21.15.90:0/26083863 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f56380b4c70 con 0x7f56140754b0 2024-04-25T01:16:54.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.600+0000 7fb547400640 1 -- 172.21.15.90:0/1696744554 <== osd.7 v2:172.21.15.103:6800/1145679154 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb54801fec0 con 0x7fb530075680 2024-04-25T01:16:54.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.600+0000 7fe970c00640 1 --2- 172.21.15.90:0/702694705 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe95406bf00 0x7fe95406e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.601+0000 7fe970c00640 1 -- 172.21.15.90:0/702694705 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fe9600ac740 con 0x7fe9740721a0 2024-04-25T01:16:54.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.601+0000 7f562fe00640 1 -- 172.21.15.90:0/26083863 <== osd.0 v2:172.21.15.90:6816/2625481430 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5630007a00 con 0x7f56140754b0 2024-04-25T01:16:54.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.601+0000 7fe970c00640 1 --2- 172.21.15.90:0/702694705 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x7fe954075680 0x7fe954077ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.601+0000 7fe970c00640 1 -- 172.21.15.90:0/702694705 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe960046e80 con 0x7fe954075680 2024-04-25T01:16:54.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.601+0000 7fe970c00640 1 -- 172.21.15.90:0/702694705 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe9600acb30 con 0x7fe9740721a0 2024-04-25T01:16:54.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.602+0000 7f563dc00640 1 -- 172.21.15.90:0/26083863 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x7f56140754b0 msgr2=0x7f56140778f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.602+0000 7f563dc00640 1 --2- 172.21.15.90:0/26083863 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x7f56140754b0 0x7f56140778f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.601+0000 7fe978e00640 1 --2- 172.21.15.90:0/702694705 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe95406bf00 0x7fe95406e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.602+0000 7f563dc00640 1 -- 172.21.15.90:0/26083863 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f561406be30 msgr2=0x7f561406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.602+0000 7f563dc00640 1 --2- 172.21.15.90:0/26083863 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f561406be30 0x7f561406e2f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5620035000 tx=0x7f5620030000 comp rx=0 tx=0).stop 2024-04-25T01:16:54.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.602+0000 7f563dc00640 1 -- 172.21.15.90:0/26083863 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56380aea50 msgr2=0x7f5638159770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.602+0000 7f563dc00640 1 --2- 172.21.15.90:0/26083863 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56380aea50 0x7f5638159770 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f562800c3e0 tx=0x7f562800c8a0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.602+0000 7f563d200640 1 -- 172.21.15.90:0/26083863 reap_dead start 2024-04-25T01:16:54.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.602+0000 7f563dc00640 1 -- 172.21.15.90:0/26083863 shutdown_connections 2024-04-25T01:16:54.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.602+0000 7f563dc00640 1 -- 172.21.15.90:0/26083863 >> 172.21.15.90:0/26083863 conn(0x7f56380b1660 msgr2=0x7f563815e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:54.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.602+0000 7fe979800640 1 --2- 172.21.15.90:0/702694705 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x7fe954075680 0x7fe954077ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.603+0000 7fe978e00640 1 --2- 172.21.15.90:0/702694705 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe95406bf00 0x7fe95406e3c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe9640011e0 tx=0x7fe964007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.603+0000 7f563dc00640 1 -- 172.21.15.90:0/26083863 shutdown_connections 2024-04-25T01:16:54.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.603+0000 7f563dc00640 1 -- 172.21.15.90:0/26083863 wait complete. 2024-04-25T01:16:54.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.604+0000 7fe979800640 1 --2- 172.21.15.90:0/702694705 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x7fe954075680 0x7fe954077ac0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.606+0000 7fe970c00640 1 -- 172.21.15.90:0/702694705 <== osd.6 v2:172.21.15.103:6816/2008388906 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe960046e80 con 0x7fe954075680 2024-04-25T01:16:54.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.611+0000 7fe535400640 1 -- 172.21.15.90:0/2639351029 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe508002dd0 con 0x7fe5100755b0 2024-04-25T01:16:54.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.612+0000 7fe527e00640 1 -- 172.21.15.90:0/2639351029 <== osd.3 v2:172.21.15.90:6812/8683104 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe508002dd0 con 0x7fe5100755b0 2024-04-25T01:16:54.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.613+0000 7fe535400640 1 -- 172.21.15.90:0/2639351029 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x7fe5100755b0 msgr2=0x7fe5100779f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.613+0000 7fe535400640 1 --2- 172.21.15.90:0/2639351029 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x7fe5100755b0 0x7fe5100779f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.615+0000 7fe535400640 1 -- 172.21.15.90:0/2639351029 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe51006be30 msgr2=0x7fe51006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.615+0000 7fe535400640 1 --2- 172.21.15.90:0/2639351029 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe51006be30 0x7fe51006e2f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe520034000 tx=0x7fe52002e000 comp rx=0 tx=0).stop 2024-04-25T01:16:54.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.615+0000 7fe535400640 1 -- 172.21.15.90:0/2639351029 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe5300ae730 msgr2=0x7fe5301612f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.615+0000 7fe535400640 1 --2- 172.21.15.90:0/2639351029 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe5300ae730 0x7fe5301612f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe51c00c480 tx=0x7fe51c00c940 comp rx=0 tx=0).stop 2024-04-25T01:16:54.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.615+0000 7fe534a00640 1 -- 172.21.15.90:0/2639351029 reap_dead start 2024-04-25T01:16:54.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.615+0000 7fe535400640 1 -- 172.21.15.90:0/2639351029 shutdown_connections 2024-04-25T01:16:54.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.615+0000 7fe535400640 1 -- 172.21.15.90:0/2639351029 >> 172.21.15.90:0/2639351029 conn(0x7fe5300b1660 msgr2=0x7fe530070a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:54.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.617+0000 7fe535400640 1 -- 172.21.15.90:0/2639351029 shutdown_connections 2024-04-25T01:16:54.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.617+0000 7fe535400640 1 -- 172.21.15.90:0/2639351029 wait complete. 2024-04-25T01:16:54.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.624+0000 7fd7b3e00640 1 -- 172.21.15.90:0/1475378106 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd78c002dd0 con 0x7fd7900757b0 2024-04-25T01:16:54.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.624+0000 7fa610e00640 1 -- 172.21.15.90:0/3393952628 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa604056bf0 con 0x7fa5ec075680 2024-04-25T01:16:54.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.625+0000 7fa5ff400640 1 -- 172.21.15.90:0/3393952628 <== osd.1 v2:172.21.15.90:6800/2685473653 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa604056bf0 con 0x7fa5ec075680 2024-04-25T01:16:54.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.625+0000 7f6d36e00640 1 -- 172.21.15.90:0/2839498347 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6d301491f0 con 0x7f6d0c075570 2024-04-25T01:16:54.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.626+0000 7fa610e00640 1 -- 172.21.15.90:0/3393952628 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x7fa5ec075680 msgr2=0x7fa5ec077ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.626+0000 7fa610e00640 1 --2- 172.21.15.90:0/3393952628 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x7fa5ec075680 0x7fa5ec077ac0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.626+0000 7fa610e00640 1 -- 172.21.15.90:0/3393952628 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa5ec06bf00 msgr2=0x7fa5ec06e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.626+0000 7fa610e00640 1 --2- 172.21.15.90:0/3393952628 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa5ec06bf00 0x7fa5ec06e3c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa600005880 tx=0x7fa600009040 comp rx=0 tx=0).stop 2024-04-25T01:16:54.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.626+0000 7fa610e00640 1 -- 172.21.15.90:0/3393952628 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa6040e3160 msgr2=0x7fa6040e56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.626+0000 7fa610e00640 1 --2- 172.21.15.90:0/3393952628 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa6040e3160 0x7fa6040e56f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa60c064c60 tx=0x7fa60c073800 comp rx=0 tx=0).stop 2024-04-25T01:16:54.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.627+0000 7fa60be00640 1 -- 172.21.15.90:0/3393952628 reap_dead start 2024-04-25T01:16:54.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.627+0000 7fa610e00640 1 -- 172.21.15.90:0/3393952628 shutdown_connections 2024-04-25T01:16:54.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.627+0000 7fa610e00640 1 -- 172.21.15.90:0/3393952628 >> 172.21.15.90:0/3393952628 conn(0x7fa604058ee0 msgr2=0x7fa604039450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:54.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.627+0000 7f6307400640 1 -- 172.21.15.90:0/1715763411 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f62e8002dd0 con 0x7f62f40755f0 2024-04-25T01:16:54.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.627+0000 7f6d27e00640 1 -- 172.21.15.90:0/2839498347 <== osd.4 v2:172.21.15.103:6821/1971061680 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6d301491f0 con 0x7f6d0c075570 2024-04-25T01:16:54.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.628+0000 7f6d36e00640 1 -- 172.21.15.90:0/2839498347 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x7f6d0c075570 msgr2=0x7f6d0c0779b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.628+0000 7f6d36e00640 1 --2- 172.21.15.90:0/2839498347 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x7f6d0c075570 0x7f6d0c0779b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.628+0000 7f6d36e00640 1 -- 172.21.15.90:0/2839498347 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f6d0c06be80 msgr2=0x7f6d0c06e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.628+0000 7f6d36e00640 1 --2- 172.21.15.90:0/2839498347 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f6d0c06be80 0x7f6d0c06e340 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6d30074990 tx=0x7f6d20008040 comp rx=0 tx=0).stop 2024-04-25T01:16:54.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.628+0000 7f6d36e00640 1 -- 172.21.15.90:0/2839498347 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d3006c130 msgr2=0x7f6d30166410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.628+0000 7f6d36e00640 1 --2- 172.21.15.90:0/2839498347 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d3006c130 0x7f6d30166410 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6d1800c460 tx=0x7f6d1800c920 comp rx=0 tx=0).stop 2024-04-25T01:16:54.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.629+0000 7f6d34e00640 1 -- 172.21.15.90:0/2839498347 reap_dead start 2024-04-25T01:16:54.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.627+0000 7fd7aaa00640 1 -- 172.21.15.90:0/1475378106 <== osd.2 v2:172.21.15.90:6808/906792737 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd78c002dd0 con 0x7fd7900757b0 2024-04-25T01:16:54.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.629+0000 7f630e000640 1 -- 172.21.15.90:0/1715763411 <== osd.5 v2:172.21.15.103:6801/1348846168 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f62e8002dd0 con 0x7f62f40755f0 2024-04-25T01:16:54.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.630+0000 7f6307400640 1 -- 172.21.15.90:0/1715763411 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x7f62f40755f0 msgr2=0x7f62f4077a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.630+0000 7f6307400640 1 --2- 172.21.15.90:0/1715763411 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x7f62f40755f0 0x7f62f4077a30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.630+0000 7f6307400640 1 -- 172.21.15.90:0/1715763411 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f62f406be80 msgr2=0x7f62f406e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.630+0000 7fd7b3e00640 1 -- 172.21.15.90:0/1475378106 >> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] conn(0x7fd7900757b0 msgr2=0x7fd790077bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.630+0000 7fd7b3e00640 1 --2- 172.21.15.90:0/1475378106 >> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] conn(0x7fd7900757b0 0x7fd790077bf0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.630+0000 7f6307400640 1 --2- 172.21.15.90:0/1715763411 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f62f406be80 0x7f62f406e340 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f631415b700 tx=0x7f6308006340 comp rx=0 tx=0).stop 2024-04-25T01:16:54.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.630+0000 7f6307400640 1 -- 172.21.15.90:0/1715763411 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63141405d0 msgr2=0x7f63140cd510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.630+0000 7f6307400640 1 --2- 172.21.15.90:0/1715763411 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63141405d0 0x7f63140cd510 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f631000e930 tx=0x7f631000edf0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.630+0000 7f631a800640 1 -- 172.21.15.90:0/1715763411 reap_dead start 2024-04-25T01:16:54.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.630+0000 7fa610e00640 1 -- 172.21.15.90:0/3393952628 shutdown_connections 2024-04-25T01:16:54.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.630+0000 7fa610e00640 1 -- 172.21.15.90:0/3393952628 wait complete. 2024-04-25T01:16:54.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.631+0000 7f6d36e00640 1 -- 172.21.15.90:0/2839498347 shutdown_connections 2024-04-25T01:16:54.649 INFO:teuthology.orchestra.run.smithi090.stdout:60129542154 2024-04-25T01:16:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.631+0000 7f6d36e00640 1 -- 172.21.15.90:0/2839498347 >> 172.21.15.90:0/2839498347 conn(0x7f6d30071550 msgr2=0x7f6d3006e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:54.650 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-25T01:16:54.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.631+0000 7fd7b3e00640 1 -- 172.21.15.90:0/1475378106 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd79006c030 msgr2=0x7fd79006e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.631+0000 7fd7b3e00640 1 --2- 172.21.15.90:0/1475378106 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd79006c030 0x7fd79006e4f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd79c034000 tx=0x7fd79c02c000 comp rx=0 tx=0).stop 2024-04-25T01:16:54.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.631+0000 7fd7b3e00640 1 -- 172.21.15.90:0/1475378106 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7b40b50e0 msgr2=0x7fd7b415ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.631+0000 7fd7b3e00640 1 --2- 172.21.15.90:0/1475378106 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7b40b50e0 0x7fd7b415ca90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac00cb40 tx=0x7fd7ac00a060 comp rx=0 tx=0).stop 2024-04-25T01:16:54.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.631+0000 7f6307400640 1 -- 172.21.15.90:0/1715763411 shutdown_connections 2024-04-25T01:16:54.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.631+0000 7f6307400640 1 -- 172.21.15.90:0/1715763411 >> 172.21.15.90:0/1715763411 conn(0x7f63140cfb30 msgr2=0x7f631415c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:54.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.631+0000 7fd7b3400640 1 -- 172.21.15.90:0/1475378106 reap_dead start 2024-04-25T01:16:54.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.632+0000 7fd7b3e00640 1 -- 172.21.15.90:0/1475378106 shutdown_connections 2024-04-25T01:16:54.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.632+0000 7fd7b3e00640 1 -- 172.21.15.90:0/1475378106 >> 172.21.15.90:0/1475378106 conn(0x7fd7b409d070 msgr2=0x7fd7b40b58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:54.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.632+0000 7f6d36e00640 1 -- 172.21.15.90:0/2839498347 shutdown_connections 2024-04-25T01:16:54.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.632+0000 7f6d36e00640 1 -- 172.21.15.90:0/2839498347 wait complete. 2024-04-25T01:16:54.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.632+0000 7f6307400640 1 -- 172.21.15.90:0/1715763411 shutdown_connections 2024-04-25T01:16:54.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.632+0000 7fd7b3e00640 1 -- 172.21.15.90:0/1475378106 shutdown_connections 2024-04-25T01:16:54.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.632+0000 7fd7b3e00640 1 -- 172.21.15.90:0/1475378106 wait complete. 2024-04-25T01:16:54.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.632+0000 7f6307400640 1 -- 172.21.15.90:0/1715763411 wait complete. 2024-04-25T01:16:54.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.638+0000 7fe96a000640 1 -- 172.21.15.90:0/702694705 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe948002dd0 con 0x7fe954075680 2024-04-25T01:16:54.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.639+0000 7fe970c00640 1 -- 172.21.15.90:0/702694705 <== osd.6 v2:172.21.15.103:6816/2008388906 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe948002dd0 con 0x7fe954075680 2024-04-25T01:16:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.640+0000 7fe96a000640 1 -- 172.21.15.90:0/702694705 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x7fe954075680 msgr2=0x7fe954077ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.640+0000 7fe96a000640 1 --2- 172.21.15.90:0/702694705 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x7fe954075680 0x7fe954077ac0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.640+0000 7fe96a000640 1 -- 172.21.15.90:0/702694705 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe95406bf00 msgr2=0x7fe95406e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.641+0000 7fe96a000640 1 --2- 172.21.15.90:0/702694705 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe95406bf00 0x7fe95406e3c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe9640011e0 tx=0x7fe964007d30 comp rx=0 tx=0).stop 2024-04-25T01:16:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.641+0000 7fe96a000640 1 -- 172.21.15.90:0/702694705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9740721a0 msgr2=0x7fe974159910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.641+0000 7fe96a000640 1 --2- 172.21.15.90:0/702694705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9740721a0 0x7fe974159910 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe96002a770 tx=0x7fe9600046c0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.641+0000 7fe979800640 1 -- 172.21.15.90:0/702694705 reap_dead start 2024-04-25T01:16:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.641+0000 7fe96a000640 1 -- 172.21.15.90:0/702694705 shutdown_connections 2024-04-25T01:16:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.641+0000 7fe96a000640 1 -- 172.21.15.90:0/702694705 >> 172.21.15.90:0/702694705 conn(0x7fe9740cfb30 msgr2=0x7fe9740b10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:54.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.641+0000 7fb556000640 1 -- 172.21.15.90:0/1696744554 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb550068fb0 con 0x7fb530075680 2024-04-25T01:16:54.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.642+0000 7fe96a000640 1 -- 172.21.15.90:0/702694705 shutdown_connections 2024-04-25T01:16:54.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.642+0000 7fb547400640 1 -- 172.21.15.90:0/1696744554 <== osd.7 v2:172.21.15.103:6800/1145679154 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb550068fb0 con 0x7fb530075680 2024-04-25T01:16:54.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.643+0000 7fe96a000640 1 -- 172.21.15.90:0/702694705 wait complete. 2024-04-25T01:16:54.657 INFO:teuthology.orchestra.run.smithi090.stdout:60129542153 2024-04-25T01:16:54.658 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-25T01:16:54.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.644+0000 7fb556000640 1 -- 172.21.15.90:0/1696744554 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x7fb530075680 msgr2=0x7fb530077ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.644+0000 7fb556000640 1 --2- 172.21.15.90:0/1696744554 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x7fb530075680 0x7fb530077ac0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.645+0000 7fb556000640 1 -- 172.21.15.90:0/1696744554 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb53006bf00 msgr2=0x7fb53006e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.645+0000 7fb556000640 1 --2- 172.21.15.90:0/1696744554 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb53006bf00 0x7fb53006e3c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb5501e6840 tx=0x7fb540009190 comp rx=0 tx=0).stop 2024-04-25T01:16:54.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.645+0000 7fb556000640 1 -- 172.21.15.90:0/1696744554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb55015e4c0 msgr2=0x7fb55015d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.645+0000 7fb556000640 1 --2- 172.21.15.90:0/1696744554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb55015e4c0 0x7fb55015d490 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb54800c6c0 tx=0x7fb54800cb80 comp rx=0 tx=0).stop 2024-04-25T01:16:54.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.645+0000 7fb54fe00640 1 -- 172.21.15.90:0/1696744554 reap_dead start 2024-04-25T01:16:54.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.645+0000 7fb556000640 1 -- 172.21.15.90:0/1696744554 shutdown_connections 2024-04-25T01:16:54.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.645+0000 7fb556000640 1 -- 172.21.15.90:0/1696744554 >> 172.21.15.90:0/1696744554 conn(0x7fb5500d3ef0 msgr2=0x7fb55006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:54.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.646+0000 7fb556000640 1 -- 172.21.15.90:0/1696744554 shutdown_connections 2024-04-25T01:16:54.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.646+0000 7fb556000640 1 -- 172.21.15.90:0/1696744554 wait complete. 2024-04-25T01:16:54.661 INFO:teuthology.orchestra.run.smithi090.stdout:60129542154 2024-04-25T01:16:54.661 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-25T01:16:54.662 INFO:teuthology.orchestra.run.smithi090.stdout:60129542153 2024-04-25T01:16:54.662 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-25T01:16:54.663 INFO:teuthology.orchestra.run.smithi090.stdout:60129542153 2024-04-25T01:16:54.663 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-25T01:16:54.664 INFO:teuthology.orchestra.run.smithi090.stdout:60129542155 2024-04-25T01:16:54.664 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-25T01:16:54.665 INFO:teuthology.orchestra.run.smithi090.stdout:60129542153 2024-04-25T01:16:54.666 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-25T01:16:54.667 INFO:teuthology.orchestra.run.smithi090.stdout:60129542154 2024-04-25T01:16:54.667 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-25T01:16:54.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.987+0000 7fee6c200640 1 Processor -- start 2024-04-25T01:16:54.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.987+0000 7fee6c200640 1 -- start start 2024-04-25T01:16:54.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.989+0000 7fee6c200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fee5c048390 0x7fee5c006f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.990+0000 7fee6c200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee5c0e7ba0 con 0x7fee5c006070 2024-04-25T01:16:54.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.990+0000 7fee6c200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee5c007690 con 0x7fee5c04fc10 2024-04-25T01:16:54.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.990+0000 7fee6c200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fee5c054ae0 con 0x7fee5c048390 2024-04-25T01:16:54.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.990+0000 7fee6ac00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fee5c04fc10 0x7fee5c0389b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.90:55244/0 (socket says 172.21.15.90:55244) 2024-04-25T01:16:54.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.991+0000 7fee6ac00640 1 -- 172.21.15.90:0/1420058695 learned_addr learned my addr 172.21.15.90:0/1420058695 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:54.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.992+0000 7fee63e00640 1 -- 172.21.15.90:0/1420058695 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1035271752 0 0) 0x7fee5c0e7ba0 con 0x7fee5c006070 2024-04-25T01:16:54.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.992+0000 7fee63e00640 1 -- 172.21.15.90:0/1420058695 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee4c003520 con 0x7fee5c006070 2024-04-25T01:16:54.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.992+0000 7fee63e00640 1 -- 172.21.15.90:0/1420058695 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3531377865 0 0) 0x7fee5c007690 con 0x7fee5c04fc10 2024-04-25T01:16:54.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.992+0000 7fee63e00640 1 -- 172.21.15.90:0/1420058695 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee4c003880 con 0x7fee5c04fc10 2024-04-25T01:16:54.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.992+0000 7f8a0da00640 1 Processor -- start 2024-04-25T01:16:54.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.993+0000 7fee63e00640 1 -- 172.21.15.90:0/1420058695 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3602907934 0 0) 0x7fee4c003520 con 0x7fee5c006070 2024-04-25T01:16:54.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.993+0000 7fee63e00640 1 -- 172.21.15.90:0/1420058695 >> v2:172.21.15.103:3300/0 conn(0x7fee5c048390 msgr2=0x7fee5c006f60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:16:54.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.993+0000 7fee63e00640 1 --2- 172.21.15.90:0/1420058695 >> v2:172.21.15.103:3300/0 conn(0x7fee5c048390 0x7fee5c006f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.993+0000 7fee63e00640 1 -- 172.21.15.90:0/1420058695 >> v1:172.21.15.103:6789/0 conn(0x7fee5c04fc10 legacy=0x7fee5c0389b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.993+0000 7fee63e00640 1 -- 172.21.15.90:0/1420058695 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee5c007970 con 0x7fee5c006070 2024-04-25T01:16:54.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.993+0000 7f8a0da00640 1 -- start start 2024-04-25T01:16:54.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.993+0000 7fee63e00640 1 -- 172.21.15.90:0/1420058695 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7fee58001650 con 0x7fee5c006070 2024-04-25T01:16:54.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.993+0000 7fee63e00640 1 -- 172.21.15.90:0/1420058695 >> v1:172.21.15.90:6789/0 conn(0x7fee5c006070 legacy=0x7fee5c04f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.993+0000 7fee63e00640 1 --2- 172.21.15.90:0/1420058695 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee4c003dd0 0x7fee4c0041f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.993+0000 7fee63e00640 1 --2- 172.21.15.90:0/1420058695 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee4c0048c0 0x7fee4c004d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.993+0000 7fee63e00640 1 --2- 172.21.15.90:0/1420058695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee4c0099b0 0x7fee4c00bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.993+0000 7fee63e00640 1 -- 172.21.15.90:0/1420058695 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c00c450 con 0x7fee4c003dd0 2024-04-25T01:16:54.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.993+0000 7fee63e00640 1 -- 172.21.15.90:0/1420058695 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c00c600 con 0x7fee4c0099b0 2024-04-25T01:16:54.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.993+0000 7fee63e00640 1 -- 172.21.15.90:0/1420058695 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fee4c00c800 con 0x7fee4c0048c0 2024-04-25T01:16:54.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.994+0000 7fee69800640 1 --2- 172.21.15.90:0/1420058695 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee4c003dd0 0x7fee4c0041f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.994+0000 7fee6ac00640 1 --2- 172.21.15.90:0/1420058695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee4c0099b0 0x7fee4c00bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.996+0000 7f8a0da00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8a080d0650 0x7f8a0806d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.996+0000 7f8a0da00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a08140a70 con 0x7f8a0815b4f0 2024-04-25T01:16:54.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.996+0000 7f8a0da00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a0806e030 con 0x7f8a0815b060 2024-04-25T01:16:54.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.996+0000 7f8a0da00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f8a0815bd70 con 0x7f8a080d0650 2024-04-25T01:16:54.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.996+0000 7f8a06a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8a080d0650 0x7f8a0806d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.997+0000 7f4833000640 1 Processor -- start 2024-04-25T01:16:54.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.997+0000 7f8a06a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8a080d0650 0x7f8a0806d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56048/0 (socket says 172.21.15.90:56048) 2024-04-25T01:16:54.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.997+0000 7f8a06a00640 1 -- 172.21.15.90:0/4285696862 learned_addr learned my addr 172.21.15.90:0/4285696862 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:54.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.997+0000 7f4833000640 1 -- start start 2024-04-25T01:16:54.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.995+0000 7fee69800640 1 -- 172.21.15.90:0/1420058695 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee4c0048c0 msgr2=0x7fee4c004d40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:16:54.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.995+0000 7fee69800640 1 --2- 172.21.15.90:0/1420058695 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee4c0048c0 0x7fee4c004d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.995+0000 7fee69800640 1 -- 172.21.15.90:0/1420058695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee4c0099b0 msgr2=0x7fee4c00bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:54.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.995+0000 7fee69800640 1 --2- 172.21.15.90:0/1420058695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee4c0099b0 0x7fee4c00bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.995+0000 7fee69800640 1 -- 172.21.15.90:0/1420058695 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee4c003520 con 0x7fee4c003dd0 2024-04-25T01:16:54.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.996+0000 7fee6ac00640 1 -- 172.21.15.90:0/1420058695 reap_dead start 2024-04-25T01:16:54.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.997+0000 7f8a07400640 1 --1- 172.21.15.90:0/4285696862 >> v1:172.21.15.90:6789/0 conn(0x7f8a0815b4f0 0x7f8a08071c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:35548/0 (socket says 172.21.15.90:35548) 2024-04-25T01:16:54.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.997+0000 7f8a06a00640 1 -- 172.21.15.90:0/4285696862 >> v1:172.21.15.103:6789/0 conn(0x7f8a0815b060 legacy=0x7f8a08140360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:16:54.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.997+0000 7f8a06a00640 1 -- 172.21.15.90:0/4285696862 >> v1:172.21.15.90:6789/0 conn(0x7f8a0815b4f0 legacy=0x7f8a08071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:54.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.998+0000 7fee69800640 1 --2- 172.21.15.90:0/1420058695 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee4c003dd0 0x7fee4c0041f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fee500097d0 tx=0x7fee4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0ef96bcbf6812e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.998+0000 7fee63e00640 1 -- 172.21.15.90:0/1420058695 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee50037070 con 0x7fee4c003dd0 2024-04-25T01:16:54.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.999+0000 7fee63e00640 1 -- 172.21.15.90:0/1420058695 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee5002a9d0 con 0x7fee4c003dd0 2024-04-25T01:16:54.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.999+0000 7fee63e00640 1 -- 172.21.15.90:0/1420058695 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee500093d0 con 0x7fee4c003dd0 2024-04-25T01:16:54.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.999+0000 7f4833000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f482c154280 0x7f482c06ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.999+0000 7f4833000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f482c06cfc0 0x7f482c0b2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.999+0000 7f4833000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f482c154660 0x7f482c0b2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.999+0000 7f4833000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f482c071330 con 0x7f482c154280 2024-04-25T01:16:54.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.999+0000 7f4833000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f482c06d820 con 0x7f482c06cfc0 2024-04-25T01:16:54.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.998+0000 7f8a06a00640 1 -- 172.21.15.90:0/4285696862 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a08160ff0 con 0x7f8a080d0650 2024-04-25T01:16:54.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:54.999+0000 7f4833000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f482c06db00 con 0x7f482c154660 2024-04-25T01:16:54.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.000+0000 7f482be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f482c154280 0x7f482c06ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.000+0000 7f482b400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f482c06cfc0 0x7f482c0b2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.000+0000 7f482be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f482c154280 0x7f482c06ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56060/0 (socket says 172.21.15.90:56060) 2024-04-25T01:16:54.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.000+0000 7f8a06a00640 1 --2- 172.21.15.90:0/4285696862 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a080d0650 0x7f8a0806d670 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8a00009e50 tx=0x7f8a0002a810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bbd3b6eda1e5fc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.000+0000 7f482be00640 1 -- 172.21.15.90:0/1646457259 learned_addr learned my addr 172.21.15.90:0/1646457259 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:54.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.001+0000 7f4831000640 1 --2- 172.21.15.90:0/1646457259 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f482c154660 0x7f482c0b2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.001+0000 7fee6c200640 1 -- 172.21.15.90:0/1420058695 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee4c003dd0 msgr2=0x7fee4c0041f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.001+0000 7fee6c200640 1 --2- 172.21.15.90:0/1420058695 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee4c003dd0 0x7fee4c0041f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fee500097d0 tx=0x7fee4c005830 comp rx=0 tx=0).stop 2024-04-25T01:16:54.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.001+0000 7fb335c00640 1 Processor -- start 2024-04-25T01:16:54.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.001+0000 7f482b400640 1 -- 172.21.15.90:0/1646457259 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f482c154660 msgr2=0x7f482c0b2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:54.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.001+0000 7f482b400640 1 --2- 172.21.15.90:0/1646457259 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f482c154660 0x7f482c0b2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.001+0000 7f8a05600640 1 -- 172.21.15.90:0/4285696862 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a00033610 con 0x7f8a080d0650 2024-04-25T01:16:54.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.001+0000 7f8a05600640 1 -- 172.21.15.90:0/4285696862 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a0002f070 con 0x7f8a080d0650 2024-04-25T01:16:54.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.001+0000 7f8a05600640 1 -- 172.21.15.90:0/4285696862 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a00033a10 con 0x7f8a080d0650 2024-04-25T01:16:54.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.001+0000 7f482b400640 1 -- 172.21.15.90:0/1646457259 >> v2:172.21.15.90:3300/0 conn(0x7f482c154280 msgr2=0x7f482c06ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:54.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.001+0000 7f482b400640 1 --2- 172.21.15.90:0/1646457259 >> v2:172.21.15.90:3300/0 conn(0x7f482c154280 0x7f482c06ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.001+0000 7f482b400640 1 -- 172.21.15.90:0/1646457259 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f482c072a50 con 0x7f482c06cfc0 2024-04-25T01:16:54.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.002+0000 7f7f0be00640 1 Processor -- start 2024-04-25T01:16:54.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.002+0000 7f7f0be00640 1 -- start start 2024-04-25T01:16:54.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.001+0000 7fb335c00640 1 -- start start 2024-04-25T01:16:54.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.002+0000 7f8a0da00640 1 -- 172.21.15.90:0/4285696862 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a080d0650 msgr2=0x7f8a0806d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.002+0000 7fee6c200640 1 -- 172.21.15.90:0/1420058695 shutdown_connections 2024-04-25T01:16:54.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.002+0000 7fee6c200640 1 --2- 172.21.15.90:0/1420058695 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee4c003dd0 0x7fee4c0041f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.002+0000 7fee6c200640 1 -- 172.21.15.90:0/1420058695 >> 172.21.15.90:0/1420058695 conn(0x7fee5c054ca0 msgr2=0x7fee5c0550b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:54.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.002+0000 7f8a0da00640 1 --2- 172.21.15.90:0/4285696862 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a080d0650 0x7f8a0806d670 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8a00009e50 tx=0x7f8a0002a810 comp rx=0 tx=0).stop 2024-04-25T01:16:54.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.003+0000 7f8a0da00640 1 -- 172.21.15.90:0/4285696862 shutdown_connections 2024-04-25T01:16:54.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.003+0000 7f482b400640 1 --2- 172.21.15.90:0/1646457259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f482c06cfc0 0x7f482c0b2930 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f482c0cf7f0 tx=0x7f481802a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c6a42253691c525 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.003+0000 7f8a0da00640 1 --2- 172.21.15.90:0/4285696862 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a080d0650 0x7f8a0806d670 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.003+0000 7f8a0da00640 1 -- 172.21.15.90:0/4285696862 >> 172.21.15.90:0/4285696862 conn(0x7f8a080cfb30 msgr2=0x7f8a080cff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:54.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.003+0000 7f8a0da00640 1 -- 172.21.15.90:0/4285696862 shutdown_connections 2024-04-25T01:16:54.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.003+0000 7fb335c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb328005e90 0x7fb3280062d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.003+0000 7fb335c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb328006810 0x7fb328006ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.003+0000 7fb335c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb328053720 con 0x7fb3280ead00 2024-04-25T01:16:54.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.003+0000 7fb335c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb328054330 con 0x7fb328005e90 2024-04-25T01:16:54.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.003+0000 7fb335c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb328053a00 con 0x7fb328006810 2024-04-25T01:16:54.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.004+0000 7f7f0be00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7f040e3160 0x7f7f040e7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.004+0000 7f482aa00640 1 -- 172.21.15.90:0/1646457259 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4818037070 con 0x7f482c06cfc0 2024-04-25T01:16:54.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.004+0000 7f482aa00640 1 -- 172.21.15.90:0/1646457259 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4818032440 con 0x7f482c06cfc0 2024-04-25T01:16:54.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.004+0000 7f482aa00640 1 -- 172.21.15.90:0/1646457259 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4818032810 con 0x7f482c06cfc0 2024-04-25T01:16:54.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.004+0000 7fb335200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb328006810 0x7fb328006ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.004+0000 7fb32f400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb328005e90 0x7fb3280062d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.004+0000 7fb335200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb328006810 0x7fb328006ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.90:48086/0 (socket says 172.21.15.90:48086) 2024-04-25T01:16:54.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.004+0000 7fb335200640 1 -- 172.21.15.90:0/737663624 learned_addr learned my addr 172.21.15.90:0/737663624 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:54.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.004+0000 7f7f0be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f04053cb0 0x7f7f04054140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:54.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.004+0000 7f7f0be00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f04054680 con 0x7f7f0403c0a0 2024-04-25T01:16:54.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.004+0000 7f7f0be00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f7f04007b70 con 0x7f7f040e3160 2024-04-25T01:16:54.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.004+0000 7f7f0be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7f04058810 con 0x7f7f04053cb0 2024-04-25T01:16:54.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.005+0000 7f7f0aa00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7f040e3160 0x7f7f040e7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.005+0000 7fee6c200640 1 -- 172.21.15.90:0/1420058695 shutdown_connections 2024-04-25T01:16:54.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.005+0000 7f7f0aa00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7f040e3160 0x7f7f040e7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.90:39672/0 (socket says 172.21.15.90:39672) 2024-04-25T01:16:54.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.005+0000 7f7f0aa00640 1 -- 172.21.15.90:0/33608349 learned_addr learned my addr 172.21.15.90:0/33608349 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:54.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.003+0000 7f8a0da00640 1 -- 172.21.15.90:0/4285696862 wait complete. 2024-04-25T01:16:54.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.005+0000 7f7f0b400640 1 --2- 172.21.15.90:0/33608349 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f04053cb0 0x7f7f04054140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:54.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.005+0000 7f4833000640 1 -- 172.21.15.90:0/1646457259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f482c06cfc0 msgr2=0x7f482c0b2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.005+0000 7f4833000640 1 --2- 172.21.15.90:0/1646457259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f482c06cfc0 0x7f482c0b2930 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f482c0cf7f0 tx=0x7f481802a1a0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.005+0000 7fb32f400640 1 -- 172.21.15.90:0/737663624 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb328006810 msgr2=0x7fb328006ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:54.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.005+0000 7fb32f400640 1 --2- 172.21.15.90:0/737663624 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb328006810 0x7fb328006ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.005+0000 7fb32f400640 1 -- 172.21.15.90:0/737663624 >> v1:172.21.15.103:6789/0 conn(0x7fb3280ead00 legacy=0x7fb32804fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:54.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.005+0000 7fb32f400640 1 -- 172.21.15.90:0/737663624 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb328053ce0 con 0x7fb328005e90 2024-04-25T01:16:54.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.005+0000 7f4833000640 1 -- 172.21.15.90:0/1646457259 shutdown_connections 2024-04-25T01:16:54.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.005+0000 7f4833000640 1 --2- 172.21.15.90:0/1646457259 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f482c154660 0x7f482c0b2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.005+0000 7f4833000640 1 --2- 172.21.15.90:0/1646457259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f482c06cfc0 0x7f482c0b2930 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.005+0000 7f4833000640 1 --2- 172.21.15.90:0/1646457259 >> v2:172.21.15.90:3300/0 conn(0x7f482c154280 0x7f482c06ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.005+0000 7f4833000640 1 -- 172.21.15.90:0/1646457259 >> 172.21.15.90:0/1646457259 conn(0x7f482c0b1760 msgr2=0x7f482c0b1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:54.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.006+0000 7f4833000640 1 -- 172.21.15.90:0/1646457259 shutdown_connections 2024-04-25T01:16:54.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.006+0000 7f4833000640 1 -- 172.21.15.90:0/1646457259 wait complete. 2024-04-25T01:16:54.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.006+0000 7f7f0aa00640 1 -- 172.21.15.90:0/33608349 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f04053cb0 msgr2=0x7f7f04054140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:54.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.006+0000 7f7f0aa00640 1 --2- 172.21.15.90:0/33608349 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f04053cb0 0x7f7f04054140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:54.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.006+0000 7f7f0aa00640 1 -- 172.21.15.90:0/33608349 >> v1:172.21.15.103:6789/0 conn(0x7f7f0403c0a0 legacy=0x7f7f04058050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:54.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.006+0000 7f7f0aa00640 1 -- 172.21.15.90:0/33608349 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f04058af0 con 0x7f7f040e3160 2024-04-25T01:16:54.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.006+0000 7f8a0da00640 1 Processor -- start 2024-04-25T01:16:54.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.006+0000 7f8a0da00640 1 -- start start 2024-04-25T01:16:54.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.007+0000 7fb32f400640 1 --2- 172.21.15.90:0/737663624 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb328005e90 0x7fb3280062d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb31c0097a0 tx=0x7fb31c02a080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1048457fb074a7b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:54.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.005+0000 7fee6c200640 1 -- 172.21.15.90:0/1420058695 wait complete. 2024-04-25T01:16:54.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.007+0000 7fee6c200640 1 Processor -- start 2024-04-25T01:16:55.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.007+0000 7fb32ea00640 1 -- 172.21.15.90:0/737663624 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb31c037070 con 0x7fb328005e90 2024-04-25T01:16:55.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.007+0000 7fb32ea00640 1 -- 172.21.15.90:0/737663624 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb31c02ad90 con 0x7fb328005e90 2024-04-25T01:16:55.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.008+0000 7fb32ea00640 1 -- 172.21.15.90:0/737663624 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb31c032660 con 0x7fb328005e90 2024-04-25T01:16:55.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.008+0000 7f4833000640 1 Processor -- start 2024-04-25T01:16:55.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.008+0000 7f8a0da00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a080d0650 0x7f8a080cbd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.008+0000 7f8a0da00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a0815b060 0x7f8a080cca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.008+0000 7f8a0da00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a0815b4f0 0x7f8a081f7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.008+0000 7f4833000640 1 -- start start 2024-04-25T01:16:55.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.008+0000 7f7f0aa00640 1 --2- 172.21.15.90:0/33608349 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f040e3160 0x7f7f040e7780 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8009870 tx=0x7f7ef802a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3818234191057245 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:55.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.008+0000 7f7f09600640 1 -- 172.21.15.90:0/33608349 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef8037070 con 0x7f7f040e3160 2024-04-25T01:16:55.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.008+0000 7f7f09600640 1 -- 172.21.15.90:0/33608349 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ef8032440 con 0x7f7f040e3160 2024-04-25T01:16:55.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.008+0000 7f7f09600640 1 -- 172.21.15.90:0/33608349 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef8032810 con 0x7f7f040e3160 2024-04-25T01:16:55.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.009+0000 7fb335c00640 1 -- 172.21.15.90:0/737663624 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb328005e90 msgr2=0x7fb3280062d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.009+0000 7fb335c00640 1 --2- 172.21.15.90:0/737663624 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb328005e90 0x7fb3280062d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb31c0097a0 tx=0x7fb31c02a080 comp rx=0 tx=0).stop 2024-04-25T01:16:55.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.008+0000 7fee6c200640 1 -- start start 2024-04-25T01:16:55.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.009+0000 7f8a07400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a0815b4f0 0x7f8a081f7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.009+0000 7f8a07400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a0815b4f0 0x7f8a081f7660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56064/0 (socket says 172.21.15.90:56064) 2024-04-25T01:16:55.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.009+0000 7f8a06a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a080d0650 0x7f8a080cbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.009+0000 7f8a06a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a080d0650 0x7f8a080cbd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.90:48108/0 (socket says 172.21.15.90:48108) 2024-04-25T01:16:55.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.009+0000 7fb335c00640 1 -- 172.21.15.90:0/737663624 shutdown_connections 2024-04-25T01:16:55.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.009+0000 7fb335c00640 1 --2- 172.21.15.90:0/737663624 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb328006810 0x7fb328006ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.009+0000 7fb335c00640 1 --2- 172.21.15.90:0/737663624 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb328005e90 0x7fb3280062d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.009+0000 7fb335c00640 1 -- 172.21.15.90:0/737663624 >> 172.21.15.90:0/737663624 conn(0x7fb328009fb0 msgr2=0x7fb32800a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:55.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.009+0000 7f8a06000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a0815b060 0x7f8a080cca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.009+0000 7f8a06000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a0815b060 0x7f8a080cca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.90:39682/0 (socket says 172.21.15.90:39682) 2024-04-25T01:16:55.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.009+0000 7f8a07400640 1 -- 172.21.15.90:0/2570345249 learned_addr learned my addr 172.21.15.90:0/2570345249 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:55.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.009+0000 7f4833000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f482c06cfc0 0x7f482c0b0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.009+0000 7f4833000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f482c154280 0x7f482c149110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.009+0000 7f7f0be00640 1 -- 172.21.15.90:0/33608349 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f040e3160 msgr2=0x7f7f040e7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.009+0000 7f7f0be00640 1 --2- 172.21.15.90:0/33608349 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f040e3160 0x7f7f040e7780 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8009870 tx=0x7f7ef802a250 comp rx=0 tx=0).stop 2024-04-25T01:16:55.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.010+0000 7f4833000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f482c154660 0x7f482c149650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.010+0000 7f8a0da00640 1 -- 172.21.15.90:0/2570345249 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8a080724f0 con 0x7f8a0815b4f0 2024-04-25T01:16:55.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.010+0000 7f8a0da00640 1 -- 172.21.15.90:0/2570345249 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8a08072370 con 0x7f8a0815b060 2024-04-25T01:16:55.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.010+0000 7f8a0da00640 1 -- 172.21.15.90:0/2570345249 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8a0815d210 con 0x7f8a080d0650 2024-04-25T01:16:55.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.010+0000 7f4833000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f482c0aec90 con 0x7f482c154280 2024-04-25T01:16:55.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.010+0000 7f4833000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f482c0aeb10 con 0x7f482c154660 2024-04-25T01:16:55.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.010+0000 7f4833000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f482c0aee10 con 0x7f482c06cfc0 2024-04-25T01:16:55.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.010+0000 7f7f0be00640 1 -- 172.21.15.90:0/33608349 shutdown_connections 2024-04-25T01:16:55.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.010+0000 7f7f0be00640 1 --2- 172.21.15.90:0/33608349 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f04053cb0 0x7f7f04054140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.010+0000 7f7f0be00640 1 --2- 172.21.15.90:0/33608349 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f040e3160 0x7f7f040e7780 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.010+0000 7f7f0be00640 1 -- 172.21.15.90:0/33608349 >> 172.21.15.90:0/33608349 conn(0x7f7f04058ee0 msgr2=0x7f7f040592c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:55.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.010+0000 7fb335c00640 1 -- 172.21.15.90:0/737663624 shutdown_connections 2024-04-25T01:16:55.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.010+0000 7f7f0be00640 1 -- 172.21.15.90:0/33608349 shutdown_connections 2024-04-25T01:16:55.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.010+0000 7f482b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f482c154280 0x7f482c149110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.010+0000 7f482be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f482c06cfc0 0x7f482c0b0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.010+0000 7f482be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f482c06cfc0 0x7f482c0b0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.90:48110/0 (socket says 172.21.15.90:48110) 2024-04-25T01:16:55.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.010+0000 7f482be00640 1 -- 172.21.15.90:0/286505113 learned_addr learned my addr 172.21.15.90:0/286505113 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:55.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.010+0000 7f4831000640 1 --2- 172.21.15.90:0/286505113 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f482c154660 0x7f482c149650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.011+0000 7fee6c200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee4c003dd0 0x7fee5c0dab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.011+0000 7f0d2d400640 1 Processor -- start 2024-04-25T01:16:55.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.011+0000 7fee6c200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee5c17de20 0x7fee5c04bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.011+0000 7f0d2d400640 1 -- start start 2024-04-25T01:16:55.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.011+0000 7fee6c200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee5c04c2b0 0x7fee5c039350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.011+0000 7fee6c200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fee5c0381b0 con 0x7fee5c17de20 2024-04-25T01:16:55.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.010+0000 7f7f0be00640 1 -- 172.21.15.90:0/33608349 wait complete. 2024-04-25T01:16:55.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.012+0000 7fee69800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee4c003dd0 0x7fee5c0dab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.012+0000 7fee69800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee4c003dd0 0x7fee5c0dab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.90:48114/0 (socket says 172.21.15.90:48114) 2024-04-25T01:16:55.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.012+0000 7fee69800640 1 -- 172.21.15.90:0/2228745707 learned_addr learned my addr 172.21.15.90:0/2228745707 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:55.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.012+0000 7fee6c200640 1 -- 172.21.15.90:0/2228745707 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fee5c038520 con 0x7fee5c04c2b0 2024-04-25T01:16:55.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.012+0000 7fee6c200640 1 -- 172.21.15.90:0/2228745707 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fee5c04f500 con 0x7fee4c003dd0 2024-04-25T01:16:55.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.012+0000 7fee6ac00640 1 --2- 172.21.15.90:0/2228745707 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee5c04c2b0 0x7fee5c039350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.012+0000 7fee68e00640 1 --2- 172.21.15.90:0/2228745707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee5c17de20 0x7fee5c04bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.013+0000 7fb335c00640 1 -- 172.21.15.90:0/737663624 wait complete. 2024-04-25T01:16:55.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.013+0000 7f7f0be00640 1 Processor -- start 2024-04-25T01:16:55.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.013+0000 7f0d2d400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0d280d2880 0x7f0d280b0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.013+0000 7f0d2d400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0d280b2da0 0x7f0d28161390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.013+0000 7f0d2d400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d280d1120 con 0x7f0d280d24a0 2024-04-25T01:16:55.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.013+0000 7fee69800640 1 -- 172.21.15.90:0/2228745707 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee5c04c2b0 msgr2=0x7fee5c039350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.013+0000 7fee69800640 1 --2- 172.21.15.90:0/2228745707 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee5c04c2b0 0x7fee5c039350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.013+0000 7fee69800640 1 -- 172.21.15.90:0/2228745707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee5c17de20 msgr2=0x7fee5c04bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.013+0000 7fee69800640 1 --2- 172.21.15.90:0/2228745707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee5c17de20 0x7fee5c04bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.013+0000 7f0d2d400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f0d280d1400 con 0x7f0d280d2880 2024-04-25T01:16:55.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.013+0000 7f0d2d400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f0d280d16e0 con 0x7f0d280b2da0 2024-04-25T01:16:55.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.013+0000 7fee69800640 1 -- 172.21.15.90:0/2228745707 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee5c180790 con 0x7fee4c003dd0 2024-04-25T01:16:55.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.012+0000 7f8a06000640 1 -- 172.21.15.90:0/2570345249 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a080d0650 msgr2=0x7f8a080cbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.013+0000 7f8a06000640 1 --2- 172.21.15.90:0/2570345249 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a080d0650 0x7f8a080cbd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.013+0000 7f8a06000640 1 -- 172.21.15.90:0/2570345249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a0815b4f0 msgr2=0x7f8a081f7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.013+0000 7f8a06000640 1 --2- 172.21.15.90:0/2570345249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a0815b4f0 0x7f8a081f7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.013+0000 7f8a06000640 1 -- 172.21.15.90:0/2570345249 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a0815ed80 con 0x7f8a0815b060 2024-04-25T01:16:55.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.013+0000 7f8a06a00640 1 --2- 172.21.15.90:0/2570345249 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a080d0650 0x7f8a080cbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:16:55.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.013+0000 7f482be00640 1 -- 172.21.15.90:0/286505113 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f482c154660 msgr2=0x7f482c149650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.013+0000 7f482be00640 1 --2- 172.21.15.90:0/286505113 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f482c154660 0x7f482c149650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.013+0000 7f482be00640 1 -- 172.21.15.90:0/286505113 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f482c154280 msgr2=0x7f482c149110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.013+0000 7f482be00640 1 --2- 172.21.15.90:0/286505113 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f482c154280 0x7f482c149110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.013+0000 7f482be00640 1 -- 172.21.15.90:0/286505113 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f482c0adc40 con 0x7f482c06cfc0 2024-04-25T01:16:55.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.013+0000 7f7f0be00640 1 -- start start 2024-04-25T01:16:55.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.014+0000 7f6338e00640 1 Processor -- start 2024-04-25T01:16:55.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.014+0000 7fee69800640 1 --2- 172.21.15.90:0/2228745707 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee4c003dd0 0x7fee5c0dab20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fee500098f0 tx=0x7fee50004700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:55.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.014+0000 7f0d27200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0d280b2da0 0x7f0d28161390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.014+0000 7f482b400640 1 --2- 172.21.15.90:0/286505113 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f482c154280 0x7f482c149110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:16:55.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.014+0000 7f482be00640 1 --2- 172.21.15.90:0/286505113 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f482c06cfc0 0x7f482c0b0910 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f481c00e810 tx=0x7f481c00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:55.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.014+0000 7f0d27200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0d280b2da0 0x7f0d28161390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.90:39712/0 (socket says 172.21.15.90:39712) 2024-04-25T01:16:55.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.014+0000 7f8a06000640 1 --2- 172.21.15.90:0/2570345249 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a0815b060 0x7f8a080cca60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f89f800b410 tx=0x7f89f800b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:55.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.014+0000 7f0d27200640 1 -- 172.21.15.90:0/989952772 learned_addr learned my addr 172.21.15.90:0/989952772 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:55.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.014+0000 7f0d25e00640 1 --2- 172.21.15.90:0/989952772 >> v2:172.21.15.90:3300/0 conn(0x7f0d280d2880 0x7f0d280b0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.014+0000 7f6338e00640 1 -- start start 2024-04-25T01:16:55.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.014+0000 7f89fea00640 1 -- 172.21.15.90:0/2570345249 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f8015020 con 0x7f8a0815b060 2024-04-25T01:16:55.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.014+0000 7fee61600640 1 -- 172.21.15.90:0/2228745707 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee50037050 con 0x7fee4c003dd0 2024-04-25T01:16:55.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.015+0000 7f4823e00640 1 -- 172.21.15.90:0/286505113 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481c017070 con 0x7f482c06cfc0 2024-04-25T01:16:55.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.015+0000 7f4823e00640 1 -- 172.21.15.90:0/286505113 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f481c00c8d0 con 0x7f482c06cfc0 2024-04-25T01:16:55.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.015+0000 7f4823e00640 1 -- 172.21.15.90:0/286505113 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481c00f040 con 0x7f482c06cfc0 2024-04-25T01:16:55.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.015+0000 7f0d25400640 1 -- 172.21.15.90:0/989952772 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 660697339 0 0) 0x7f0d280d1120 con 0x7f0d280d24a0 2024-04-25T01:16:55.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.015+0000 7f4833000640 1 -- 172.21.15.90:0/286505113 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f482c0adf20 con 0x7f482c06cfc0 2024-04-25T01:16:55.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.015+0000 7f4833000640 1 -- 172.21.15.90:0/286505113 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f482c0aa770 con 0x7f482c06cfc0 2024-04-25T01:16:55.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.015+0000 7f0d25400640 1 -- 172.21.15.90:0/989952772 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d0c003520 con 0x7f0d280d24a0 2024-04-25T01:16:55.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.015+0000 7fee6c200640 1 -- 172.21.15.90:0/2228745707 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee5c180a70 con 0x7fee4c003dd0 2024-04-25T01:16:55.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.015+0000 7fee6c200640 1 -- 172.21.15.90:0/2228745707 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fee5c181030 con 0x7fee4c003dd0 2024-04-25T01:16:55.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.015+0000 7fee61600640 1 -- 172.21.15.90:0/2228745707 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee50043070 con 0x7fee4c003dd0 2024-04-25T01:16:55.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.015+0000 7fee61600640 1 -- 172.21.15.90:0/2228745707 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee50037050 con 0x7fee4c003dd0 2024-04-25T01:16:55.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.015+0000 7fb335c00640 1 Processor -- start 2024-04-25T01:16:55.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.015+0000 7f0d25e00640 1 -- 172.21.15.90:0/989952772 >> v2:172.21.15.103:3300/0 conn(0x7f0d280b2da0 msgr2=0x7f0d28161390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:55.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.015+0000 7f7f0be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f0403c0a0 0x7f7f040de1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.015+0000 7f7f0be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f04053cb0 0x7f7f0417d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.016+0000 7f7f0be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f040e3160 0x7f7f0403d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.015+0000 7f8a0da00640 1 -- 172.21.15.90:0/2570345249 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a0813ef70 con 0x7f8a0815b060 2024-04-25T01:16:55.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.016+0000 7f89fea00640 1 -- 172.21.15.90:0/2570345249 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89f8011850 con 0x7f8a0815b060 2024-04-25T01:16:55.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.016+0000 7f8a0da00640 1 -- 172.21.15.90:0/2570345249 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a0813f530 con 0x7f8a0815b060 2024-04-25T01:16:55.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.015+0000 7f0d25e00640 1 --2- 172.21.15.90:0/989952772 >> v2:172.21.15.103:3300/0 conn(0x7f0d280b2da0 0x7f0d28161390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.015+0000 7f0d25e00640 1 -- 172.21.15.90:0/989952772 >> v1:172.21.15.90:6789/0 conn(0x7f0d280d24a0 legacy=0x7f0d280d08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.015+0000 7f0d25e00640 1 -- 172.21.15.90:0/989952772 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d280cee60 con 0x7f0d280d2880 2024-04-25T01:16:55.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.016+0000 7f0d27200640 1 --2- 172.21.15.90:0/989952772 >> v2:172.21.15.103:3300/0 conn(0x7f0d280b2da0 0x7f0d28161390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:16:55.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.016+0000 7f7f0be00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7f040e3540 con 0x7f7f04053cb0 2024-04-25T01:16:55.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.016+0000 7f7f0be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7f04006640 con 0x7f7f040e3160 2024-04-25T01:16:55.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.016+0000 7f7f0aa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f0403c0a0 0x7f7f040de1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.017+0000 7f7f0a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f04053cb0 0x7f7f0417d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.017+0000 7f7f0be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7f04006b60 con 0x7f7f0403c0a0 2024-04-25T01:16:55.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.017+0000 7f7f0aa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f0403c0a0 0x7f7f040de1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.90:48120/0 (socket says 172.21.15.90:48120) 2024-04-25T01:16:55.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.017+0000 7f7f0aa00640 1 -- 172.21.15.90:0/1688390906 learned_addr learned my addr 172.21.15.90:0/1688390906 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:55.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.016+0000 7f89fea00640 1 -- 172.21.15.90:0/2570345249 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f8011c40 con 0x7f8a0815b060 2024-04-25T01:16:55.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.017+0000 7f0d25e00640 1 --2- 172.21.15.90:0/989952772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d280d2880 0x7f0d280b0bd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0d280b9f80 tx=0x7f0d1002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85756b3d29d38b12 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:55.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.017+0000 7f7f0b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f040e3160 0x7f7f0403d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.016+0000 7f6338e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f63340b2b20 0x7f63340b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.016+0000 7f6338e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f633406cf60 con 0x7f63340cefe0 2024-04-25T01:16:55.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.016+0000 7f6338e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f633406d240 con 0x7f63340b4af0 2024-04-25T01:16:55.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.016+0000 7f6338e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f633406d520 con 0x7f63340b2b20 2024-04-25T01:16:55.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.017+0000 7f6329400640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f63340b4af0 0x7f6334163440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.90:55266/0 (socket says 172.21.15.90:55266) 2024-04-25T01:16:55.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.017+0000 7f6329400640 1 -- 172.21.15.90:0/1154065997 learned_addr learned my addr 172.21.15.90:0/1154065997 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:55.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.018+0000 7f0d25400640 1 -- 172.21.15.90:0/989952772 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d10037070 con 0x7f0d280d2880 2024-04-25T01:16:55.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.018+0000 7f4821600640 1 -- 172.21.15.90:0/286505113 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4800005290 con 0x7f482c06cfc0 2024-04-25T01:16:55.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.018+0000 7f0d25400640 1 -- 172.21.15.90:0/989952772 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d10032440 con 0x7f0d280d2880 2024-04-25T01:16:55.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.018+0000 7f0d25400640 1 -- 172.21.15.90:0/989952772 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d10032810 con 0x7f0d280d2880 2024-04-25T01:16:55.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.018+0000 7f6332800640 1 --2- 172.21.15.90:0/1154065997 >> v2:172.21.15.90:3300/0 conn(0x7f63340b2b20 0x7f63340b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.018+0000 7fee56a00640 1 -- 172.21.15.90:0/2228745707 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee38005290 con 0x7fee4c003dd0 2024-04-25T01:16:55.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.018+0000 7f7f0aa00640 1 -- 172.21.15.90:0/1688390906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f040e3160 msgr2=0x7f7f0403d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.018+0000 7f7f0aa00640 1 --2- 172.21.15.90:0/1688390906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f040e3160 0x7f7f0403d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.018+0000 7f7f0aa00640 1 -- 172.21.15.90:0/1688390906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f04053cb0 msgr2=0x7f7f0417d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.018+0000 7f8a0da00640 1 -- 172.21.15.90:0/2570345249 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a08068fb0 con 0x7f8a0815b060 2024-04-25T01:16:55.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.019+0000 7f7f0aa00640 1 --2- 172.21.15.90:0/1688390906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f04053cb0 0x7f7f0417d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.019+0000 7f7f0aa00640 1 -- 172.21.15.90:0/1688390906 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f04181dd0 con 0x7f7f0403c0a0 2024-04-25T01:16:55.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.019+0000 7f7f0a000640 1 --2- 172.21.15.90:0/1688390906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f04053cb0 0x7f7f0417d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:16:55.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.019+0000 7f6332800640 1 -- 172.21.15.90:0/1154065997 >> v1:172.21.15.103:6789/0 conn(0x7f63340b4af0 legacy=0x7f6334163440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.019+0000 7f6332800640 1 -- 172.21.15.90:0/1154065997 >> v1:172.21.15.90:6789/0 conn(0x7f63340cefe0 legacy=0x7f633406e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.019+0000 7f6332800640 1 -- 172.21.15.90:0/1154065997 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f633406d800 con 0x7f63340b2b20 2024-04-25T01:16:55.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.019+0000 7f0d2d400640 1 -- 172.21.15.90:0/989952772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d280d2880 msgr2=0x7f0d280b0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.019+0000 7f0d2d400640 1 --2- 172.21.15.90:0/989952772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d280d2880 0x7f0d280b0bd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0d280b9f80 tx=0x7f0d1002a1a0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.017+0000 7fb335c00640 1 -- start start 2024-04-25T01:16:55.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.019+0000 7fb335c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb328005e90 0x7fb328180b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.019+0000 7fb335c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb328006810 0x7fb32817de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.019+0000 7fb335c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3280ead00 0x7fb32804de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.019+0000 7fb335c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb3280547c0 con 0x7fb328005e90 2024-04-25T01:16:55.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.019+0000 7fb335c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb3280501a0 con 0x7fb328006810 2024-04-25T01:16:55.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.019+0000 7fb335c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb328054940 con 0x7fb3280ead00 2024-04-25T01:16:55.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.019+0000 7f0d2d400640 1 -- 172.21.15.90:0/989952772 shutdown_connections 2024-04-25T01:16:55.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.019+0000 7f0d2d400640 1 --2- 172.21.15.90:0/989952772 >> v2:172.21.15.103:3300/0 conn(0x7f0d280b2da0 0x7f0d28161390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.019+0000 7f0d2d400640 1 --2- 172.21.15.90:0/989952772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d280d2880 0x7f0d280b0bd0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.020+0000 7f0d2d400640 1 -- 172.21.15.90:0/989952772 >> 172.21.15.90:0/989952772 conn(0x7f0d2809d200 msgr2=0x7f0d280d3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:55.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.020+0000 7f7f0aa00640 1 --2- 172.21.15.90:0/1688390906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f0403c0a0 0x7f7f040de1d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8000c00 tx=0x7f7ef80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:55.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.020+0000 7f0d2d400640 1 -- 172.21.15.90:0/989952772 shutdown_connections 2024-04-25T01:16:55.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.020+0000 7fb335200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3280ead00 0x7fb32804de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.020+0000 7f89fea00640 1 -- 172.21.15.90:0/2570345249 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f89f8018020 con 0x7f8a0815b060 2024-04-25T01:16:55.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.020+0000 7f0d2d400640 1 -- 172.21.15.90:0/989952772 wait complete. 2024-04-25T01:16:55.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.020+0000 7fb32fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb328005e90 0x7fb328180b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.020+0000 7fb32fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb328005e90 0x7fb328180b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56136/0 (socket says 172.21.15.90:56136) 2024-04-25T01:16:55.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.020+0000 7fb32fe00640 1 -- 172.21.15.90:0/3710900340 learned_addr learned my addr 172.21.15.90:0/3710900340 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:55.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.021+0000 7fb32f400640 1 --2- 172.21.15.90:0/3710900340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb328006810 0x7fb32817de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.021+0000 7fb32fe00640 1 -- 172.21.15.90:0/3710900340 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3280ead00 msgr2=0x7fb32804de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.021+0000 7fb32fe00640 1 --2- 172.21.15.90:0/3710900340 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3280ead00 0x7fb32804de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.021+0000 7fb32fe00640 1 -- 172.21.15.90:0/3710900340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb328006810 msgr2=0x7fb32817de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.021+0000 7fb32fe00640 1 --2- 172.21.15.90:0/3710900340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb328006810 0x7fb32817de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.021+0000 7fb32fe00640 1 -- 172.21.15.90:0/3710900340 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3281813b0 con 0x7fb328005e90 2024-04-25T01:16:55.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.021+0000 7fdf3f800640 1 Processor -- start 2024-04-25T01:16:55.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.020+0000 7fee61600640 1 -- 172.21.15.90:0/2228745707 <== mon.2 v2:172.21.15.103:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fee50032c30 con 0x7fee4c003dd0 2024-04-25T01:16:55.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.022+0000 7fb32fe00640 1 --2- 172.21.15.90:0/3710900340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb328005e90 0x7fb328180b60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb32400b3e0 tx=0x7fb32400b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:55.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.022+0000 7f89fea00640 1 --2- 172.21.15.90:0/2570345249 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f89e806be30 0x7f89e806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.022+0000 7fb323e00640 1 -- 172.21.15.90:0/3710900340 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb324015020 con 0x7fb328005e90 2024-04-25T01:16:55.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.022+0000 7fb323e00640 1 -- 172.21.15.90:0/3710900340 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb324011850 con 0x7fb328005e90 2024-04-25T01:16:55.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.022+0000 7fb323e00640 1 -- 172.21.15.90:0/3710900340 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb324011c40 con 0x7fb328005e90 2024-04-25T01:16:55.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.022+0000 7fb335c00640 1 -- 172.21.15.90:0/3710900340 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb328181690 con 0x7fb328005e90 2024-04-25T01:16:55.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.022+0000 7f4823e00640 1 -- 172.21.15.90:0/286505113 <== mon.2 v2:172.21.15.103:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f481c01a070 con 0x7f482c06cfc0 2024-04-25T01:16:55.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.022+0000 7fee61600640 1 --2- 172.21.15.90:0/2228745707 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fee4406c060 0x7fee4406e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.022+0000 7f0d2d400640 1 Processor -- start 2024-04-25T01:16:55.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.023+0000 7f89fea00640 1 -- 172.21.15.90:0/2570345249 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f89f808b020 con 0x7f8a0815b060 2024-04-25T01:16:55.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.023+0000 7f8a06a00640 1 --2- 172.21.15.90:0/2570345249 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f89e806be30 0x7f89e806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.022+0000 7fdf3f800640 1 -- start start 2024-04-25T01:16:55.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.022+0000 7fb335c00640 1 -- 172.21.15.90:0/3710900340 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb32817d680 con 0x7fb328005e90 2024-04-25T01:16:55.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.023+0000 7fee61600640 1 -- 172.21.15.90:0/2228745707 <== mon.2 v2:172.21.15.103:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fee500ab260 con 0x7fee4c003dd0 2024-04-25T01:16:55.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.024+0000 7fdf3f800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdf380cefe0 0x7fdf380b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.024+0000 7f4823e00640 1 --2- 172.21.15.90:0/286505113 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f480c06be30 0x7f480c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.024+0000 7f4823e00640 1 -- 172.21.15.90:0/286505113 <== mon.2 v2:172.21.15.103:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f481c08b0c0 con 0x7f482c06cfc0 2024-04-25T01:16:55.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.024+0000 7fb335c00640 1 -- 172.21.15.90:0/3710900340 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb328005390 con 0x7fb328005e90 2024-04-25T01:16:55.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.024+0000 7f6332800640 1 --2- 172.21.15.90:0/1154065997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63340b2b20 0x7f63340b2fd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f632c0097a0 tx=0x7f632c02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4ad820f63b90209 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:55.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.024+0000 7fdf3f800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf3806cf60 con 0x7fdf3806e000 2024-04-25T01:16:55.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.024+0000 7fdf3f800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf3806d240 con 0x7fdf380b2b20 2024-04-25T01:16:55.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.023+0000 7f0d2d400640 1 -- start start 2024-04-25T01:16:55.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.024+0000 7f0d2d400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d280b2da0 0x7f0d280b8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.024+0000 7f0d2d400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d280d24a0 0x7f0d28150160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.024+0000 7f0d2d400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d280d2880 0x7f0d281506a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.024+0000 7fdf3f800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fdf3806d520 con 0x7fdf380cefe0 2024-04-25T01:16:55.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.025+0000 7fdf3ce00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdf380cefe0 0x7fdf380b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.025+0000 7fdf2fe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fdf3806e000 0x7fdf38163440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:35584/0 (socket says 172.21.15.90:35584) 2024-04-25T01:16:55.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.025+0000 7fee68e00640 1 --2- 172.21.15.90:0/2228745707 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fee4406c060 0x7fee4406e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.025+0000 7fdf2fe00640 1 -- 172.21.15.90:0/2856357372 learned_addr learned my addr 172.21.15.90:0/2856357372 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:55.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.025+0000 7f0d2d400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0d28161a50 con 0x7f0d280d24a0 2024-04-25T01:16:55.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.025+0000 7f8a06a00640 1 --2- 172.21.15.90:0/2570345249 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f89e806be30 0x7f89e806e2f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8a00034000 tx=0x7f8a00009000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:55.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.025+0000 7f482b400640 1 --2- 172.21.15.90:0/286505113 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f480c06be30 0x7f480c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.025+0000 7f6331400640 1 -- 172.21.15.90:0/1154065997 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632c037070 con 0x7f63340b2b20 2024-04-25T01:16:55.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.025+0000 7f6331400640 1 -- 172.21.15.90:0/1154065997 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f632c032440 con 0x7f63340b2b20 2024-04-25T01:16:55.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.025+0000 7f6331400640 1 -- 172.21.15.90:0/1154065997 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632c032850 con 0x7f63340b2b20 2024-04-25T01:16:55.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.025+0000 7f0d2d400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0d281618d0 con 0x7f0d280b2da0 2024-04-25T01:16:55.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.025+0000 7f0d2d400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0d28161bd0 con 0x7f0d280d2880 2024-04-25T01:16:55.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.026+0000 7fee68e00640 1 --2- 172.21.15.90:0/2228745707 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fee4406c060 0x7fee4406e520 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fee5c007240 tx=0x7fee58009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:55.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.026+0000 7f0d26800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d280b2da0 0x7f0d280b8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.026+0000 7f0d25e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d280d24a0 0x7f0d28150160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.026+0000 7f0d25e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d280d24a0 0x7f0d28150160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56150/0 (socket says 172.21.15.90:56150) 2024-04-25T01:16:55.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.026+0000 7f0d25e00640 1 -- 172.21.15.90:0/3723125966 learned_addr learned my addr 172.21.15.90:0/3723125966 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:55.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.026+0000 7fdf37e00640 1 -- 172.21.15.90:0/2856357372 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2167754185 0 0) 0x7fdf3806cf60 con 0x7fdf3806e000 2024-04-25T01:16:55.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.026+0000 7fdf37e00640 1 -- 172.21.15.90:0/2856357372 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf1c003520 con 0x7fdf3806e000 2024-04-25T01:16:55.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.026+0000 7fdf3ce00640 1 -- 172.21.15.90:0/2856357372 >> v1:172.21.15.103:6789/0 conn(0x7fdf380b2b20 legacy=0x7fdf38070c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.026+0000 7fdf3ce00640 1 -- 172.21.15.90:0/2856357372 >> v1:172.21.15.90:6789/0 conn(0x7fdf3806e000 legacy=0x7fdf38163440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.026+0000 7fdf3ce00640 1 -- 172.21.15.90:0/2856357372 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf3806d800 con 0x7fdf380cefe0 2024-04-25T01:16:55.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.026+0000 7f482b400640 1 --2- 172.21.15.90:0/286505113 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f480c06be30 0x7f480c06e2f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4818034000 tx=0x7f481802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:55.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.026+0000 7f6338e00640 1 -- 172.21.15.90:0/1154065997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63340b2b20 msgr2=0x7f63340b2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.026+0000 7f6338e00640 1 --2- 172.21.15.90:0/1154065997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63340b2b20 0x7f63340b2fd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f632c0097a0 tx=0x7f632c02a1c0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.026+0000 7f0d27200640 1 --2- 172.21.15.90:0/3723125966 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d280d2880 0x7f0d281506a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.026+0000 7f0d25e00640 1 -- 172.21.15.90:0/3723125966 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d280d2880 msgr2=0x7f0d281506a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.026+0000 7f0d25e00640 1 --2- 172.21.15.90:0/3723125966 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d280d2880 0x7f0d281506a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.026+0000 7f0d25e00640 1 -- 172.21.15.90:0/3723125966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d280b2da0 msgr2=0x7f0d280b8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.026+0000 7f0d25e00640 1 --2- 172.21.15.90:0/3723125966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d280b2da0 0x7f0d280b8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.026+0000 7f0d25e00640 1 -- 172.21.15.90:0/3723125966 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d280b8ac0 con 0x7f0d280d24a0 2024-04-25T01:16:55.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.027+0000 7fb323e00640 1 -- 172.21.15.90:0/3710900340 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb324018020 con 0x7fb328005e90 2024-04-25T01:16:55.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.026+0000 7f6338e00640 1 -- 172.21.15.90:0/1154065997 shutdown_connections 2024-04-25T01:16:55.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.026+0000 7f6338e00640 1 --2- 172.21.15.90:0/1154065997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63340b2b20 0x7f63340b2fd0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.027+0000 7f6338e00640 1 -- 172.21.15.90:0/1154065997 >> 172.21.15.90:0/1154065997 conn(0x7f63340b1820 msgr2=0x7f63340b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:55.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.027+0000 7f6338e00640 1 -- 172.21.15.90:0/1154065997 shutdown_connections 2024-04-25T01:16:55.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.027+0000 7f0d25e00640 1 --2- 172.21.15.90:0/3723125966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d280d24a0 0x7f0d28150160 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0d1002a790 tx=0x7f0d100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:55.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.027+0000 7fb323e00640 1 --2- 172.21.15.90:0/3710900340 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb31006be80 0x7fb31006e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.028+0000 7f6338e00640 1 -- 172.21.15.90:0/1154065997 wait complete. 2024-04-25T01:16:55.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.028+0000 7fdf3ce00640 1 --2- 172.21.15.90:0/2856357372 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf380cefe0 0x7fdf380b4af0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdf28009d90 tx=0x7fdf2802a610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cded6137ff68d76 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:55.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.028+0000 7fb323e00640 1 -- 172.21.15.90:0/3710900340 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fb32408afe0 con 0x7fb328005e90 2024-04-25T01:16:55.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.028+0000 7fb32f400640 1 --2- 172.21.15.90:0/3710900340 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb31006be80 0x7fb31006e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.028+0000 7fb32f400640 1 --2- 172.21.15.90:0/3710900340 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb31006be80 0x7fb31006e340 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb31c034000 tx=0x7fb31c02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:55.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.029+0000 7f6338e00640 1 Processor -- start 2024-04-25T01:16:55.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.029+0000 7f6338e00640 1 -- start start 2024-04-25T01:16:55.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.030+0000 7f6338e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63340b2b20 0x7f6334157380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.030+0000 7f6338e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63340b4af0 0x7f6334154790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.030+0000 7f6338e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63340cefe0 0x7f6334164670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.030+0000 7f6338e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6334070dd0 con 0x7f63340b4af0 2024-04-25T01:16:55.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.030+0000 7f6338e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6334070c50 con 0x7f63340cefe0 2024-04-25T01:16:55.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.030+0000 7f6338e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6334070f50 con 0x7f63340b2b20 2024-04-25T01:16:55.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.031+0000 7f6329400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63340b4af0 0x7f6334154790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.031+0000 7f6329400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63340b4af0 0x7f6334154790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56154/0 (socket says 172.21.15.90:56154) 2024-04-25T01:16:55.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.031+0000 7f6329400640 1 -- 172.21.15.90:0/1973629256 learned_addr learned my addr 172.21.15.90:0/1973629256 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:55.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.031+0000 7f6331e00640 1 --2- 172.21.15.90:0/1973629256 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63340b2b20 0x7f6334157380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.031+0000 7f6332800640 1 --2- 172.21.15.90:0/1973629256 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63340cefe0 0x7f6334164670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.031+0000 7f6329400640 1 -- 172.21.15.90:0/1973629256 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63340b2b20 msgr2=0x7f6334157380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.031+0000 7f6329400640 1 --2- 172.21.15.90:0/1973629256 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63340b2b20 0x7f6334157380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.031+0000 7f6329400640 1 -- 172.21.15.90:0/1973629256 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63340cefe0 msgr2=0x7f6334164670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.031+0000 7f6329400640 1 --2- 172.21.15.90:0/1973629256 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63340cefe0 0x7f6334164670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.031+0000 7f6329400640 1 -- 172.21.15.90:0/1973629256 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63341ebdd0 con 0x7f63340b4af0 2024-04-25T01:16:55.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.031+0000 7f6332800640 1 --2- 172.21.15.90:0/1973629256 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63340cefe0 0x7f6334164670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:55.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.031+0000 7f6329400640 1 --2- 172.21.15.90:0/1973629256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63340b4af0 0x7f6334154790 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f631c00c530 tx=0x7f631c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:55.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.032+0000 7f632aa00640 1 -- 172.21.15.90:0/1973629256 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f631c017070 con 0x7f63340b4af0 2024-04-25T01:16:55.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.032+0000 7f632aa00640 1 -- 172.21.15.90:0/1973629256 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f631c005ce0 con 0x7f63340b4af0 2024-04-25T01:16:55.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.032+0000 7f632aa00640 1 -- 172.21.15.90:0/1973629256 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f631c007270 con 0x7f63340b4af0 2024-04-25T01:16:55.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.032+0000 7f6338e00640 1 -- 172.21.15.90:0/1973629256 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63341ec070 con 0x7f63340b4af0 2024-04-25T01:16:55.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.032+0000 7f0d1ea00640 1 -- 172.21.15.90:0/3723125966 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d10037050 con 0x7f0d280d24a0 2024-04-25T01:16:55.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.032+0000 7f6338e00640 1 -- 172.21.15.90:0/1973629256 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63341ec630 con 0x7f63340b4af0 2024-04-25T01:16:55.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.032+0000 7f0d1ea00640 1 -- 172.21.15.90:0/3723125966 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d10043070 con 0x7f0d280d24a0 2024-04-25T01:16:55.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.032+0000 7f0d1ea00640 1 -- 172.21.15.90:0/3723125966 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d1003ca50 con 0x7f0d280d24a0 2024-04-25T01:16:55.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.033+0000 7f0d2d400640 1 -- 172.21.15.90:0/3723125966 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d280b8d40 con 0x7f0d280d24a0 2024-04-25T01:16:55.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.032+0000 7fb323e00640 1 -- 172.21.15.90:0/3710900340 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb32401f020 con 0x7fb328005e90 2024-04-25T01:16:55.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.033+0000 7f0d2d400640 1 -- 172.21.15.90:0/3723125966 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d280b9300 con 0x7f0d280d24a0 2024-04-25T01:16:55.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.034+0000 7f0d2d400640 1 -- 172.21.15.90:0/3723125966 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d28068fb0 con 0x7f0d280d24a0 2024-04-25T01:16:55.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.034+0000 7f6338e00640 1 -- 172.21.15.90:0/1973629256 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6334068fb0 con 0x7f63340b4af0 2024-04-25T01:16:55.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.035+0000 7f0d1ea00640 1 -- 172.21.15.90:0/3723125966 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0d1003cbf0 con 0x7f0d280d24a0 2024-04-25T01:16:55.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.036+0000 7f0d1ea00640 1 --2- 172.21.15.90:0/3723125966 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0d0406be80 0x7f0d0406e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.036+0000 7f0d1ea00640 1 -- 172.21.15.90:0/3723125966 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f0d100abd90 con 0x7f0d280d24a0 2024-04-25T01:16:55.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.036+0000 7f632aa00640 1 -- 172.21.15.90:0/1973629256 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f631c01a070 con 0x7f63340b4af0 2024-04-25T01:16:55.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.036+0000 7f0d26800640 1 --2- 172.21.15.90:0/3723125966 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0d0406be80 0x7f0d0406e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.037+0000 7f0d26800640 1 --2- 172.21.15.90:0/3723125966 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0d0406be80 0x7f0d0406e340 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0d18001560 tx=0x7f0d18009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:55.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.037+0000 7f632aa00640 1 --2- 172.21.15.90:0/1973629256 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f631006be80 0x7f631006e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.037+0000 7f632aa00640 1 -- 172.21.15.90:0/1973629256 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f631c08a3e0 con 0x7f63340b4af0 2024-04-25T01:16:55.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.037+0000 7f7f03400640 1 -- 172.21.15.90:0/1688390906 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef8037040 con 0x7f7f0403c0a0 2024-04-25T01:16:55.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.037+0000 7f7f0be00640 1 -- 172.21.15.90:0/1688390906 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f0417daf0 con 0x7f7f0403c0a0 2024-04-25T01:16:55.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.038+0000 7f7f0be00640 1 -- 172.21.15.90:0/1688390906 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f0417e0b0 con 0x7f7f0403c0a0 2024-04-25T01:16:55.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.038+0000 7f6331e00640 1 --2- 172.21.15.90:0/1973629256 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f631006be80 0x7f631006e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.038+0000 7f4823e00640 1 -- 172.21.15.90:0/286505113 <== mon.2 v2:172.21.15.103:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f481c0591c0 con 0x7f482c06cfc0 2024-04-25T01:16:55.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.039+0000 7f6331e00640 1 --2- 172.21.15.90:0/1973629256 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f631006be80 0x7f631006e340 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f63240012e0 tx=0x7f6324009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:55.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.039+0000 7f89fea00640 1 -- 172.21.15.90:0/2570345249 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f89f801f020 con 0x7f8a0815b060 2024-04-25T01:16:55.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.040+0000 7f7f0be00640 1 -- 172.21.15.90:0/1688390906 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f04005390 con 0x7f7f0403c0a0 2024-04-25T01:16:55.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.041+0000 7fdf37e00640 1 -- 172.21.15.90:0/2856357372 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf28031600 con 0x7fdf380cefe0 2024-04-25T01:16:55.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.041+0000 7fdf37e00640 1 -- 172.21.15.90:0/2856357372 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf280317a0 con 0x7fdf380cefe0 2024-04-25T01:16:55.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.041+0000 7fdf37e00640 1 -- 172.21.15.90:0/2856357372 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf28031b70 con 0x7fdf380cefe0 2024-04-25T01:16:55.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.042+0000 7fdf3f800640 1 -- 172.21.15.90:0/2856357372 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf380cefe0 msgr2=0x7fdf380b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.042+0000 7fdf3f800640 1 --2- 172.21.15.90:0/2856357372 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf380cefe0 0x7fdf380b4af0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdf28009d90 tx=0x7fdf2802a610 comp rx=0 tx=0).stop 2024-04-25T01:16:55.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.042+0000 7fdf3f800640 1 -- 172.21.15.90:0/2856357372 shutdown_connections 2024-04-25T01:16:55.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.042+0000 7fdf3f800640 1 --2- 172.21.15.90:0/2856357372 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf380cefe0 0x7fdf380b4af0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.042+0000 7fdf3f800640 1 -- 172.21.15.90:0/2856357372 >> 172.21.15.90:0/2856357372 conn(0x7fdf380b1820 msgr2=0x7fdf380b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:55.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.042+0000 7fdf3f800640 1 -- 172.21.15.90:0/2856357372 shutdown_connections 2024-04-25T01:16:55.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.042+0000 7fdf3f800640 1 -- 172.21.15.90:0/2856357372 wait complete. 2024-04-25T01:16:55.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.043+0000 7f0d1ea00640 1 -- 172.21.15.90:0/3723125966 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0d280b8ac0 con 0x7f0d280d24a0 2024-04-25T01:16:55.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.045+0000 7fdf3f800640 1 Processor -- start 2024-04-25T01:16:55.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.045+0000 7fdf3f800640 1 -- start start 2024-04-25T01:16:55.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.046+0000 7fdf3f800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf3806e000 0x7fdf38152ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.047+0000 7fdf3f800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf380b2b20 0x7fdf38155e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.047+0000 7fdf3f800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf380cefe0 0x7fdf381563a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.047+0000 7fdf3f800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdf38071360 con 0x7fdf380cefe0 2024-04-25T01:16:55.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.047+0000 7fdf3f800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdf3806e490 con 0x7fdf380b2b20 2024-04-25T01:16:55.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.047+0000 7fdf3f800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdf380714e0 con 0x7fdf3806e000 2024-04-25T01:16:55.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.047+0000 7f632aa00640 1 -- 172.21.15.90:0/1973629256 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f631c0583e0 con 0x7f63340b4af0 2024-04-25T01:16:55.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.047+0000 7fdf3d800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf380cefe0 0x7fdf381563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.047+0000 7fdf3ce00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf3806e000 0x7fdf38152ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.047+0000 7fdf3d800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf380cefe0 0x7fdf381563a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56164/0 (socket says 172.21.15.90:56164) 2024-04-25T01:16:55.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.047+0000 7fdf3d800640 1 -- 172.21.15.90:0/707329156 learned_addr learned my addr 172.21.15.90:0/707329156 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:55.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.047+0000 7fdf2fe00640 1 --2- 172.21.15.90:0/707329156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf380b2b20 0x7fdf38155e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.048+0000 7fdf3d800640 1 -- 172.21.15.90:0/707329156 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf3806e000 msgr2=0x7fdf38152ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.048+0000 7fdf3d800640 1 --2- 172.21.15.90:0/707329156 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf3806e000 0x7fdf38152ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.048+0000 7fdf3d800640 1 -- 172.21.15.90:0/707329156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf380b2b20 msgr2=0x7fdf38155e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.048+0000 7fdf3d800640 1 --2- 172.21.15.90:0/707329156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf380b2b20 0x7fdf38155e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.048+0000 7fdf3d800640 1 -- 172.21.15.90:0/707329156 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf38153220 con 0x7fdf380cefe0 2024-04-25T01:16:55.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.049+0000 7fdf3d800640 1 --2- 172.21.15.90:0/707329156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf380cefe0 0x7fdf381563a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdf3000c3c0 tx=0x7fdf3000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:55.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.049+0000 7fdf35600640 1 -- 172.21.15.90:0/707329156 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf300052d0 con 0x7fdf380cefe0 2024-04-25T01:16:55.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.049+0000 7fdf35600640 1 -- 172.21.15.90:0/707329156 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf30005490 con 0x7fdf380cefe0 2024-04-25T01:16:55.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.049+0000 7fdf35600640 1 -- 172.21.15.90:0/707329156 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf30007260 con 0x7fdf380cefe0 2024-04-25T01:16:55.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.050+0000 7fdf3f800640 1 -- 172.21.15.90:0/707329156 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf381eca80 con 0x7fdf380cefe0 2024-04-25T01:16:55.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.050+0000 7fdf3f800640 1 -- 172.21.15.90:0/707329156 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf381ecd20 con 0x7fdf380cefe0 2024-04-25T01:16:55.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.052+0000 7f7f03400640 1 -- 172.21.15.90:0/1688390906 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ef8042070 con 0x7f7f0403c0a0 2024-04-25T01:16:55.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.052+0000 7f7f03400640 1 -- 172.21.15.90:0/1688390906 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef8030970 con 0x7f7f0403c0a0 2024-04-25T01:16:55.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.052+0000 7fdf3f800640 1 -- 172.21.15.90:0/707329156 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf38068fb0 con 0x7fdf380cefe0 2024-04-25T01:16:55.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.052+0000 7fee61600640 1 -- 172.21.15.90:0/2228745707 <== mon.2 v2:172.21.15.103:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fee50079270 con 0x7fee4c003dd0 2024-04-25T01:16:55.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.054+0000 7fdf35600640 1 -- 172.21.15.90:0/707329156 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fdf30007400 con 0x7fdf380cefe0 2024-04-25T01:16:55.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.056+0000 7fdf35600640 1 --2- 172.21.15.90:0/707329156 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdf1406be80 0x7fdf1406e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.055+0000 7f7f03400640 1 -- 172.21.15.90:0/1688390906 <== mon.2 v2:172.21.15.103:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7ef8030b10 con 0x7f7f0403c0a0 2024-04-25T01:16:55.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.056+0000 7fdf35600640 1 -- 172.21.15.90:0/707329156 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fdf30088890 con 0x7fdf380cefe0 2024-04-25T01:16:55.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.057+0000 7f7f03400640 1 --2- 172.21.15.90:0/1688390906 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7eec06be30 0x7f7eec06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:55.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.057+0000 7f7f03400640 1 -- 172.21.15.90:0/1688390906 <== mon.2 v2:172.21.15.103:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f7ef80ac760 con 0x7f7f0403c0a0 2024-04-25T01:16:55.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.058+0000 7fdf3ce00640 1 --2- 172.21.15.90:0/707329156 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdf1406be80 0x7fdf1406e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.058+0000 7f7f0a000640 1 --2- 172.21.15.90:0/1688390906 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7eec06be30 0x7f7eec06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:55.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.059+0000 7f7f0a000640 1 --2- 172.21.15.90:0/1688390906 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7eec06be30 0x7f7eec06e2f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7f0400e150 tx=0x7f7efc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:55.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.060+0000 7fdf3ce00640 1 --2- 172.21.15.90:0/707329156 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdf1406be80 0x7fdf1406e340 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdf28034000 tx=0x7fdf2802f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:55.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.064+0000 7fdf35600640 1 -- 172.21.15.90:0/707329156 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fdf300585b0 con 0x7fdf380cefe0 2024-04-25T01:16:55.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.065+0000 7f7f03400640 1 -- 172.21.15.90:0/1688390906 <== mon.2 v2:172.21.15.103:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7ef807adb0 con 0x7f7f0403c0a0 2024-04-25T01:16:55.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.456+0000 7fb335c00640 1 -- 172.21.15.90:0/3710900340 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fb32804fda0 con 0x7fb328005e90 2024-04-25T01:16:55.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.457+0000 7fb323e00640 1 -- 172.21.15.90:0/3710900340 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb3240172c0 con 0x7fb328005e90 2024-04-25T01:16:55.444 INFO:teuthology.orchestra.run.smithi090.stdout:60129542153 2024-04-25T01:16:55.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.461+0000 7f4821600640 1 -- 172.21.15.90:0/286505113 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f4800005850 con 0x7f482c06cfc0 2024-04-25T01:16:55.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.461+0000 7fb335c00640 1 -- 172.21.15.90:0/3710900340 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb31006be80 msgr2=0x7fb31006e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.461+0000 7fb335c00640 1 --2- 172.21.15.90:0/3710900340 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb31006be80 0x7fb31006e340 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb31c034000 tx=0x7fb31c02e000 comp rx=0 tx=0).stop 2024-04-25T01:16:55.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.462+0000 7fb335c00640 1 -- 172.21.15.90:0/3710900340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb328005e90 msgr2=0x7fb328180b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.462+0000 7f4823e00640 1 -- 172.21.15.90:0/286505113 <== mon.2 v2:172.21.15.103:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f481c05d4c0 con 0x7f482c06cfc0 2024-04-25T01:16:55.446 INFO:teuthology.orchestra.run.smithi090.stdout:60129542152 2024-04-25T01:16:55.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.462+0000 7fb335c00640 1 --2- 172.21.15.90:0/3710900340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb328005e90 0x7fb328180b60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb32400b3e0 tx=0x7fb32400b7e0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.463+0000 7fb335c00640 1 -- 172.21.15.90:0/3710900340 shutdown_connections 2024-04-25T01:16:55.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.463+0000 7fb335c00640 1 --2- 172.21.15.90:0/3710900340 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb31006be80 0x7fb31006e340 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.463+0000 7fb335c00640 1 --2- 172.21.15.90:0/3710900340 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb3280ead00 0x7fb32804de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.463+0000 7fb335c00640 1 --2- 172.21.15.90:0/3710900340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb328006810 0x7fb32817de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.463+0000 7fb335c00640 1 --2- 172.21.15.90:0/3710900340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb328005e90 0x7fb328180b60 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.463+0000 7fb335c00640 1 -- 172.21.15.90:0/3710900340 >> 172.21.15.90:0/3710900340 conn(0x7fb328009fb0 msgr2=0x7fb328048bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:55.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.463+0000 7fb335c00640 1 -- 172.21.15.90:0/3710900340 shutdown_connections 2024-04-25T01:16:55.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.463+0000 7fb335c00640 1 -- 172.21.15.90:0/3710900340 wait complete. 2024-04-25T01:16:55.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.466+0000 7f4821600640 1 -- 172.21.15.90:0/286505113 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f480c06be30 msgr2=0x7f480c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.466+0000 7f4821600640 1 --2- 172.21.15.90:0/286505113 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f480c06be30 0x7f480c06e2f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4818034000 tx=0x7f481802e000 comp rx=0 tx=0).stop 2024-04-25T01:16:55.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.466+0000 7f4821600640 1 -- 172.21.15.90:0/286505113 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f482c06cfc0 msgr2=0x7f482c0b0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.466+0000 7f4821600640 1 --2- 172.21.15.90:0/286505113 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f482c06cfc0 0x7f482c0b0910 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f481c00e810 tx=0x7f481c00ecd0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.467+0000 7f4821600640 1 -- 172.21.15.90:0/286505113 shutdown_connections 2024-04-25T01:16:55.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.467+0000 7f4821600640 1 --2- 172.21.15.90:0/286505113 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f480c06be30 0x7f480c06e2f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.467+0000 7f4821600640 1 --2- 172.21.15.90:0/286505113 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f482c154660 0x7f482c149650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.467+0000 7f4821600640 1 --2- 172.21.15.90:0/286505113 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f482c154280 0x7f482c149110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.467+0000 7f4821600640 1 --2- 172.21.15.90:0/286505113 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f482c06cfc0 0x7f482c0b0910 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.467+0000 7f4821600640 1 -- 172.21.15.90:0/286505113 >> 172.21.15.90:0/286505113 conn(0x7f482c0b1760 msgr2=0x7f482c15adb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:55.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.467+0000 7f4821600640 1 -- 172.21.15.90:0/286505113 shutdown_connections 2024-04-25T01:16:55.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.467+0000 7f4821600640 1 -- 172.21.15.90:0/286505113 wait complete. 2024-04-25T01:16:55.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.474+0000 7f8a0da00640 1 -- 172.21.15.90:0/2570345249 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f8a08072940 con 0x7f8a0815b060 2024-04-25T01:16:55.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.475+0000 7f6338e00640 1 -- 172.21.15.90:0/1973629256 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f63340cbb00 con 0x7f63340b4af0 2024-04-25T01:16:55.460 INFO:teuthology.orchestra.run.smithi090.stdout:60129542152 2024-04-25T01:16:55.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.475+0000 7f89fea00640 1 -- 172.21.15.90:0/2570345249 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f89f8058fa0 con 0x7f8a0815b060 2024-04-25T01:16:55.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.476+0000 7f632aa00640 1 -- 172.21.15.90:0/1973629256 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f631c05c6e0 con 0x7f63340b4af0 2024-04-25T01:16:55.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.478+0000 7f0d2d400640 1 -- 172.21.15.90:0/3723125966 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f0d2809d750 con 0x7f0d280d24a0 2024-04-25T01:16:55.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.479+0000 7f0d1ea00640 1 -- 172.21.15.90:0/3723125966 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0d10046360 con 0x7f0d280d24a0 2024-04-25T01:16:55.463 INFO:teuthology.orchestra.run.smithi090.stdout:60129542152 2024-04-25T01:16:55.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.479+0000 7f8a0da00640 1 -- 172.21.15.90:0/2570345249 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f89e806be30 msgr2=0x7f89e806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.479+0000 7f8a0da00640 1 --2- 172.21.15.90:0/2570345249 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f89e806be30 0x7f89e806e2f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8a00034000 tx=0x7f8a00009000 comp rx=0 tx=0).stop 2024-04-25T01:16:55.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.480+0000 7f8a0da00640 1 -- 172.21.15.90:0/2570345249 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a0815b060 msgr2=0x7f8a080cca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.480+0000 7f8a0da00640 1 --2- 172.21.15.90:0/2570345249 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a0815b060 0x7f8a080cca60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f89f800b410 tx=0x7f89f800b8d0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.480+0000 7f8a0da00640 1 -- 172.21.15.90:0/2570345249 shutdown_connections 2024-04-25T01:16:55.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.480+0000 7f8a0da00640 1 --2- 172.21.15.90:0/2570345249 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f89e806be30 0x7f89e806e2f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.480+0000 7f8a0da00640 1 --2- 172.21.15.90:0/2570345249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a0815b4f0 0x7f8a081f7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.480+0000 7f8a0da00640 1 --2- 172.21.15.90:0/2570345249 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a0815b060 0x7f8a080cca60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.480+0000 7f8a0da00640 1 --2- 172.21.15.90:0/2570345249 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8a080d0650 0x7f8a080cbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.480+0000 7f8a0da00640 1 -- 172.21.15.90:0/2570345249 >> 172.21.15.90:0/2570345249 conn(0x7f8a080cfb30 msgr2=0x7f8a080af750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:55.465 INFO:teuthology.orchestra.run.smithi090.stdout:60129542154 2024-04-25T01:16:55.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.480+0000 7f8a0da00640 1 -- 172.21.15.90:0/2570345249 shutdown_connections 2024-04-25T01:16:55.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.481+0000 7f8a0da00640 1 -- 172.21.15.90:0/2570345249 wait complete. 2024-04-25T01:16:55.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.482+0000 7f6338e00640 1 -- 172.21.15.90:0/1973629256 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f631006be80 msgr2=0x7f631006e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.482+0000 7f6338e00640 1 --2- 172.21.15.90:0/1973629256 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f631006be80 0x7f631006e340 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f63240012e0 tx=0x7f6324009040 comp rx=0 tx=0).stop 2024-04-25T01:16:55.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.482+0000 7f6338e00640 1 -- 172.21.15.90:0/1973629256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63340b4af0 msgr2=0x7f6334154790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.482+0000 7f6338e00640 1 --2- 172.21.15.90:0/1973629256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63340b4af0 0x7f6334154790 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f631c00c530 tx=0x7f631c00c9f0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.483+0000 7f6338e00640 1 -- 172.21.15.90:0/1973629256 shutdown_connections 2024-04-25T01:16:55.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.483+0000 7f6338e00640 1 --2- 172.21.15.90:0/1973629256 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f631006be80 0x7f631006e340 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.483+0000 7f6338e00640 1 --2- 172.21.15.90:0/1973629256 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63340cefe0 0x7f6334164670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.483+0000 7f6338e00640 1 --2- 172.21.15.90:0/1973629256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63340b4af0 0x7f6334154790 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.483+0000 7f6338e00640 1 --2- 172.21.15.90:0/1973629256 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63340b2b20 0x7f6334157380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.483+0000 7f6338e00640 1 -- 172.21.15.90:0/1973629256 >> 172.21.15.90:0/1973629256 conn(0x7f63340b1820 msgr2=0x7f6334163be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:55.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.483+0000 7f6338e00640 1 -- 172.21.15.90:0/1973629256 shutdown_connections 2024-04-25T01:16:55.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.484+0000 7f6338e00640 1 -- 172.21.15.90:0/1973629256 wait complete. 2024-04-25T01:16:55.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.485+0000 7f0d2d400640 1 -- 172.21.15.90:0/3723125966 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0d0406be80 msgr2=0x7f0d0406e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.485+0000 7f0d2d400640 1 --2- 172.21.15.90:0/3723125966 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0d0406be80 0x7f0d0406e340 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0d18001560 tx=0x7f0d18009040 comp rx=0 tx=0).stop 2024-04-25T01:16:55.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.485+0000 7f0d2d400640 1 -- 172.21.15.90:0/3723125966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d280d24a0 msgr2=0x7f0d28150160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.485+0000 7f0d2d400640 1 --2- 172.21.15.90:0/3723125966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d280d24a0 0x7f0d28150160 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0d1002a790 tx=0x7f0d100046c0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.485+0000 7f0d2d400640 1 -- 172.21.15.90:0/3723125966 shutdown_connections 2024-04-25T01:16:55.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.485+0000 7f0d2d400640 1 --2- 172.21.15.90:0/3723125966 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0d0406be80 0x7f0d0406e340 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.485+0000 7f0d2d400640 1 --2- 172.21.15.90:0/3723125966 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0d280d2880 0x7f0d281506a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.485+0000 7f0d2d400640 1 --2- 172.21.15.90:0/3723125966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d280d24a0 0x7f0d28150160 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.485+0000 7f0d2d400640 1 --2- 172.21.15.90:0/3723125966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d280b2da0 0x7f0d280b8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.485+0000 7f0d2d400640 1 -- 172.21.15.90:0/3723125966 >> 172.21.15.90:0/3723125966 conn(0x7f0d2809d200 msgr2=0x7f0d280cafd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:55.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.486+0000 7f0d2d400640 1 -- 172.21.15.90:0/3723125966 shutdown_connections 2024-04-25T01:16:55.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.486+0000 7f0d2d400640 1 -- 172.21.15.90:0/3723125966 wait complete. 2024-04-25T01:16:55.473 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542153 for osd.5 2024-04-25T01:16:55.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.491+0000 7fee56a00640 1 -- 172.21.15.90:0/2228745707 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fee38005580 con 0x7fee4c003dd0 2024-04-25T01:16:55.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.491+0000 7f7f0be00640 1 -- 172.21.15.90:0/1688390906 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f7f04057d70 con 0x7f7f0403c0a0 2024-04-25T01:16:55.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.492+0000 7f7f03400640 1 -- 172.21.15.90:0/1688390906 <== mon.2 v2:172.21.15.103:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7ef807a6b0 con 0x7f7f0403c0a0 2024-04-25T01:16:55.476 INFO:teuthology.orchestra.run.smithi090.stdout:60129542152 2024-04-25T01:16:55.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.492+0000 7fee61600640 1 -- 172.21.15.90:0/2228745707 <== mon.2 v2:172.21.15.103:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fee5007d570 con 0x7fee4c003dd0 2024-04-25T01:16:55.477 INFO:teuthology.orchestra.run.smithi090.stdout:60129542153 2024-04-25T01:16:55.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.494+0000 7f7f0be00640 1 -- 172.21.15.90:0/1688390906 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7eec06be30 msgr2=0x7f7eec06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.494+0000 7f7f0be00640 1 --2- 172.21.15.90:0/1688390906 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7eec06be30 0x7f7eec06e2f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7f0400e150 tx=0x7f7efc009040 comp rx=0 tx=0).stop 2024-04-25T01:16:55.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.494+0000 7f7f0be00640 1 -- 172.21.15.90:0/1688390906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f0403c0a0 msgr2=0x7f7f040de1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.494+0000 7f7f0be00640 1 --2- 172.21.15.90:0/1688390906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f0403c0a0 0x7f7f040de1d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8000c00 tx=0x7f7ef80046c0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.495+0000 7f7f0be00640 1 -- 172.21.15.90:0/1688390906 shutdown_connections 2024-04-25T01:16:55.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.495+0000 7f7f0be00640 1 --2- 172.21.15.90:0/1688390906 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7eec06be30 0x7f7eec06e2f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.495+0000 7f7f0be00640 1 --2- 172.21.15.90:0/1688390906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f040e3160 0x7f7f0403d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.495+0000 7f7f0be00640 1 --2- 172.21.15.90:0/1688390906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f04053cb0 0x7f7f0417d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.495+0000 7f7f0be00640 1 --2- 172.21.15.90:0/1688390906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f0403c0a0 0x7f7f040de1d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.495+0000 7f7f0be00640 1 -- 172.21.15.90:0/1688390906 >> 172.21.15.90:0/1688390906 conn(0x7f7f04058ee0 msgr2=0x7f7f04037cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:55.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.495+0000 7f7f0be00640 1 -- 172.21.15.90:0/1688390906 shutdown_connections 2024-04-25T01:16:55.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.495+0000 7f7f0be00640 1 -- 172.21.15.90:0/1688390906 wait complete. 2024-04-25T01:16:55.480 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542152 for osd.3 2024-04-25T01:16:55.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.496+0000 7fee56a00640 1 -- 172.21.15.90:0/2228745707 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fee4406c060 msgr2=0x7fee4406e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.496+0000 7fee56a00640 1 --2- 172.21.15.90:0/2228745707 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fee4406c060 0x7fee4406e520 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fee5c007240 tx=0x7fee58009040 comp rx=0 tx=0).stop 2024-04-25T01:16:55.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.496+0000 7fee56a00640 1 -- 172.21.15.90:0/2228745707 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee4c003dd0 msgr2=0x7fee5c0dab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.496+0000 7fee56a00640 1 --2- 172.21.15.90:0/2228745707 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee4c003dd0 0x7fee5c0dab20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fee500098f0 tx=0x7fee50004700 comp rx=0 tx=0).stop 2024-04-25T01:16:55.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.497+0000 7fee56a00640 1 -- 172.21.15.90:0/2228745707 shutdown_connections 2024-04-25T01:16:55.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.497+0000 7fee56a00640 1 --2- 172.21.15.90:0/2228745707 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fee4406c060 0x7fee4406e520 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.497+0000 7fee56a00640 1 --2- 172.21.15.90:0/2228745707 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee5c04c2b0 0x7fee5c039350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.497+0000 7fee56a00640 1 --2- 172.21.15.90:0/2228745707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee5c17de20 0x7fee5c04bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.497+0000 7fee56a00640 1 --2- 172.21.15.90:0/2228745707 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fee4c003dd0 0x7fee5c0dab20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.497+0000 7fee56a00640 1 -- 172.21.15.90:0/2228745707 >> 172.21.15.90:0/2228745707 conn(0x7fee5c054ca0 msgr2=0x7fee5c055080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:55.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.497+0000 7fdf3f800640 1 -- 172.21.15.90:0/707329156 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fdf380cc320 con 0x7fdf380cefe0 2024-04-25T01:16:55.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.497+0000 7fee56a00640 1 -- 172.21.15.90:0/2228745707 shutdown_connections 2024-04-25T01:16:55.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.497+0000 7fee56a00640 1 -- 172.21.15.90:0/2228745707 wait complete. 2024-04-25T01:16:55.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.498+0000 7fdf35600640 1 -- 172.21.15.90:0/707329156 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdf3005c8b0 con 0x7fdf380cefe0 2024-04-25T01:16:55.483 INFO:teuthology.orchestra.run.smithi090.stdout:60129542153 2024-04-25T01:16:55.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.502+0000 7fdf3f800640 1 -- 172.21.15.90:0/707329156 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdf1406be80 msgr2=0x7fdf1406e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.502+0000 7fdf3f800640 1 --2- 172.21.15.90:0/707329156 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdf1406be80 0x7fdf1406e340 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdf28034000 tx=0x7fdf2802f000 comp rx=0 tx=0).stop 2024-04-25T01:16:55.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.502+0000 7fdf3f800640 1 -- 172.21.15.90:0/707329156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf380cefe0 msgr2=0x7fdf381563a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:55.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.502+0000 7fdf3f800640 1 --2- 172.21.15.90:0/707329156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf380cefe0 0x7fdf381563a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdf3000c3c0 tx=0x7fdf3000c880 comp rx=0 tx=0).stop 2024-04-25T01:16:55.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.502+0000 7fdf3f800640 1 -- 172.21.15.90:0/707329156 shutdown_connections 2024-04-25T01:16:55.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.502+0000 7fdf3f800640 1 --2- 172.21.15.90:0/707329156 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdf1406be80 0x7fdf1406e340 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.502+0000 7fdf3f800640 1 --2- 172.21.15.90:0/707329156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf380cefe0 0x7fdf381563a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.502+0000 7fdf3f800640 1 --2- 172.21.15.90:0/707329156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf380b2b20 0x7fdf38155e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.502+0000 7fdf3f800640 1 --2- 172.21.15.90:0/707329156 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf3806e000 0x7fdf38152ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:55.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.502+0000 7fdf3f800640 1 -- 172.21.15.90:0/707329156 >> 172.21.15.90:0/707329156 conn(0x7fdf380b1820 msgr2=0x7fdf38164140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:55.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.502+0000 7fdf3f800640 1 -- 172.21.15.90:0/707329156 shutdown_connections 2024-04-25T01:16:55.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:55.502+0000 7fdf3f800640 1 -- 172.21.15.90:0/707329156 wait complete. 2024-04-25T01:16:55.492 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542152 for osd.4 2024-04-25T01:16:55.497 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542152 for osd.6 2024-04-25T01:16:55.498 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542155 got 60129542154 for osd.2 2024-04-25T01:16:55.504 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542152 for osd.1 2024-04-25T01:16:55.504 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542153 for osd.0 2024-04-25T01:16:55.510 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542153 for osd.7 2024-04-25T01:16:56.474 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-25T01:16:56.481 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-25T01:16:56.493 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-25T01:16:56.497 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-25T01:16:56.499 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-25T01:16:56.506 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-25T01:16:56.506 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-25T01:16:56.512 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-25T01:16:56.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.763+0000 7f9a09a00640 1 Processor -- start 2024-04-25T01:16:56.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.763+0000 7f331fe00640 1 Processor -- start 2024-04-25T01:16:56.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.763+0000 7f9a09a00640 1 -- start start 2024-04-25T01:16:56.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.763+0000 7f331fe00640 1 -- start start 2024-04-25T01:16:56.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.765+0000 7f331fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3318005e90 0x7f33180062d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.765+0000 7f331fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3318006810 0x7f3318006ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.765+0000 7f331fe00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3318053720 con 0x7f33180ead00 2024-04-25T01:16:56.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.766+0000 7f9a09a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f99fc0e1310 0x7f99fc03a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.767+0000 7f9a09a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f99fc00ab40 0x7f99fc009d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.765+0000 7f331fe00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f3318054330 con 0x7f3318006810 2024-04-25T01:16:56.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.765+0000 7f331fe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3318053a00 con 0x7f3318005e90 2024-04-25T01:16:56.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.766+0000 7f331ea00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f33180ead00 0x7f331804fa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:35592/0 (socket says 172.21.15.90:35592) 2024-04-25T01:16:56.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.766+0000 7f331ea00640 1 -- 172.21.15.90:0/3002279175 learned_addr learned my addr 172.21.15.90:0/3002279175 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:56.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.766+0000 7f331f400640 1 --2- 172.21.15.90:0/3002279175 >> v2:172.21.15.90:3300/0 conn(0x7f3318006810 0x7f3318006ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.766+0000 7f331e000640 1 --2- 172.21.15.90:0/3002279175 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3318005e90 0x7f33180062d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.767+0000 7f331f400640 1 -- 172.21.15.90:0/3002279175 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3318005e90 msgr2=0x7f33180062d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:56.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.767+0000 7f9a09a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99fc03a6b0 con 0x7f99fc0516e0 2024-04-25T01:16:56.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.767+0000 7f331f400640 1 --2- 172.21.15.90:0/3002279175 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3318005e90 0x7f33180062d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:56.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.767+0000 7f331f400640 1 -- 172.21.15.90:0/3002279175 >> v1:172.21.15.90:6789/0 conn(0x7f33180ead00 legacy=0x7f331804fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:56.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.768+0000 7f9a09a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f99fc055920 con 0x7f99fc00ab40 2024-04-25T01:16:56.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.768+0000 7f9a09a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f99fc054d10 con 0x7f99fc0e1310 2024-04-25T01:16:56.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.768+0000 7f9a03400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f99fc00ab40 0x7f99fc009d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.768+0000 7f9a09000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f99fc0516e0 0x7f99fc055db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:35608/0 (socket says 172.21.15.90:35608) 2024-04-25T01:16:56.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.768+0000 7f9a09000640 1 -- 172.21.15.90:0/2353339056 learned_addr learned my addr 172.21.15.90:0/2353339056 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:56.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.768+0000 7f9a03e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f99fc0e1310 0x7f99fc03a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.767+0000 7f331f400640 1 -- 172.21.15.90:0/3002279175 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3318053ce0 con 0x7f3318006810 2024-04-25T01:16:56.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.769+0000 7f9a03e00640 1 -- 172.21.15.90:0/2353339056 >> v2:172.21.15.103:3300/0 conn(0x7f99fc00ab40 msgr2=0x7f99fc009d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:56.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.769+0000 7f9a03e00640 1 --2- 172.21.15.90:0/2353339056 >> v2:172.21.15.103:3300/0 conn(0x7f99fc00ab40 0x7f99fc009d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:56.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.769+0000 7f9a03e00640 1 -- 172.21.15.90:0/2353339056 >> v1:172.21.15.90:6789/0 conn(0x7f99fc0516e0 legacy=0x7f99fc055db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:56.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.769+0000 7f9a03e00640 1 -- 172.21.15.90:0/2353339056 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99fc054ff0 con 0x7f99fc0e1310 2024-04-25T01:16:56.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.770+0000 7f331f400640 1 --2- 172.21.15.90:0/3002279175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3318006810 0x7f3318006ca0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3320000b60 tx=0x7f33200a5d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50bd6286c02ae971 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:56.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.771+0000 7f331d600640 1 -- 172.21.15.90:0/3002279175 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33200b3070 con 0x7f3318006810 2024-04-25T01:16:56.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.771+0000 7f331d600640 1 -- 172.21.15.90:0/3002279175 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33200a6b50 con 0x7f3318006810 2024-04-25T01:16:56.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.771+0000 7f331d600640 1 -- 172.21.15.90:0/3002279175 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33200ae440 con 0x7f3318006810 2024-04-25T01:16:56.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.772+0000 7f331fe00640 1 -- 172.21.15.90:0/3002279175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3318006810 msgr2=0x7f3318006ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:56.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.772+0000 7f331fe00640 1 --2- 172.21.15.90:0/3002279175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3318006810 0x7f3318006ca0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3320000b60 tx=0x7f33200a5d80 comp rx=0 tx=0).stop 2024-04-25T01:16:56.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.772+0000 7f9a03e00640 1 --2- 172.21.15.90:0/2353339056 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f99fc0e1310 0x7f99fc03a170 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f99f00097a0 tx=0x7f99f002e520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5eb18689372aa7fa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:56.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.772+0000 7f331fe00640 1 -- 172.21.15.90:0/3002279175 shutdown_connections 2024-04-25T01:16:56.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.772+0000 7f331fe00640 1 --2- 172.21.15.90:0/3002279175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3318006810 0x7f3318006ca0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:56.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.772+0000 7f331fe00640 1 --2- 172.21.15.90:0/3002279175 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3318005e90 0x7f33180062d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:56.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.772+0000 7f331fe00640 1 -- 172.21.15.90:0/3002279175 >> 172.21.15.90:0/3002279175 conn(0x7f3318009fb0 msgr2=0x7f331800a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:56.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.773+0000 7f331fe00640 1 -- 172.21.15.90:0/3002279175 shutdown_connections 2024-04-25T01:16:56.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.773+0000 7f9a02a00640 1 -- 172.21.15.90:0/2353339056 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f003a070 con 0x7f99fc0e1310 2024-04-25T01:16:56.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.773+0000 7f9a02a00640 1 -- 172.21.15.90:0/2353339056 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99f0033710 con 0x7f99fc0e1310 2024-04-25T01:16:56.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.773+0000 7f331fe00640 1 -- 172.21.15.90:0/3002279175 wait complete. 2024-04-25T01:16:56.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.775+0000 7f331fe00640 1 Processor -- start 2024-04-25T01:16:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.774+0000 7f9a09a00640 1 -- 172.21.15.90:0/2353339056 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f99fc0e1310 msgr2=0x7f99fc03a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.774+0000 7f9a09a00640 1 --2- 172.21.15.90:0/2353339056 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f99fc0e1310 0x7f99fc03a170 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f99f00097a0 tx=0x7f99f002e520 comp rx=0 tx=0).stop 2024-04-25T01:16:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.776+0000 7f331fe00640 1 -- start start 2024-04-25T01:16:56.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.777+0000 7f9a09a00640 1 -- 172.21.15.90:0/2353339056 shutdown_connections 2024-04-25T01:16:56.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.777+0000 7f9a09a00640 1 --2- 172.21.15.90:0/2353339056 >> v2:172.21.15.103:3300/0 conn(0x7f99fc00ab40 0x7f99fc009d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:56.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.777+0000 7f9a09a00640 1 --2- 172.21.15.90:0/2353339056 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f99fc0e1310 0x7f99fc03a170 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:56.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.777+0000 7f9a09a00640 1 -- 172.21.15.90:0/2353339056 >> 172.21.15.90:0/2353339056 conn(0x7f99fc056cd0 msgr2=0x7f99fc0570e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:56.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.777+0000 7f9a09a00640 1 -- 172.21.15.90:0/2353339056 shutdown_connections 2024-04-25T01:16:56.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.777+0000 7f331fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3318005e90 0x7f33180d7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.777+0000 7f9a09a00640 1 -- 172.21.15.90:0/2353339056 wait complete. 2024-04-25T01:16:56.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.777+0000 7f331fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3318006810 0x7f3318039360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.778+0000 7f331fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33180ead00 0x7f33180398a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.778+0000 7f331fe00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f33180547c0 con 0x7f33180ead00 2024-04-25T01:16:56.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.778+0000 7f331e000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3318006810 0x7f3318039360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.778+0000 7f331e000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3318006810 0x7f3318039360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.90:48164/0 (socket says 172.21.15.90:48164) 2024-04-25T01:16:56.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.779+0000 7f331e000640 1 -- 172.21.15.90:0/72273572 learned_addr learned my addr 172.21.15.90:0/72273572 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:56.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.779+0000 7f331fe00640 1 -- 172.21.15.90:0/72273572 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f33180501a0 con 0x7f3318005e90 2024-04-25T01:16:56.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.779+0000 7f331fe00640 1 -- 172.21.15.90:0/72273572 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3318054940 con 0x7f3318006810 2024-04-25T01:16:56.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.779+0000 7f331ea00640 1 --2- 172.21.15.90:0/72273572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3318005e90 0x7f33180d7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.779+0000 7f331f400640 1 --2- 172.21.15.90:0/72273572 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33180ead00 0x7f33180398a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.780+0000 7f331ea00640 1 -- 172.21.15.90:0/72273572 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3318006810 msgr2=0x7f3318039360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:56.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.780+0000 7f331ea00640 1 --2- 172.21.15.90:0/72273572 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3318006810 0x7f3318039360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:56.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.780+0000 7f331ea00640 1 -- 172.21.15.90:0/72273572 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33180ead00 msgr2=0x7f33180398a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:56.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.780+0000 7f331ea00640 1 --2- 172.21.15.90:0/72273572 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33180ead00 0x7f33180398a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:56.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.780+0000 7f331ea00640 1 -- 172.21.15.90:0/72273572 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33180edf70 con 0x7f3318005e90 2024-04-25T01:16:56.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.781+0000 7f331ea00640 1 --2- 172.21.15.90:0/72273572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3318005e90 0x7f33180d7d80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f331000c3e0 tx=0x7f331000c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:56.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.782+0000 7f3316a00640 1 -- 172.21.15.90:0/72273572 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3310017070 con 0x7f3318005e90 2024-04-25T01:16:56.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.782+0000 7f3316a00640 1 -- 172.21.15.90:0/72273572 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3310005ce0 con 0x7f3318005e90 2024-04-25T01:16:56.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.782+0000 7f3316a00640 1 -- 172.21.15.90:0/72273572 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3310007310 con 0x7f3318005e90 2024-04-25T01:16:56.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.782+0000 7f331fe00640 1 -- 172.21.15.90:0/72273572 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33180ee250 con 0x7f3318005e90 2024-04-25T01:16:56.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.782+0000 7f331fe00640 1 -- 172.21.15.90:0/72273572 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33180ee810 con 0x7f3318005e90 2024-04-25T01:16:56.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.786+0000 7f331fe00640 1 -- 172.21.15.90:0/72273572 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3318038dd0 con 0x7f3318005e90 2024-04-25T01:16:56.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.784+0000 7f9a09a00640 1 Processor -- start 2024-04-25T01:16:56.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.784+0000 7f9a09a00640 1 -- start start 2024-04-25T01:16:56.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.785+0000 7f9a09a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99fc00ab40 0x7f99fc0d98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.785+0000 7f9a09a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f99fc0516e0 0x7f99fc008b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.785+0000 7f9a09a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99fc0e1310 0x7f99fc0090d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.789+0000 7f9a09000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99fc0e1310 0x7f99fc0090d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.789+0000 7f9a09000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99fc0e1310 0x7f99fc0090d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56200/0 (socket says 172.21.15.90:56200) 2024-04-25T01:16:56.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.787+0000 7f3316a00640 1 -- 172.21.15.90:0/72273572 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f331001a070 con 0x7f3318005e90 2024-04-25T01:16:56.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.788+0000 7f3316a00640 1 --2- 172.21.15.90:0/72273572 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f330006be30 0x7f330006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.789+0000 7f3316a00640 1 -- 172.21.15.90:0/72273572 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f331008a530 con 0x7f3318005e90 2024-04-25T01:16:56.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.789+0000 7f331e000640 1 --2- 172.21.15.90:0/72273572 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f330006be30 0x7f330006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.789+0000 7f9a09000640 1 -- 172.21.15.90:0/759775613 learned_addr learned my addr 172.21.15.90:0/759775613 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:56.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.790+0000 7f331e000640 1 --2- 172.21.15.90:0/72273572 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f330006be30 0x7f330006e2f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3318001090 tx=0x7f330c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:56.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.789+0000 7f9a09a00640 1 -- 172.21.15.90:0/759775613 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f99fc0548a0 con 0x7f99fc0e1310 2024-04-25T01:16:56.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.789+0000 7f9a09a00640 1 -- 172.21.15.90:0/759775613 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f99fc0555b0 con 0x7f99fc00ab40 2024-04-25T01:16:56.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.789+0000 7f9a09a00640 1 -- 172.21.15.90:0/759775613 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f99fc054a20 con 0x7f99fc0516e0 2024-04-25T01:16:56.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.790+0000 7f9a09000640 1 -- 172.21.15.90:0/759775613 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f99fc0516e0 msgr2=0x7f99fc008b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:16:56.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.790+0000 7f9a03400640 1 --2- 172.21.15.90:0/759775613 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f99fc0516e0 0x7f99fc008b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.790+0000 7f9a09000640 1 --2- 172.21.15.90:0/759775613 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f99fc0516e0 0x7f99fc008b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:56.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.790+0000 7f9a09000640 1 -- 172.21.15.90:0/759775613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99fc00ab40 msgr2=0x7f99fc0d98b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:16:56.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.790+0000 7f9a03e00640 1 --2- 172.21.15.90:0/759775613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99fc00ab40 0x7f99fc0d98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.790+0000 7f9a09000640 1 --2- 172.21.15.90:0/759775613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99fc00ab40 0x7f99fc0d98b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:56.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.790+0000 7f9a09000640 1 -- 172.21.15.90:0/759775613 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99fc1789e0 con 0x7f99fc0e1310 2024-04-25T01:16:56.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.791+0000 7f9a09000640 1 --2- 172.21.15.90:0/759775613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99fc0e1310 0x7f99fc0090d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9a04062d00 tx=0x7f9a04068c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:56.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.791+0000 7f99f7e00640 1 -- 172.21.15.90:0/759775613 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a04073c20 con 0x7f99fc0e1310 2024-04-25T01:16:56.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.792+0000 7f99f7e00640 1 -- 172.21.15.90:0/759775613 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a04073e20 con 0x7f99fc0e1310 2024-04-25T01:16:56.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.792+0000 7f99f7e00640 1 -- 172.21.15.90:0/759775613 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a04074250 con 0x7f99fc0e1310 2024-04-25T01:16:56.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.792+0000 7f9a09a00640 1 -- 172.21.15.90:0/759775613 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99fc178cc0 con 0x7f99fc0e1310 2024-04-25T01:16:56.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.792+0000 7f9a09a00640 1 -- 172.21.15.90:0/759775613 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99fc179280 con 0x7f99fc0e1310 2024-04-25T01:16:56.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.793+0000 7f9a09a00640 1 -- 172.21.15.90:0/759775613 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99fc006fb0 con 0x7f99fc0e1310 2024-04-25T01:16:56.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.795+0000 7f99f7e00640 1 -- 172.21.15.90:0/759775613 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9a04094810 con 0x7f99fc0e1310 2024-04-25T01:16:56.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.796+0000 7f99f7e00640 1 --2- 172.21.15.90:0/759775613 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f99e406be80 0x7f99e406e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.797+0000 7f99f7e00640 1 -- 172.21.15.90:0/759775613 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f9a0410eef0 con 0x7f99fc0e1310 2024-04-25T01:16:56.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.798+0000 7f9a03e00640 1 --2- 172.21.15.90:0/759775613 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f99e406be80 0x7f99e406e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.799+0000 7f9a03e00640 1 --2- 172.21.15.90:0/759775613 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f99e406be80 0x7f99e406e340 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f99f0037000 tx=0x7f99f0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:56.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.802+0000 7f3316a00640 1 -- 172.21.15.90:0/72273572 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f3310058550 con 0x7f3318005e90 2024-04-25T01:16:56.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.806+0000 7f99f7e00640 1 -- 172.21.15.90:0/759775613 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f99fc006fb0 con 0x7f99fc0e1310 2024-04-25T01:16:56.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.836+0000 7f8409600640 1 Processor -- start 2024-04-25T01:16:56.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.836+0000 7f8409600640 1 -- start start 2024-04-25T01:16:56.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.838+0000 7f8409600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8404161760 0x7f84040d2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.839+0000 7f8409600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84040cebb0 con 0x7f84040b5340 2024-04-25T01:16:56.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.839+0000 7f8409600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84040d46d0 con 0x7f84040d0f60 2024-04-25T01:16:56.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.839+0000 7f8409600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f84040d49b0 con 0x7f8404161760 2024-04-25T01:16:56.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.840+0000 7f8403e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8404161760 0x7f84040d2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.840+0000 7f8408c00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f84040d0f60 0x7f84040cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.90:55290/0 (socket says 172.21.15.90:55290) 2024-04-25T01:16:56.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.840+0000 7f8408c00640 1 -- 172.21.15.90:0/253875644 learned_addr learned my addr 172.21.15.90:0/253875644 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:56.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.840+0000 7f8403e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8404161760 0x7f84040d2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.90:39802/0 (socket says 172.21.15.90:39802) 2024-04-25T01:16:56.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.841+0000 7f8402a00640 1 -- 172.21.15.90:0/253875644 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1758384084 0 0) 0x7f84040d46d0 con 0x7f84040d0f60 2024-04-25T01:16:56.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.841+0000 7f8402a00640 1 -- 172.21.15.90:0/253875644 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83e8003520 con 0x7f84040d0f60 2024-04-25T01:16:56.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.841+0000 7f8402a00640 1 -- 172.21.15.90:0/253875644 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2487339516 0 0) 0x7f84040cebb0 con 0x7f84040b5340 2024-04-25T01:16:56.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.842+0000 7f8402a00640 1 -- 172.21.15.90:0/253875644 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f84040d46d0 con 0x7f84040b5340 2024-04-25T01:16:56.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.842+0000 7f8403e00640 1 -- 172.21.15.90:0/253875644 >> v1:172.21.15.103:6789/0 conn(0x7f84040d0f60 legacy=0x7f84040cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:56.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.842+0000 7f8403e00640 1 -- 172.21.15.90:0/253875644 >> v1:172.21.15.90:6789/0 conn(0x7f84040b5340 legacy=0x7f84040d0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:56.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.842+0000 7f8403e00640 1 -- 172.21.15.90:0/253875644 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84040cee50 con 0x7f8404161760 2024-04-25T01:16:56.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.843+0000 7feb85400640 1 Processor -- start 2024-04-25T01:16:56.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.844+0000 7feb85400640 1 -- start start 2024-04-25T01:16:56.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.844+0000 7f8403e00640 1 --2- 172.21.15.90:0/253875644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8404161760 0x7f84040d2b10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f83fc009d90 tx=0x7f83fc02a610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f49f12af1c8ec0a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:56.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.844+0000 7f8402a00640 1 -- 172.21.15.90:0/253875644 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83fc031600 con 0x7f8404161760 2024-04-25T01:16:56.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.844+0000 7f8402a00640 1 -- 172.21.15.90:0/253875644 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83fc0317a0 con 0x7f8404161760 2024-04-25T01:16:56.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.844+0000 7f8402a00640 1 -- 172.21.15.90:0/253875644 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83fc031b70 con 0x7f8404161760 2024-04-25T01:16:56.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.845+0000 7ff572200640 1 Processor -- start 2024-04-25T01:16:56.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.845+0000 7f1f83e00640 1 Processor -- start 2024-04-25T01:16:56.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.846+0000 7f8409600640 1 -- 172.21.15.90:0/253875644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8404161760 msgr2=0x7f84040d2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:56.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.846+0000 7f8409600640 1 --2- 172.21.15.90:0/253875644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8404161760 0x7f84040d2b10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f83fc009d90 tx=0x7f83fc02a610 comp rx=0 tx=0).stop 2024-04-25T01:16:56.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.846+0000 7f40e1000640 1 Processor -- start 2024-04-25T01:16:56.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.846+0000 7f8409600640 1 -- 172.21.15.90:0/253875644 shutdown_connections 2024-04-25T01:16:56.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.846+0000 7f8409600640 1 --2- 172.21.15.90:0/253875644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8404161760 0x7f84040d2b10 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f83fc009d90 tx=0x7f83fc02a610 comp rx=0 tx=0).stop 2024-04-25T01:16:56.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.846+0000 7f8409600640 1 -- 172.21.15.90:0/253875644 >> 172.21.15.90:0/253875644 conn(0x7f840409d170 msgr2=0x7f84040d3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:56.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.846+0000 7f1f83e00640 1 -- start start 2024-04-25T01:16:56.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.846+0000 7f8409600640 1 -- 172.21.15.90:0/253875644 shutdown_connections 2024-04-25T01:16:56.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.846+0000 7feb85400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7feb780e3160 0x7feb780e7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.846+0000 7f40e1000640 1 -- start start 2024-04-25T01:16:56.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.847+0000 7feb85400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb78058810 con 0x7feb7803c0a0 2024-04-25T01:16:56.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.847+0000 7feb85400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb78058af0 con 0x7feb78053cb0 2024-04-25T01:16:56.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.847+0000 7feb85400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7feb78007b70 con 0x7feb780e3160 2024-04-25T01:16:56.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.847+0000 7feb7fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7feb780e3160 0x7feb780e7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.845+0000 7ff572200640 1 -- start start 2024-04-25T01:16:56.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.847+0000 7ff572200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff56c0d28a0 0x7ff56c06eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.847+0000 7ff572200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff56c06f3e0 0x7ff56c06f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.847+0000 7ff572200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff56c0af340 con 0x7ff56c15f630 2024-04-25T01:16:56.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.847+0000 7ff572200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff56c0af100 con 0x7ff56c0d28a0 2024-04-25T01:16:56.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.847+0000 7ff572200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff56c0af620 con 0x7ff56c06f3e0 2024-04-25T01:16:56.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.847+0000 7feb7fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7feb780e3160 0x7feb780e7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56214/0 (socket says 172.21.15.90:56214) 2024-04-25T01:16:56.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.847+0000 7feb7fe00640 1 -- 172.21.15.90:0/356223702 learned_addr learned my addr 172.21.15.90:0/356223702 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:56.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.847+0000 7feb7fe00640 1 -- 172.21.15.90:0/356223702 >> v1:172.21.15.103:6789/0 conn(0x7feb78053cb0 legacy=0x7feb78054100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:16:56.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.847+0000 7feb7fe00640 1 -- 172.21.15.90:0/356223702 >> v1:172.21.15.90:6789/0 conn(0x7feb7803c0a0 legacy=0x7feb78058050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:16:56.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.846+0000 7f8409600640 1 -- 172.21.15.90:0/253875644 wait complete. 2024-04-25T01:16:56.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.848+0000 7f40e1000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f40dc0b2b20 0x7f40dc0b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.848+0000 7f40e1000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40dc06cf60 con 0x7f40dc0b4af0 2024-04-25T01:16:56.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.848+0000 7f40e1000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40dc06d240 con 0x7f40dc0cefe0 2024-04-25T01:16:56.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.848+0000 7f40e1000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f40dc06d520 con 0x7f40dc0b2b20 2024-04-25T01:16:56.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.848+0000 7ff56b400640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7ff56c15f630 0x7ff56c0ae9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.90:55294/0 (socket says 172.21.15.90:55294) 2024-04-25T01:16:56.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.848+0000 7ff56b400640 1 -- 172.21.15.90:0/1966796373 learned_addr learned my addr 172.21.15.90:0/1966796373 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:56.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.848+0000 7ff56be00640 1 --2- 172.21.15.90:0/1966796373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff56c06f3e0 0x7ff56c06f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.849+0000 7ff56aa00640 1 --2- 172.21.15.90:0/1966796373 >> v2:172.21.15.103:3300/0 conn(0x7ff56c0d28a0 0x7ff56c06eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.848+0000 7f1f83e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1f840d02c0 0x7f1f8416d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.848+0000 7f1f83e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f84166630 con 0x7f1f840b49b0 2024-04-25T01:16:56.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.848+0000 7f1f83e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f84166910 con 0x7f1f840b2fb0 2024-04-25T01:16:56.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.848+0000 7f1f83e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1f84162c20 con 0x7f1f840d02c0 2024-04-25T01:16:56.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.849+0000 7f40d9400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f40dc0b4af0 0x7f40dc163440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:35654/0 (socket says 172.21.15.90:35654) 2024-04-25T01:16:56.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.849+0000 7f1f7be00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f1f840b49b0 0x7f1f840cfbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:35640/0 (socket says 172.21.15.90:35640) 2024-04-25T01:16:56.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.849+0000 7f1f7be00640 1 -- 172.21.15.90:0/2750796909 learned_addr learned my addr 172.21.15.90:0/2750796909 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:56.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.849+0000 7f40d9400640 1 -- 172.21.15.90:0/846006495 learned_addr learned my addr 172.21.15.90:0/846006495 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:56.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.849+0000 7f8409600640 1 Processor -- start 2024-04-25T01:16:56.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.849+0000 7f1f83400640 1 --2- 172.21.15.90:0/2750796909 >> v2:172.21.15.103:3300/0 conn(0x7f1f840d02c0 0x7f1f8416d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.850+0000 7f40da800640 1 --2- 172.21.15.90:0/846006495 >> v2:172.21.15.90:3300/0 conn(0x7f40dc0b2b20 0x7f40dc0b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.849+0000 7ffb04600640 1 Processor -- start 2024-04-25T01:16:56.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.849+0000 7ffb04600640 1 -- start start 2024-04-25T01:16:56.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.850+0000 7feb7fe00640 1 -- 172.21.15.90:0/356223702 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb78006360 con 0x7feb780e3160 2024-04-25T01:16:56.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.851+0000 7f40d8a00640 1 -- 172.21.15.90:0/846006495 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3626169766 0 0) 0x7f40dc06cf60 con 0x7f40dc0b4af0 2024-04-25T01:16:56.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.851+0000 7f40d8a00640 1 -- 172.21.15.90:0/846006495 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40c0003520 con 0x7f40dc0b4af0 2024-04-25T01:16:56.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.850+0000 7f1f83400640 1 -- 172.21.15.90:0/2750796909 >> v1:172.21.15.103:6789/0 conn(0x7f1f840b2fb0 legacy=0x7f1f84165e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:56.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.851+0000 7ffb04600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ffafc0c5040 0x7ffafc0afb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.851+0000 7ffb04600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffafc1634f0 con 0x7ffafc0c4b90 2024-04-25T01:16:56.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.850+0000 7f1f83400640 1 -- 172.21.15.90:0/2750796909 >> v1:172.21.15.90:6789/0 conn(0x7f1f840b49b0 legacy=0x7f1f840cfbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:56.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.850+0000 7f1f83400640 1 -- 172.21.15.90:0/2750796909 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f84166bf0 con 0x7f1f840d02c0 2024-04-25T01:16:56.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.851+0000 7ffb04600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffafc1637d0 con 0x7ffafc15ca50 2024-04-25T01:16:56.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.851+0000 7ffb04600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7ffafc0d29e0 con 0x7ffafc0c5040 2024-04-25T01:16:56.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.852+0000 7ffb02600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ffafc0c5040 0x7ffafc0afb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.852+0000 7f1f83400640 1 --2- 172.21.15.90:0/2750796909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f840d02c0 0x7f1f8416d200 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c009d90 tx=0x7f1f7c02a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1604f9ab69334650 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:56.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.852+0000 7ffb02600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ffafc0c5040 0x7ffafc0afb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56234/0 (socket says 172.21.15.90:56234) 2024-04-25T01:16:56.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.852+0000 7ffb02600640 1 -- 172.21.15.90:0/2349107579 learned_addr learned my addr 172.21.15.90:0/2349107579 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:56.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.852+0000 7feb7fe00640 1 --2- 172.21.15.90:0/356223702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb780e3160 0x7feb780e7780 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7feb8007aeb0 tx=0x7feb800a5e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69b79bb783c39e3a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:56.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.852+0000 7ffb02600640 1 -- 172.21.15.90:0/2349107579 >> v1:172.21.15.103:6789/0 conn(0x7ffafc15ca50 legacy=0x7ffafc162980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:56.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.852+0000 7ffb02600640 1 -- 172.21.15.90:0/2349107579 >> v1:172.21.15.90:6789/0 conn(0x7ffafc0c4b90 legacy=0x7ffafc0af450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:56.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.853+0000 7feb7ea00640 1 -- 172.21.15.90:0/356223702 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb800b3070 con 0x7feb780e3160 2024-04-25T01:16:56.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.851+0000 7f8409600640 1 -- start start 2024-04-25T01:16:56.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.851+0000 7f40da800640 1 -- 172.21.15.90:0/846006495 >> v1:172.21.15.103:6789/0 conn(0x7f40dc0cefe0 legacy=0x7f40dc06e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:56.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.851+0000 7f40da800640 1 -- 172.21.15.90:0/846006495 >> v1:172.21.15.90:6789/0 conn(0x7f40dc0b4af0 legacy=0x7f40dc163440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:56.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.852+0000 7f40da800640 1 -- 172.21.15.90:0/846006495 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40dc06d800 con 0x7f40dc0b2b20 2024-04-25T01:16:56.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.853+0000 7f8409600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84040b5340 0x7f84040cb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.853+0000 7f8409600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84040d0f60 0x7f840415b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.853+0000 7f8409600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f840415f7f0 0x7f84040c8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.853+0000 7f8409600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8404161b40 con 0x7f84040d0f60 2024-04-25T01:16:56.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.853+0000 7f8409600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f84040b3740 con 0x7f84040b5340 2024-04-25T01:16:56.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.853+0000 7f8409600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f84040c7970 con 0x7f840415f7f0 2024-04-25T01:16:56.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.853+0000 7f1f82000640 1 -- 172.21.15.90:0/2750796909 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c007740 con 0x7f1f840d02c0 2024-04-25T01:16:56.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.853+0000 7f1f82000640 1 -- 172.21.15.90:0/2750796909 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f7c0078e0 con 0x7f1f840d02c0 2024-04-25T01:16:56.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.853+0000 7f1f82000640 1 -- 172.21.15.90:0/2750796909 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c007cf0 con 0x7f1f840d02c0 2024-04-25T01:16:56.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.849+0000 7ff56be00640 1 -- 172.21.15.90:0/1966796373 >> v2:172.21.15.103:3300/0 conn(0x7ff56c0d28a0 msgr2=0x7ff56c06eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:56.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.849+0000 7ff56be00640 1 --2- 172.21.15.90:0/1966796373 >> v2:172.21.15.103:3300/0 conn(0x7ff56c0d28a0 0x7ff56c06eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:56.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.849+0000 7ff56be00640 1 -- 172.21.15.90:0/1966796373 >> v1:172.21.15.103:6789/0 conn(0x7ff56c15f630 legacy=0x7ff56c0ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:56.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.849+0000 7ff56be00640 1 -- 172.21.15.90:0/1966796373 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff56c0af900 con 0x7ff56c06f3e0 2024-04-25T01:16:56.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.853+0000 7f40da800640 1 --2- 172.21.15.90:0/846006495 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40dc0b2b20 0x7f40dc0b2fd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f40d4009d90 tx=0x7f40d402a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b62ead2e8c1fa301 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:56.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.853+0000 7feb7ea00640 1 -- 172.21.15.90:0/356223702 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb800a6c00 con 0x7feb780e3160 2024-04-25T01:16:56.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.853+0000 7feb7ea00640 1 -- 172.21.15.90:0/356223702 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb800ae440 con 0x7feb780e3160 2024-04-25T01:16:56.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.852+0000 7ffb02600640 1 -- 172.21.15.90:0/2349107579 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffafc0ae970 con 0x7ffafc0c5040 2024-04-25T01:16:56.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.854+0000 7f8403e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84040b5340 0x7f84040cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.854+0000 7f40d8a00640 1 -- 172.21.15.90:0/846006495 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d4035020 con 0x7f40dc0b2b20 2024-04-25T01:16:56.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.854+0000 7f40d8a00640 1 -- 172.21.15.90:0/846006495 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40d4007540 con 0x7f40dc0b2b20 2024-04-25T01:16:56.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.854+0000 7f40d8a00640 1 -- 172.21.15.90:0/846006495 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c8001670 con 0x7f40dc0b2b20 2024-04-25T01:16:56.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.854+0000 7ff56be00640 1 --2- 172.21.15.90:0/1966796373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff56c06f3e0 0x7ff56c06f850 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff55c0097a0 tx=0x7ff55c02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=974886ddd800526 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:56.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.854+0000 7f8403e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84040b5340 0x7f84040cb1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.90:39832/0 (socket says 172.21.15.90:39832) 2024-04-25T01:16:56.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.854+0000 7ffb02600640 1 --2- 172.21.15.90:0/2349107579 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffafc0c5040 0x7ffafc0afb60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ffaf0009870 tx=0x7ffaf002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0aa289203f52232 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:56.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.854+0000 7f8403400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84040d0f60 0x7f840415b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.854+0000 7feb85400640 1 -- 172.21.15.90:0/356223702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb780e3160 msgr2=0x7feb780e7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:56.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.854+0000 7feb85400640 1 --2- 172.21.15.90:0/356223702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb780e3160 0x7feb780e7780 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7feb8007aeb0 tx=0x7feb800a5e30 comp rx=0 tx=0).stop 2024-04-25T01:16:56.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.854+0000 7f8403400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84040d0f60 0x7f840415b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56242/0 (socket says 172.21.15.90:56242) 2024-04-25T01:16:56.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.854+0000 7f8403400640 1 -- 172.21.15.90:0/485413632 learned_addr learned my addr 172.21.15.90:0/485413632 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:56.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.855+0000 7f1f83e00640 1 -- 172.21.15.90:0/2750796909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f840d02c0 msgr2=0x7f1f8416d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:56.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.855+0000 7f1f83e00640 1 --2- 172.21.15.90:0/2750796909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f840d02c0 0x7f1f8416d200 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c009d90 tx=0x7f1f7c02a710 comp rx=0 tx=0).stop 2024-04-25T01:16:56.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.855+0000 7f8408c00640 1 --2- 172.21.15.90:0/485413632 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f840415f7f0 0x7f84040c8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.855+0000 7ffafbe00640 1 -- 172.21.15.90:0/2349107579 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf0037070 con 0x7ffafc0c5040 2024-04-25T01:16:56.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.855+0000 7feb85400640 1 -- 172.21.15.90:0/356223702 shutdown_connections 2024-04-25T01:16:56.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.855+0000 7feb85400640 1 --2- 172.21.15.90:0/356223702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb780e3160 0x7feb780e7780 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:56.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.855+0000 7feb85400640 1 -- 172.21.15.90:0/356223702 >> 172.21.15.90:0/356223702 conn(0x7feb78058ee0 msgr2=0x7feb780592c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:56.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.855+0000 7ffafbe00640 1 -- 172.21.15.90:0/2349107579 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaf0032440 con 0x7ffafc0c5040 2024-04-25T01:16:56.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.855+0000 7ffafbe00640 1 -- 172.21.15.90:0/2349107579 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf0032810 con 0x7ffafc0c5040 2024-04-25T01:16:56.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.855+0000 7feb85400640 1 -- 172.21.15.90:0/356223702 shutdown_connections 2024-04-25T01:16:56.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.855+0000 7f40e1000640 1 -- 172.21.15.90:0/846006495 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40dc0b2b20 msgr2=0x7f40dc0b2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:56.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.855+0000 7f40e1000640 1 --2- 172.21.15.90:0/846006495 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40dc0b2b20 0x7f40dc0b2fd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f40d4009d90 tx=0x7f40d402a5d0 comp rx=0 tx=0).stop 2024-04-25T01:16:56.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.855+0000 7ff56a000640 1 -- 172.21.15.90:0/1966796373 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff55c037070 con 0x7ff56c06f3e0 2024-04-25T01:16:56.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.855+0000 7f8403400640 1 -- 172.21.15.90:0/485413632 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f840415f7f0 msgr2=0x7f84040c8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:56.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.855+0000 7ff56a000640 1 -- 172.21.15.90:0/1966796373 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff55c032440 con 0x7ff56c06f3e0 2024-04-25T01:16:56.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.855+0000 7ff56a000640 1 -- 172.21.15.90:0/1966796373 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff55c032810 con 0x7ff56c06f3e0 2024-04-25T01:16:56.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.855+0000 7f8403400640 1 --2- 172.21.15.90:0/485413632 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f840415f7f0 0x7f84040c8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:56.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.855+0000 7f8403400640 1 -- 172.21.15.90:0/485413632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84040b5340 msgr2=0x7f84040cb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:56.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.855+0000 7f8403400640 1 --2- 172.21.15.90:0/485413632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84040b5340 0x7f84040cb1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:56.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.856+0000 7f40e1000640 1 -- 172.21.15.90:0/846006495 shutdown_connections 2024-04-25T01:16:56.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.856+0000 7f40e1000640 1 --2- 172.21.15.90:0/846006495 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40dc0b2b20 0x7f40dc0b2fd0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:56.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.856+0000 7f8403400640 1 -- 172.21.15.90:0/485413632 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84041606b0 con 0x7f84040d0f60 2024-04-25T01:16:56.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.856+0000 7f40e1000640 1 -- 172.21.15.90:0/846006495 >> 172.21.15.90:0/846006495 conn(0x7f40dc0b1820 msgr2=0x7f40dc0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:56.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.856+0000 7ffb04600640 1 -- 172.21.15.90:0/2349107579 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffafc0c5040 msgr2=0x7ffafc0afb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:56.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.856+0000 7ffb04600640 1 --2- 172.21.15.90:0/2349107579 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffafc0c5040 0x7ffafc0afb60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ffaf0009870 tx=0x7ffaf002a270 comp rx=0 tx=0).stop 2024-04-25T01:16:56.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.856+0000 7f8403400640 1 --2- 172.21.15.90:0/485413632 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84040d0f60 0x7f840415b770 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f83f400c4e0 tx=0x7f83f400c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:56.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.856+0000 7ffb04600640 1 -- 172.21.15.90:0/2349107579 shutdown_connections 2024-04-25T01:16:56.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.856+0000 7ffb04600640 1 --2- 172.21.15.90:0/2349107579 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffafc0c5040 0x7ffafc0afb60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:56.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.856+0000 7ffb04600640 1 -- 172.21.15.90:0/2349107579 >> 172.21.15.90:0/2349107579 conn(0x7ffafc0d1bc0 msgr2=0x7ffafc0d1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:56.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.855+0000 7feb85400640 1 -- 172.21.15.90:0/356223702 wait complete. 2024-04-25T01:16:56.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.857+0000 7f40e1000640 1 -- 172.21.15.90:0/846006495 shutdown_connections 2024-04-25T01:16:56.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.856+0000 7ff572200640 1 -- 172.21.15.90:0/1966796373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff56c06f3e0 msgr2=0x7ff56c06f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:56.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.856+0000 7ff572200640 1 --2- 172.21.15.90:0/1966796373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff56c06f3e0 0x7ff56c06f850 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff55c0097a0 tx=0x7ff55c02a180 comp rx=0 tx=0).stop 2024-04-25T01:16:56.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.857+0000 7f40e1000640 1 -- 172.21.15.90:0/846006495 wait complete. 2024-04-25T01:16:56.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.857+0000 7ff572200640 1 -- 172.21.15.90:0/1966796373 shutdown_connections 2024-04-25T01:16:56.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.857+0000 7ff572200640 1 --2- 172.21.15.90:0/1966796373 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff56c06f3e0 0x7ff56c06f850 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:56.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.857+0000 7ff572200640 1 --2- 172.21.15.90:0/1966796373 >> v2:172.21.15.103:3300/0 conn(0x7ff56c0d28a0 0x7ff56c06eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:56.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.857+0000 7ff572200640 1 -- 172.21.15.90:0/1966796373 >> 172.21.15.90:0/1966796373 conn(0x7ff56c0d1ca0 msgr2=0x7ff56c0d20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:56.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.857+0000 7f83fbe00640 1 -- 172.21.15.90:0/485413632 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f40052d0 con 0x7f84040d0f60 2024-04-25T01:16:56.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.857+0000 7f83fbe00640 1 -- 172.21.15.90:0/485413632 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83f4012630 con 0x7f84040d0f60 2024-04-25T01:16:56.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.857+0000 7f83fbe00640 1 -- 172.21.15.90:0/485413632 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f40054b0 con 0x7f84040d0f60 2024-04-25T01:16:56.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.857+0000 7f8409600640 1 -- 172.21.15.90:0/485413632 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84040c7bc0 con 0x7f84040d0f60 2024-04-25T01:16:56.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.857+0000 7ff572200640 1 -- 172.21.15.90:0/1966796373 shutdown_connections 2024-04-25T01:16:56.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.858+0000 7feb85400640 1 Processor -- start 2024-04-25T01:16:56.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.858+0000 7f1f83e00640 1 -- 172.21.15.90:0/2750796909 shutdown_connections 2024-04-25T01:16:56.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.858+0000 7f1f83e00640 1 --2- 172.21.15.90:0/2750796909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f840d02c0 0x7f1f8416d200 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:56.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.858+0000 7f1f83e00640 1 -- 172.21.15.90:0/2750796909 >> 172.21.15.90:0/2750796909 conn(0x7f1f8406b300 msgr2=0x7f1f8406b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:56.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.857+0000 7ff572200640 1 -- 172.21.15.90:0/1966796373 wait complete. 2024-04-25T01:16:56.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.858+0000 7feb85400640 1 -- start start 2024-04-25T01:16:56.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.858+0000 7f8409600640 1 -- 172.21.15.90:0/485413632 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84040c8180 con 0x7f84040d0f60 2024-04-25T01:16:56.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.858+0000 7f1f83e00640 1 -- 172.21.15.90:0/2750796909 shutdown_connections 2024-04-25T01:16:56.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.858+0000 7f1f83e00640 1 -- 172.21.15.90:0/2750796909 wait complete. 2024-04-25T01:16:56.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.859+0000 7ffb04600640 1 -- 172.21.15.90:0/2349107579 shutdown_connections 2024-04-25T01:16:56.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.859+0000 7ff572200640 1 Processor -- start 2024-04-25T01:16:56.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.859+0000 7ffb04600640 1 -- 172.21.15.90:0/2349107579 wait complete. 2024-04-25T01:16:56.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.860+0000 7f8409600640 1 -- 172.21.15.90:0/485413632 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8404075350 con 0x7f84040d0f60 2024-04-25T01:16:56.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.861+0000 7f1f83e00640 1 Processor -- start 2024-04-25T01:16:56.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.861+0000 7f1f83e00640 1 -- start start 2024-04-25T01:16:56.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.861+0000 7ffb04600640 1 Processor -- start 2024-04-25T01:16:56.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.861+0000 7ffb04600640 1 -- start start 2024-04-25T01:16:56.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.860+0000 7f40e1000640 1 Processor -- start 2024-04-25T01:16:56.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.860+0000 7f40e1000640 1 -- start start 2024-04-25T01:16:56.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.862+0000 7f40e1000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40dc0b2b20 0x7f40dc200b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.862+0000 7feb85400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb7803c0a0 0x7feb78181110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.862+0000 7f40e1000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40dc0b4af0 0x7f40dc0b0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.862+0000 7feb85400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb78053cb0 0x7feb7817b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.862+0000 7f40e1000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40dc0cefe0 0x7f40dc0cd890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.862+0000 7f40e1000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f40dc070dd0 con 0x7f40dc0b2b20 2024-04-25T01:16:56.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.862+0000 7f40e1000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f40dc070c50 con 0x7f40dc0b4af0 2024-04-25T01:16:56.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.862+0000 7f40e1000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f40dc070f50 con 0x7f40dc0cefe0 2024-04-25T01:16:56.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.862+0000 7feb85400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb780e3160 0x7feb7817d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.862+0000 7feb85400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7feb780ee050 con 0x7feb780e3160 2024-04-25T01:16:56.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.862+0000 7feb85400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7feb780e3540 con 0x7feb78053cb0 2024-04-25T01:16:56.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.862+0000 7feb85400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7feb780ee1d0 con 0x7feb7803c0a0 2024-04-25T01:16:56.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.862+0000 7f40d9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40dc0b2b20 0x7f40dc200b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.862+0000 7f40d9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40dc0b2b20 0x7f40dc200b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56266/0 (socket says 172.21.15.90:56266) 2024-04-25T01:16:56.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.862+0000 7f40d9e00640 1 -- 172.21.15.90:0/3554281415 learned_addr learned my addr 172.21.15.90:0/3554281415 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:56.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.862+0000 7feb7fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb7803c0a0 0x7feb78181110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.863+0000 7feb7fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb7803c0a0 0x7feb78181110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.90:48188/0 (socket says 172.21.15.90:48188) 2024-04-25T01:16:56.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.863+0000 7feb7fe00640 1 -- 172.21.15.90:0/3487013644 learned_addr learned my addr 172.21.15.90:0/3487013644 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:56.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.863+0000 7feb7f400640 1 --2- 172.21.15.90:0/3487013644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb78053cb0 0x7feb7817b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.862+0000 7f40d9400640 1 --2- 172.21.15.90:0/3554281415 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40dc0b4af0 0x7f40dc0b0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.863+0000 7f40da800640 1 --2- 172.21.15.90:0/3554281415 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40dc0cefe0 0x7f40dc0cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.863+0000 7feb84a00640 1 --2- 172.21.15.90:0/3487013644 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb780e3160 0x7feb7817d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.863+0000 7f40d9e00640 1 -- 172.21.15.90:0/3554281415 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40dc0cefe0 msgr2=0x7f40dc0cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:56.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.863+0000 7f40d9e00640 1 --2- 172.21.15.90:0/3554281415 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40dc0cefe0 0x7f40dc0cd890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:56.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.863+0000 7f40d9e00640 1 -- 172.21.15.90:0/3554281415 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40dc0b4af0 msgr2=0x7f40dc0b0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:56.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.863+0000 7f40d9e00640 1 --2- 172.21.15.90:0/3554281415 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40dc0b4af0 0x7f40dc0b0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:56.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.863+0000 7f40d9e00640 1 -- 172.21.15.90:0/3554281415 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40dc201060 con 0x7f40dc0b2b20 2024-04-25T01:16:56.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.863+0000 7ffb04600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffafc0c4b90 0x7ffafc0c56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.863+0000 7feb7fe00640 1 -- 172.21.15.90:0/3487013644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb78053cb0 msgr2=0x7feb7817b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:56.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.863+0000 7feb7fe00640 1 --2- 172.21.15.90:0/3487013644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb78053cb0 0x7feb7817b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:56.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.863+0000 7feb7fe00640 1 -- 172.21.15.90:0/3487013644 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb780e3160 msgr2=0x7feb7817d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:56.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.863+0000 7ffb04600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffafc0c5040 0x7ffafc0c84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.864+0000 7f40d9e00640 1 --2- 172.21.15.90:0/3554281415 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40dc0b2b20 0x7f40dc200b20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f40cc00c400 tx=0x7f40cc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:56.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.863+0000 7ffb04600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffafc15ca50 0x7ffafc1f7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.864+0000 7f1f83e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f840b2fb0 0x7f1f841531a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.864+0000 7feb7fe00640 1 --2- 172.21.15.90:0/3487013644 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb780e3160 0x7feb7817d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:56.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.864+0000 7feb7fe00640 1 -- 172.21.15.90:0/3487013644 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb78181fd0 con 0x7feb7803c0a0 2024-04-25T01:16:56.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.864+0000 7ffb04600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ffafc0aef30 con 0x7ffafc15ca50 2024-04-25T01:16:56.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.864+0000 7f1f83e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f840b49b0 0x7f1f841ebbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.864+0000 7ffb04600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ffafc15ce30 con 0x7ffafc0c4b90 2024-04-25T01:16:56.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.864+0000 7ffb04600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ffafc0c5450 con 0x7ffafc0c5040 2024-04-25T01:16:56.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.864+0000 7ffb02600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffafc15ca50 0x7ffafc1f7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.864+0000 7feb84a00640 1 --2- 172.21.15.90:0/3487013644 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb780e3160 0x7feb7817d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:56.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.864+0000 7ffb02600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffafc15ca50 0x7ffafc1f7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56280/0 (socket says 172.21.15.90:56280) 2024-04-25T01:16:56.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.864+0000 7ffb02600640 1 -- 172.21.15.90:0/3939072357 learned_addr learned my addr 172.21.15.90:0/3939072357 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:56.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.864+0000 7ffb01200640 1 --2- 172.21.15.90:0/3939072357 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffafc0c5040 0x7ffafc0c84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.864+0000 7f1f83e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f840d02c0 0x7f1f841560e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.864+0000 7f1f83e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1f841640c0 con 0x7f1f840d02c0 2024-04-25T01:16:56.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.864+0000 7f1f83e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1f840b4df0 con 0x7f1f840b49b0 2024-04-25T01:16:56.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.864+0000 7f1f83e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1f84164240 con 0x7f1f840b2fb0 2024-04-25T01:16:56.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.864+0000 7ffb01c00640 1 --2- 172.21.15.90:0/3939072357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffafc0c4b90 0x7ffafc0c56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.865+0000 7feb7fe00640 1 --2- 172.21.15.90:0/3487013644 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb7803c0a0 0x7feb78181110 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7feb80068830 tx=0x7feb80075020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:56.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.865+0000 7f1f82a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f840b2fb0 0x7f1f841531a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.865+0000 7f1f7be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f840b49b0 0x7f1f841ebbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.865+0000 7f1f82a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f840b2fb0 0x7f1f841531a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.90:48202/0 (socket says 172.21.15.90:48202) 2024-04-25T01:16:56.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.865+0000 7f1f82a00640 1 -- 172.21.15.90:0/449414845 learned_addr learned my addr 172.21.15.90:0/449414845 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:56.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.865+0000 7f1f83400640 1 --2- 172.21.15.90:0/449414845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f840d02c0 0x7f1f841560e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.865+0000 7feb73e00640 1 -- 172.21.15.90:0/3487013644 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb800b3040 con 0x7feb7803c0a0 2024-04-25T01:16:56.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.865+0000 7feb73e00640 1 -- 172.21.15.90:0/3487013644 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb800c0070 con 0x7feb7803c0a0 2024-04-25T01:16:56.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.865+0000 7feb73e00640 1 -- 172.21.15.90:0/3487013644 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb800b9870 con 0x7feb7803c0a0 2024-04-25T01:16:56.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.865+0000 7ffb02600640 1 -- 172.21.15.90:0/3939072357 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffafc0c5040 msgr2=0x7ffafc0c84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:56.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.865+0000 7ffb02600640 1 --2- 172.21.15.90:0/3939072357 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffafc0c5040 0x7ffafc0c84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:56.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.865+0000 7ffb02600640 1 -- 172.21.15.90:0/3939072357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffafc0c4b90 msgr2=0x7ffafc0c56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:56.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.865+0000 7ffb02600640 1 --2- 172.21.15.90:0/3939072357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffafc0c4b90 0x7ffafc0c56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:56.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.865+0000 7ffb02600640 1 -- 172.21.15.90:0/3939072357 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffafc0c5d10 con 0x7ffafc15ca50 2024-04-25T01:16:56.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.866+0000 7f1f83400640 1 -- 172.21.15.90:0/449414845 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f840b2fb0 msgr2=0x7f1f841531a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:56.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.866+0000 7f1f83400640 1 --2- 172.21.15.90:0/449414845 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f840b2fb0 0x7f1f841531a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:56.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.866+0000 7f1f83400640 1 -- 172.21.15.90:0/449414845 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f840b49b0 msgr2=0x7f1f841ebbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:56.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.866+0000 7f1f83400640 1 --2- 172.21.15.90:0/449414845 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f840b49b0 0x7f1f841ebbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:56.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.866+0000 7f1f83400640 1 -- 172.21.15.90:0/449414845 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f841ec550 con 0x7f1f840d02c0 2024-04-25T01:16:56.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.866+0000 7feb85400640 1 -- 172.21.15.90:0/3487013644 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb78181a10 con 0x7feb7803c0a0 2024-04-25T01:16:56.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.866+0000 7feb85400640 1 -- 172.21.15.90:0/3487013644 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7feb780e06e0 con 0x7feb7803c0a0 2024-04-25T01:16:56.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.866+0000 7ff572200640 1 -- start start 2024-04-25T01:16:56.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.866+0000 7f83fbe00640 1 -- 172.21.15.90:0/485413632 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f83f4012a20 con 0x7f84040d0f60 2024-04-25T01:16:56.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.866+0000 7ffb02600640 1 --2- 172.21.15.90:0/3939072357 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffafc15ca50 0x7ffafc1f7c10 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ffaf002a2b0 tx=0x7ffaf00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:56.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.867+0000 7ff572200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff56c06f3e0 0x7ff56c0b69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.867+0000 7ff572200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff56c0d28a0 0x7ff56c15ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.867+0000 7ff572200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff56c15f630 0x7ff56c0ce280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.867+0000 7ff572200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff56c0b13f0 con 0x7ff56c06f3e0 2024-04-25T01:16:56.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.867+0000 7ff572200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff56c0cc6d0 con 0x7ff56c0d28a0 2024-04-25T01:16:56.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.867+0000 7ff572200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff56c0b77c0 con 0x7ff56c15f630 2024-04-25T01:16:56.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.867+0000 7f1f83400640 1 --2- 172.21.15.90:0/449414845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f840d02c0 0x7f1f841560e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1f840d0a00 tx=0x7f1f7c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:56.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.867+0000 7ff56b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff56c06f3e0 0x7ff56c0b69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.867+0000 7ff56b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff56c06f3e0 0x7ff56c0b69b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56294/0 (socket says 172.21.15.90:56294) 2024-04-25T01:16:56.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.867+0000 7ff56b400640 1 -- 172.21.15.90:0/3819531420 learned_addr learned my addr 172.21.15.90:0/3819531420 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:56.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.867+0000 7ff56be00640 1 --2- 172.21.15.90:0/3819531420 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff56c15f630 0x7ff56c0ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.868+0000 7ff56aa00640 1 --2- 172.21.15.90:0/3819531420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff56c0d28a0 0x7ff56c15ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.868+0000 7feb85400640 1 -- 172.21.15.90:0/3487013644 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb78005390 con 0x7feb7803c0a0 2024-04-25T01:16:56.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.868+0000 7ff56b400640 1 -- 172.21.15.90:0/3819531420 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff56c15f630 msgr2=0x7ff56c0ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:56.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.868+0000 7ff56b400640 1 --2- 172.21.15.90:0/3819531420 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff56c15f630 0x7ff56c0ce280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:56.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.868+0000 7ff56b400640 1 -- 172.21.15.90:0/3819531420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff56c0d28a0 msgr2=0x7ff56c15ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:56.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.868+0000 7ff56b400640 1 --2- 172.21.15.90:0/3819531420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff56c0d28a0 0x7ff56c15ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:56.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.868+0000 7ff56b400640 1 -- 172.21.15.90:0/3819531420 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff56c0b7200 con 0x7ff56c06f3e0 2024-04-25T01:16:56.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.869+0000 7ff56b400640 1 --2- 172.21.15.90:0/3819531420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff56c06f3e0 0x7ff56c0b69b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff56400cbf0 tx=0x7ff56400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:56.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.868+0000 7f83fbe00640 1 --2- 172.21.15.90:0/485413632 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f83e006bf00 0x7f83e006e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.868+0000 7f83fbe00640 1 -- 172.21.15.90:0/485413632 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f83f408ae70 con 0x7f84040d0f60 2024-04-25T01:16:56.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.869+0000 7f8403e00640 1 --2- 172.21.15.90:0/485413632 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f83e006bf00 0x7f83e006e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.870+0000 7f8403e00640 1 --2- 172.21.15.90:0/485413632 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f83e006bf00 0x7f83e006e3c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f83fc034000 tx=0x7f83fc02f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:56.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.871+0000 7feb73e00640 1 -- 172.21.15.90:0/3487013644 <== mon.2 v2:172.21.15.103:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7feb800b9a10 con 0x7feb7803c0a0 2024-04-25T01:16:56.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.872+0000 7feb73e00640 1 --2- 172.21.15.90:0/3487013644 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7feb6006be30 0x7feb6006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.872+0000 7feb73e00640 1 -- 172.21.15.90:0/3487013644 <== mon.2 v2:172.21.15.103:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7feb801281c0 con 0x7feb7803c0a0 2024-04-25T01:16:56.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.872+0000 7feb7f400640 1 --2- 172.21.15.90:0/3487013644 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7feb6006be30 0x7feb6006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.873+0000 7f40d2000640 1 -- 172.21.15.90:0/3554281415 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40cc017070 con 0x7f40dc0b2b20 2024-04-25T01:16:56.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.873+0000 7f83fbe00640 1 -- 172.21.15.90:0/485413632 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f84041606b0 con 0x7f84040d0f60 2024-04-25T01:16:56.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.873+0000 7f40e1000640 1 -- 172.21.15.90:0/3554281415 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40dc0cc7f0 con 0x7f40dc0b2b20 2024-04-25T01:16:56.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.873+0000 7ffaf9600640 1 -- 172.21.15.90:0/3939072357 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf0041070 con 0x7ffafc15ca50 2024-04-25T01:16:56.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.873+0000 7f40e1000640 1 -- 172.21.15.90:0/3554281415 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40dc0cd090 con 0x7f40dc0b2b20 2024-04-25T01:16:56.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.873+0000 7feb7f400640 1 --2- 172.21.15.90:0/3487013644 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7feb6006be30 0x7feb6006e2f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7feb78052f00 tx=0x7feb74007e30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:56.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.873+0000 7ffb04600640 1 -- 172.21.15.90:0/3939072357 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffafc14dcb0 con 0x7ffafc15ca50 2024-04-25T01:16:56.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.873+0000 7ffb04600640 1 -- 172.21.15.90:0/3939072357 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffafc14e270 con 0x7ffafc15ca50 2024-04-25T01:16:56.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.874+0000 7ff563400640 1 -- 172.21.15.90:0/3819531420 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff564017070 con 0x7ff56c06f3e0 2024-04-25T01:16:56.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.874+0000 7ff572200640 1 -- 172.21.15.90:0/3819531420 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff56c0b74e0 con 0x7ff56c06f3e0 2024-04-25T01:16:56.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.874+0000 7ff572200640 1 -- 172.21.15.90:0/3819531420 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff56c0b6ef0 con 0x7ff56c06f3e0 2024-04-25T01:16:56.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.874+0000 7f1f7aa00640 1 -- 172.21.15.90:0/449414845 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c03f070 con 0x7f1f840d02c0 2024-04-25T01:16:56.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.874+0000 7f1f83e00640 1 -- 172.21.15.90:0/449414845 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f841ec830 con 0x7f1f840d02c0 2024-04-25T01:16:56.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.874+0000 7f1f83e00640 1 -- 172.21.15.90:0/449414845 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f841ecdf0 con 0x7f1f840d02c0 2024-04-25T01:16:56.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.875+0000 7f1f7aa00640 1 -- 172.21.15.90:0/449414845 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f7c007840 con 0x7f1f840d02c0 2024-04-25T01:16:56.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.875+0000 7f1f7aa00640 1 -- 172.21.15.90:0/449414845 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c031dc0 con 0x7f1f840d02c0 2024-04-25T01:16:56.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.875+0000 7f40d2000640 1 -- 172.21.15.90:0/3554281415 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40cc0052d0 con 0x7f40dc0b2b20 2024-04-25T01:16:56.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.875+0000 7ffaf9600640 1 -- 172.21.15.90:0/3939072357 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaf0032440 con 0x7ffafc15ca50 2024-04-25T01:16:56.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.875+0000 7ffaf9600640 1 -- 172.21.15.90:0/3939072357 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf004b060 con 0x7ffafc15ca50 2024-04-25T01:16:56.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.875+0000 7f1f83e00640 1 -- 172.21.15.90:0/449414845 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f84068fb0 con 0x7f1f840d02c0 2024-04-25T01:16:56.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.875+0000 7f40d2000640 1 -- 172.21.15.90:0/3554281415 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40cc021060 con 0x7f40dc0b2b20 2024-04-25T01:16:56.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.875+0000 7ff563400640 1 -- 172.21.15.90:0/3819531420 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff56400aca0 con 0x7ff56c06f3e0 2024-04-25T01:16:56.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.875+0000 7ff563400640 1 -- 172.21.15.90:0/3819531420 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff564021060 con 0x7ff56c06f3e0 2024-04-25T01:16:56.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.876+0000 7ffb04600640 1 -- 172.21.15.90:0/3939072357 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffad8005290 con 0x7ffafc15ca50 2024-04-25T01:16:56.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.877+0000 7f40e1000640 1 -- 172.21.15.90:0/3554281415 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40b4005290 con 0x7f40dc0b2b20 2024-04-25T01:16:56.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.878+0000 7ff572200640 1 -- 172.21.15.90:0/3819531420 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff56c068fb0 con 0x7ff56c06f3e0 2024-04-25T01:16:56.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.879+0000 7f40d2000640 1 -- 172.21.15.90:0/3554281415 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f40cc01a070 con 0x7f40dc0b2b20 2024-04-25T01:16:56.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.880+0000 7f1f7aa00640 1 -- 172.21.15.90:0/449414845 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1f7c044460 con 0x7f1f840d02c0 2024-04-25T01:16:56.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.881+0000 7f40d2000640 1 --2- 172.21.15.90:0/3554281415 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f40bc06be30 0x7f40bc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.881+0000 7f40d2000640 1 -- 172.21.15.90:0/3554281415 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f40cc08aeb0 con 0x7f40dc0b2b20 2024-04-25T01:16:56.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.881+0000 7ffaf9600640 1 -- 172.21.15.90:0/3939072357 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ffaf0044070 con 0x7ffafc15ca50 2024-04-25T01:16:56.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.882+0000 7f1f7aa00640 1 --2- 172.21.15.90:0/449414845 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1f6006be30 0x7f1f6006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.882+0000 7f1f7aa00640 1 -- 172.21.15.90:0/449414845 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f1f7c0aceb0 con 0x7f1f840d02c0 2024-04-25T01:16:56.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.881+0000 7ff563400640 1 -- 172.21.15.90:0/3819531420 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff56401a070 con 0x7ff56c06f3e0 2024-04-25T01:16:56.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.882+0000 7ffaf9600640 1 --2- 172.21.15.90:0/3939072357 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ffadc06be30 0x7ffadc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.882+0000 7ffaf9600640 1 -- 172.21.15.90:0/3939072357 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7ffaf00ac240 con 0x7ffafc15ca50 2024-04-25T01:16:56.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.883+0000 7ff563400640 1 --2- 172.21.15.90:0/3819531420 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff54c06be30 0x7ff54c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:56.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.883+0000 7ff563400640 1 -- 172.21.15.90:0/3819531420 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7ff56408b720 con 0x7ff56c06f3e0 2024-04-25T01:16:56.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.883+0000 7f1f82a00640 1 --2- 172.21.15.90:0/449414845 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1f6006be30 0x7f1f6006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.884+0000 7ffb01c00640 1 --2- 172.21.15.90:0/3939072357 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ffadc06be30 0x7ffadc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.884+0000 7feb73e00640 1 -- 172.21.15.90:0/3487013644 <== mon.2 v2:172.21.15.103:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7feb800f67a0 con 0x7feb7803c0a0 2024-04-25T01:16:56.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.884+0000 7f40d9400640 1 --2- 172.21.15.90:0/3554281415 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f40bc06be30 0x7f40bc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.885+0000 7f40d9400640 1 --2- 172.21.15.90:0/3554281415 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f40bc06be30 0x7f40bc06e2f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f40c80015a0 tx=0x7f40c8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:56.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.885+0000 7f1f82a00640 1 --2- 172.21.15.90:0/449414845 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1f6006be30 0x7f1f6006e2f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1f74007fd0 tx=0x7f1f74009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:56.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.885+0000 7ffb01c00640 1 --2- 172.21.15.90:0/3939072357 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ffadc06be30 0x7ffadc06e2f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ffaf4005660 tx=0x7ffaf4003d90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:56.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.883+0000 7ff56aa00640 1 --2- 172.21.15.90:0/3819531420 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff54c06be30 0x7ff54c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:56.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.886+0000 7ff56aa00640 1 --2- 172.21.15.90:0/3819531420 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff54c06be30 0x7ff54c06e2f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff56c1fb720 tx=0x7ff558008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:56.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.888+0000 7f40d2000640 1 -- 172.21.15.90:0/3554281415 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f40cc058fb0 con 0x7f40dc0b2b20 2024-04-25T01:16:56.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.897+0000 7ffaf9600640 1 -- 172.21.15.90:0/3939072357 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ffaf007a240 con 0x7ffafc15ca50 2024-04-25T01:16:56.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.906+0000 7f1f7aa00640 1 -- 172.21.15.90:0/449414845 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1f7c053080 con 0x7f1f840d02c0 2024-04-25T01:16:56.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:56.915+0000 7ff563400640 1 -- 172.21.15.90:0/3819531420 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff564059720 con 0x7ff56c06f3e0 2024-04-25T01:16:57.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.213+0000 7f331fe00640 1 -- 172.21.15.90:0/72273572 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f33180dd5b0 con 0x7f3318005e90 2024-04-25T01:16:57.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.214+0000 7f3316a00640 1 -- 172.21.15.90:0/72273572 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f331005c850 con 0x7f3318005e90 2024-04-25T01:16:57.201 INFO:teuthology.orchestra.run.smithi090.stdout:60129542154 2024-04-25T01:16:57.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.218+0000 7f331fe00640 1 -- 172.21.15.90:0/72273572 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f330006be30 msgr2=0x7f330006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:57.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.218+0000 7f331fe00640 1 --2- 172.21.15.90:0/72273572 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f330006be30 0x7f330006e2f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3318001090 tx=0x7f330c008040 comp rx=0 tx=0).stop 2024-04-25T01:16:57.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.218+0000 7f331fe00640 1 -- 172.21.15.90:0/72273572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3318005e90 msgr2=0x7f33180d7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:57.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.218+0000 7f331fe00640 1 --2- 172.21.15.90:0/72273572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3318005e90 0x7f33180d7d80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f331000c3e0 tx=0x7f331000c8a0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.220+0000 7f331fe00640 1 -- 172.21.15.90:0/72273572 shutdown_connections 2024-04-25T01:16:57.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.220+0000 7f331fe00640 1 --2- 172.21.15.90:0/72273572 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f330006be30 0x7f330006e2f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.220+0000 7f331fe00640 1 --2- 172.21.15.90:0/72273572 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33180ead00 0x7f33180398a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.220+0000 7f331fe00640 1 --2- 172.21.15.90:0/72273572 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3318006810 0x7f3318039360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.220+0000 7f331fe00640 1 --2- 172.21.15.90:0/72273572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3318005e90 0x7f33180d7d80 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.220+0000 7f331fe00640 1 -- 172.21.15.90:0/72273572 >> 172.21.15.90:0/72273572 conn(0x7f3318009fb0 msgr2=0x7f331804dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:57.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.220+0000 7f331fe00640 1 -- 172.21.15.90:0/72273572 shutdown_connections 2024-04-25T01:16:57.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.220+0000 7f331fe00640 1 -- 172.21.15.90:0/72273572 wait complete. 2024-04-25T01:16:57.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.231+0000 7f9a09a00640 1 -- 172.21.15.90:0/759775613 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f99fc0578f0 con 0x7f99fc0e1310 2024-04-25T01:16:57.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.232+0000 7f99f7e00640 1 -- 172.21.15.90:0/759775613 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9a0406f070 con 0x7f99fc0e1310 2024-04-25T01:16:57.217 INFO:teuthology.orchestra.run.smithi090.stdout:60129542155 2024-04-25T01:16:57.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.236+0000 7f9a09a00640 1 -- 172.21.15.90:0/759775613 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f99e406be80 msgr2=0x7f99e406e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:57.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.236+0000 7f9a09a00640 1 --2- 172.21.15.90:0/759775613 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f99e406be80 0x7f99e406e340 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f99f0037000 tx=0x7f99f0008030 comp rx=0 tx=0).stop 2024-04-25T01:16:57.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.236+0000 7f9a09a00640 1 -- 172.21.15.90:0/759775613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99fc0e1310 msgr2=0x7f99fc0090d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:57.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.237+0000 7f9a09a00640 1 --2- 172.21.15.90:0/759775613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99fc0e1310 0x7f99fc0090d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9a04062d00 tx=0x7f9a04068c60 comp rx=0 tx=0).stop 2024-04-25T01:16:57.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.237+0000 7f9a09a00640 1 -- 172.21.15.90:0/759775613 shutdown_connections 2024-04-25T01:16:57.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.237+0000 7f9a09a00640 1 --2- 172.21.15.90:0/759775613 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f99e406be80 0x7f99e406e340 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.237+0000 7f9a09a00640 1 --2- 172.21.15.90:0/759775613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99fc0e1310 0x7f99fc0090d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.238+0000 7f9a09a00640 1 --2- 172.21.15.90:0/759775613 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f99fc0516e0 0x7f99fc008b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.238+0000 7f9a09a00640 1 --2- 172.21.15.90:0/759775613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99fc00ab40 0x7f99fc0d98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.238+0000 7f9a09a00640 1 -- 172.21.15.90:0/759775613 >> 172.21.15.90:0/759775613 conn(0x7f99fc056cd0 msgr2=0x7f99fc03b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:57.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.238+0000 7f9a09a00640 1 -- 172.21.15.90:0/759775613 shutdown_connections 2024-04-25T01:16:57.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.238+0000 7f9a09a00640 1 -- 172.21.15.90:0/759775613 wait complete. 2024-04-25T01:16:57.230 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542154 for osd.3 2024-04-25T01:16:57.231 DEBUG:teuthology.parallel:result is None 2024-04-25T01:16:57.248 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542155 for osd.5 2024-04-25T01:16:57.248 DEBUG:teuthology.parallel:result is None 2024-04-25T01:16:57.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.290+0000 7f1f83e00640 1 -- 172.21.15.90:0/449414845 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f1f840729f0 con 0x7f1f840d02c0 2024-04-25T01:16:57.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.290+0000 7f1f7aa00640 1 -- 172.21.15.90:0/449414845 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1f7c07b590 con 0x7f1f840d02c0 2024-04-25T01:16:57.274 INFO:teuthology.orchestra.run.smithi090.stdout:60129542156 2024-04-25T01:16:57.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.292+0000 7f1f83e00640 1 -- 172.21.15.90:0/449414845 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1f6006be30 msgr2=0x7f1f6006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:57.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.292+0000 7f1f83e00640 1 --2- 172.21.15.90:0/449414845 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1f6006be30 0x7f1f6006e2f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1f74007fd0 tx=0x7f1f74009040 comp rx=0 tx=0).stop 2024-04-25T01:16:57.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.292+0000 7f1f83e00640 1 -- 172.21.15.90:0/449414845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f840d02c0 msgr2=0x7f1f841560e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:57.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.292+0000 7f1f83e00640 1 --2- 172.21.15.90:0/449414845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f840d02c0 0x7f1f841560e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1f840d0a00 tx=0x7f1f7c004a40 comp rx=0 tx=0).stop 2024-04-25T01:16:57.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.293+0000 7f1f83e00640 1 -- 172.21.15.90:0/449414845 shutdown_connections 2024-04-25T01:16:57.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.293+0000 7f1f83e00640 1 --2- 172.21.15.90:0/449414845 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1f6006be30 0x7f1f6006e2f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.293+0000 7f1f83e00640 1 --2- 172.21.15.90:0/449414845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f840d02c0 0x7f1f841560e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.293+0000 7f1f83e00640 1 --2- 172.21.15.90:0/449414845 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f840b49b0 0x7f1f841ebbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.293+0000 7f1f83e00640 1 --2- 172.21.15.90:0/449414845 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1f840b2fb0 0x7f1f841531a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.293+0000 7f1f83e00640 1 -- 172.21.15.90:0/449414845 >> 172.21.15.90:0/449414845 conn(0x7f1f8406b300 msgr2=0x7f1f84071d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:57.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.293+0000 7f1f83e00640 1 -- 172.21.15.90:0/449414845 shutdown_connections 2024-04-25T01:16:57.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.293+0000 7f1f83e00640 1 -- 172.21.15.90:0/449414845 wait complete. 2024-04-25T01:16:57.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.295+0000 7f40e1000640 1 -- 172.21.15.90:0/3554281415 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f40b4005850 con 0x7f40dc0b2b20 2024-04-25T01:16:57.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.296+0000 7f40d2000640 1 -- 172.21.15.90:0/3554281415 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f40cc05d2b0 con 0x7f40dc0b2b20 2024-04-25T01:16:57.280 INFO:teuthology.orchestra.run.smithi090.stdout:60129542154 2024-04-25T01:16:57.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.299+0000 7f8409600640 1 -- 172.21.15.90:0/485413632 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f84040678f0 con 0x7f84040d0f60 2024-04-25T01:16:57.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.300+0000 7f40e1000640 1 -- 172.21.15.90:0/3554281415 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f40bc06be30 msgr2=0x7f40bc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:57.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.300+0000 7f40e1000640 1 --2- 172.21.15.90:0/3554281415 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f40bc06be30 0x7f40bc06e2f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f40c80015a0 tx=0x7f40c8009040 comp rx=0 tx=0).stop 2024-04-25T01:16:57.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.300+0000 7f83fbe00640 1 -- 172.21.15.90:0/485413632 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f83f4059580 con 0x7f84040d0f60 2024-04-25T01:16:57.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.300+0000 7f40e1000640 1 -- 172.21.15.90:0/3554281415 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40dc0b2b20 msgr2=0x7f40dc200b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:57.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.300+0000 7f40e1000640 1 --2- 172.21.15.90:0/3554281415 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40dc0b2b20 0x7f40dc200b20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f40cc00c400 tx=0x7f40cc00c8c0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.300+0000 7f40e1000640 1 -- 172.21.15.90:0/3554281415 shutdown_connections 2024-04-25T01:16:57.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.300+0000 7f40e1000640 1 --2- 172.21.15.90:0/3554281415 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f40bc06be30 0x7f40bc06e2f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.300+0000 7f40e1000640 1 --2- 172.21.15.90:0/3554281415 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f40dc0cefe0 0x7f40dc0cd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.300+0000 7f40e1000640 1 --2- 172.21.15.90:0/3554281415 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40dc0b4af0 0x7f40dc0b0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.300+0000 7f40e1000640 1 --2- 172.21.15.90:0/3554281415 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40dc0b2b20 0x7f40dc200b20 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.300+0000 7f40e1000640 1 -- 172.21.15.90:0/3554281415 >> 172.21.15.90:0/3554281415 conn(0x7f40dc0b1820 msgr2=0x7f40dc0d0380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:57.285 INFO:teuthology.orchestra.run.smithi090.stdout:60129542154 2024-04-25T01:16:57.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.300+0000 7f40e1000640 1 -- 172.21.15.90:0/3554281415 shutdown_connections 2024-04-25T01:16:57.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.300+0000 7f40e1000640 1 -- 172.21.15.90:0/3554281415 wait complete. 2024-04-25T01:16:57.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.303+0000 7f8409600640 1 -- 172.21.15.90:0/485413632 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f83e006bf00 msgr2=0x7f83e006e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:57.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.303+0000 7f8409600640 1 --2- 172.21.15.90:0/485413632 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f83e006bf00 0x7f83e006e3c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f83fc034000 tx=0x7f83fc02f000 comp rx=0 tx=0).stop 2024-04-25T01:16:57.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.303+0000 7f8409600640 1 -- 172.21.15.90:0/485413632 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84040d0f60 msgr2=0x7f840415b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:57.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.303+0000 7f8409600640 1 --2- 172.21.15.90:0/485413632 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84040d0f60 0x7f840415b770 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f83f400c4e0 tx=0x7f83f400c9a0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.304+0000 7f8409600640 1 -- 172.21.15.90:0/485413632 shutdown_connections 2024-04-25T01:16:57.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.304+0000 7f8409600640 1 --2- 172.21.15.90:0/485413632 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f83e006bf00 0x7f83e006e3c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.304+0000 7f8409600640 1 --2- 172.21.15.90:0/485413632 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f840415f7f0 0x7f84040c8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.304+0000 7f8409600640 1 --2- 172.21.15.90:0/485413632 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84040d0f60 0x7f840415b770 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.304+0000 7f8409600640 1 --2- 172.21.15.90:0/485413632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84040b5340 0x7f84040cb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.304+0000 7f8409600640 1 -- 172.21.15.90:0/485413632 >> 172.21.15.90:0/485413632 conn(0x7f840409d170 msgr2=0x7f84040b0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:57.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.304+0000 7f8409600640 1 -- 172.21.15.90:0/485413632 shutdown_connections 2024-04-25T01:16:57.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.304+0000 7f8409600640 1 -- 172.21.15.90:0/485413632 wait complete. 2024-04-25T01:16:57.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.316+0000 7feb85400640 1 -- 172.21.15.90:0/3487013644 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7feb78057d70 con 0x7feb7803c0a0 2024-04-25T01:16:57.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.317+0000 7feb73e00640 1 -- 172.21.15.90:0/3487013644 <== mon.2 v2:172.21.15.103:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7feb800f60a0 con 0x7feb7803c0a0 2024-04-25T01:16:57.301 INFO:teuthology.orchestra.run.smithi090.stdout:60129542155 2024-04-25T01:16:57.302 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542155 got 60129542156 for osd.2 2024-04-25T01:16:57.302 DEBUG:teuthology.parallel:result is None 2024-04-25T01:16:57.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.320+0000 7feb85400640 1 -- 172.21.15.90:0/3487013644 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7feb6006be30 msgr2=0x7feb6006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:57.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.320+0000 7feb85400640 1 --2- 172.21.15.90:0/3487013644 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7feb6006be30 0x7feb6006e2f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7feb78052f00 tx=0x7feb74007e30 comp rx=0 tx=0).stop 2024-04-25T01:16:57.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.320+0000 7feb85400640 1 -- 172.21.15.90:0/3487013644 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb7803c0a0 msgr2=0x7feb78181110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:57.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.320+0000 7feb85400640 1 --2- 172.21.15.90:0/3487013644 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb7803c0a0 0x7feb78181110 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7feb80068830 tx=0x7feb80075020 comp rx=0 tx=0).stop 2024-04-25T01:16:57.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.320+0000 7feb85400640 1 -- 172.21.15.90:0/3487013644 shutdown_connections 2024-04-25T01:16:57.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.320+0000 7feb85400640 1 --2- 172.21.15.90:0/3487013644 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7feb6006be30 0x7feb6006e2f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.320+0000 7feb85400640 1 --2- 172.21.15.90:0/3487013644 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb780e3160 0x7feb7817d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.320+0000 7feb85400640 1 --2- 172.21.15.90:0/3487013644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb78053cb0 0x7feb7817b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.320+0000 7feb85400640 1 --2- 172.21.15.90:0/3487013644 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7feb7803c0a0 0x7feb78181110 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.320+0000 7feb85400640 1 -- 172.21.15.90:0/3487013644 >> 172.21.15.90:0/3487013644 conn(0x7feb78058ee0 msgr2=0x7feb780540e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:57.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.320+0000 7feb85400640 1 -- 172.21.15.90:0/3487013644 shutdown_connections 2024-04-25T01:16:57.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.320+0000 7feb85400640 1 -- 172.21.15.90:0/3487013644 wait complete. 2024-04-25T01:16:57.307 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542154 for osd.4 2024-04-25T01:16:57.307 DEBUG:teuthology.parallel:result is None 2024-04-25T01:16:57.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.327+0000 7ffb04600640 1 -- 172.21.15.90:0/3939072357 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7ffad8005850 con 0x7ffafc15ca50 2024-04-25T01:16:57.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.328+0000 7ffaf9600640 1 -- 172.21.15.90:0/3939072357 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ffaf007e540 con 0x7ffafc15ca50 2024-04-25T01:16:57.312 INFO:teuthology.orchestra.run.smithi090.stdout:60129542155 2024-04-25T01:16:57.313 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542154 for osd.6 2024-04-25T01:16:57.313 DEBUG:teuthology.parallel:result is None 2024-04-25T01:16:57.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.330+0000 7ffb04600640 1 -- 172.21.15.90:0/3939072357 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ffadc06be30 msgr2=0x7ffadc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:57.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.330+0000 7ffb04600640 1 --2- 172.21.15.90:0/3939072357 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ffadc06be30 0x7ffadc06e2f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ffaf4005660 tx=0x7ffaf4003d90 comp rx=0 tx=0).stop 2024-04-25T01:16:57.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.330+0000 7ffb04600640 1 -- 172.21.15.90:0/3939072357 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffafc15ca50 msgr2=0x7ffafc1f7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:57.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.330+0000 7ffb04600640 1 --2- 172.21.15.90:0/3939072357 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffafc15ca50 0x7ffafc1f7c10 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ffaf002a2b0 tx=0x7ffaf00046c0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.330+0000 7ffb04600640 1 -- 172.21.15.90:0/3939072357 shutdown_connections 2024-04-25T01:16:57.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.330+0000 7ffb04600640 1 --2- 172.21.15.90:0/3939072357 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ffadc06be30 0x7ffadc06e2f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.330+0000 7ffb04600640 1 --2- 172.21.15.90:0/3939072357 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffafc15ca50 0x7ffafc1f7c10 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.330+0000 7ffb04600640 1 --2- 172.21.15.90:0/3939072357 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffafc0c5040 0x7ffafc0c84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.330+0000 7ffb04600640 1 --2- 172.21.15.90:0/3939072357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffafc0c4b90 0x7ffafc0c56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.330+0000 7ffb04600640 1 -- 172.21.15.90:0/3939072357 >> 172.21.15.90:0/3939072357 conn(0x7ffafc0d1bc0 msgr2=0x7ffafc0d14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:57.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.330+0000 7ffb04600640 1 -- 172.21.15.90:0/3939072357 shutdown_connections 2024-04-25T01:16:57.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.330+0000 7ffb04600640 1 -- 172.21.15.90:0/3939072357 wait complete. 2024-04-25T01:16:57.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.337+0000 7ff572200640 1 -- 172.21.15.90:0/3819531420 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7ff56c0b4810 con 0x7ff56c06f3e0 2024-04-25T01:16:57.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.338+0000 7ff563400640 1 -- 172.21.15.90:0/3819531420 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff56405da20 con 0x7ff56c06f3e0 2024-04-25T01:16:57.322 INFO:teuthology.orchestra.run.smithi090.stdout:60129542154 2024-04-25T01:16:57.325 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542155 for osd.7 2024-04-25T01:16:57.325 DEBUG:teuthology.parallel:result is None 2024-04-25T01:16:57.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.341+0000 7ff572200640 1 -- 172.21.15.90:0/3819531420 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff54c06be30 msgr2=0x7ff54c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:57.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.341+0000 7ff572200640 1 --2- 172.21.15.90:0/3819531420 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff54c06be30 0x7ff54c06e2f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff56c1fb720 tx=0x7ff558008040 comp rx=0 tx=0).stop 2024-04-25T01:16:57.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.341+0000 7ff572200640 1 -- 172.21.15.90:0/3819531420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff56c06f3e0 msgr2=0x7ff56c0b69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:57.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.341+0000 7ff572200640 1 --2- 172.21.15.90:0/3819531420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff56c06f3e0 0x7ff56c0b69b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff56400cbf0 tx=0x7ff56400a060 comp rx=0 tx=0).stop 2024-04-25T01:16:57.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.341+0000 7ff572200640 1 -- 172.21.15.90:0/3819531420 shutdown_connections 2024-04-25T01:16:57.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.341+0000 7ff572200640 1 --2- 172.21.15.90:0/3819531420 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff54c06be30 0x7ff54c06e2f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.341+0000 7ff572200640 1 --2- 172.21.15.90:0/3819531420 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff56c15f630 0x7ff56c0ce280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.341+0000 7ff572200640 1 --2- 172.21.15.90:0/3819531420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff56c0d28a0 0x7ff56c15ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.341+0000 7ff572200640 1 --2- 172.21.15.90:0/3819531420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff56c06f3e0 0x7ff56c0b69b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.341+0000 7ff572200640 1 -- 172.21.15.90:0/3819531420 >> 172.21.15.90:0/3819531420 conn(0x7ff56c0d1ca0 msgr2=0x7ff56c06c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:57.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.341+0000 7ff572200640 1 -- 172.21.15.90:0/3819531420 shutdown_connections 2024-04-25T01:16:57.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.341+0000 7ff572200640 1 -- 172.21.15.90:0/3819531420 wait complete. 2024-04-25T01:16:57.335 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542155 for osd.0 2024-04-25T01:16:57.335 DEBUG:teuthology.parallel:result is None 2024-04-25T01:16:57.343 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542154 for osd.1 2024-04-25T01:16:57.343 DEBUG:teuthology.parallel:result is None 2024-04-25T01:16:57.343 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-25T01:16:57.343 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:16:57.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.596+0000 7fb59f200640 1 Processor -- start 2024-04-25T01:16:57.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.596+0000 7fb59f200640 1 -- start start 2024-04-25T01:16:57.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.597+0000 7fb59f200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb5a006b120 0x7fb5a006b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:57.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.597+0000 7fb59f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb5a00ae730 0x7fb5a00cd900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:57.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.598+0000 7fb59f200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5a006ba40 con 0x7fb5a015ca10 2024-04-25T01:16:57.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.598+0000 7fb59f200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fb5a00cef40 con 0x7fb5a006b120 2024-04-25T01:16:57.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.598+0000 7fb59f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb5a015c580 con 0x7fb5a00ae730 2024-04-25T01:16:57.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.598+0000 7fb59de00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb5a006b120 0x7fb5a006b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:57.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.598+0000 7fb59e800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fb5a015ca10 0x7fb5a00cc030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:35680/0 (socket says 172.21.15.90:35680) 2024-04-25T01:16:57.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.598+0000 7fb59e800640 1 -- 172.21.15.90:0/803458209 learned_addr learned my addr 172.21.15.90:0/803458209 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:57.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.598+0000 7fb59d400640 1 --2- 172.21.15.90:0/803458209 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb5a00ae730 0x7fb5a00cd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:57.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.598+0000 7fb59de00640 1 -- 172.21.15.90:0/803458209 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb5a00ae730 msgr2=0x7fb5a00cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:57.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.598+0000 7fb59de00640 1 --2- 172.21.15.90:0/803458209 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb5a00ae730 0x7fb5a00cd900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.598+0000 7fb59de00640 1 -- 172.21.15.90:0/803458209 >> v1:172.21.15.90:6789/0 conn(0x7fb5a015ca10 legacy=0x7fb5a00cc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:57.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.598+0000 7fb59de00640 1 -- 172.21.15.90:0/803458209 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5a015c220 con 0x7fb5a006b120 2024-04-25T01:16:57.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.599+0000 7fb59de00640 1 --2- 172.21.15.90:0/803458209 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5a006b120 0x7fb5a006b500 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb5900097a0 tx=0x7fb59002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd6ee8ff034ca636 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.600+0000 7fb59ca00640 1 -- 172.21.15.90:0/803458209 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb590037070 con 0x7fb5a006b120 2024-04-25T01:16:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.600+0000 7fb59ca00640 1 -- 172.21.15.90:0/803458209 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb590032440 con 0x7fb5a006b120 2024-04-25T01:16:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.600+0000 7fb59ca00640 1 -- 172.21.15.90:0/803458209 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb590032810 con 0x7fb5a006b120 2024-04-25T01:16:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.600+0000 7fb59f200640 1 -- 172.21.15.90:0/803458209 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5a006b120 msgr2=0x7fb5a006b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.600+0000 7fb59f200640 1 --2- 172.21.15.90:0/803458209 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5a006b120 0x7fb5a006b500 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb5900097a0 tx=0x7fb59002a180 comp rx=0 tx=0).stop 2024-04-25T01:16:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.600+0000 7fb59f200640 1 -- 172.21.15.90:0/803458209 shutdown_connections 2024-04-25T01:16:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.600+0000 7fb59f200640 1 --2- 172.21.15.90:0/803458209 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb5a00ae730 0x7fb5a00cd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.600+0000 7fb59f200640 1 --2- 172.21.15.90:0/803458209 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5a006b120 0x7fb5a006b500 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.600+0000 7fb59f200640 1 -- 172.21.15.90:0/803458209 >> 172.21.15.90:0/803458209 conn(0x7fb5a00b1660 msgr2=0x7fb5a00b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.600+0000 7fb59f200640 1 -- 172.21.15.90:0/803458209 shutdown_connections 2024-04-25T01:16:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.600+0000 7fb59f200640 1 -- 172.21.15.90:0/803458209 wait complete. 2024-04-25T01:16:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.602+0000 7fb59f200640 1 Processor -- start 2024-04-25T01:16:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.602+0000 7fb59f200640 1 -- start start 2024-04-25T01:16:57.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.603+0000 7fb59f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5a006b120 0x7fb5a01e4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:57.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.603+0000 7fb59f200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5a00ae730 0x7fb5a01e3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:57.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.603+0000 7fb59f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb5a015ca10 0x7fb5a01f1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:57.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.603+0000 7fb59f200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a00af240 con 0x7fb5a00ae730 2024-04-25T01:16:57.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.603+0000 7fb59f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a00aef60 con 0x7fb5a006b120 2024-04-25T01:16:57.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.603+0000 7fb59f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb5a0153080 con 0x7fb5a015ca10 2024-04-25T01:16:57.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.603+0000 7fb59d400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5a00ae730 0x7fb5a01e3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:57.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.604+0000 7fb59d400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5a00ae730 0x7fb5a01e3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56316/0 (socket says 172.21.15.90:56316) 2024-04-25T01:16:57.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.604+0000 7fb59d400640 1 -- 172.21.15.90:0/747447614 learned_addr learned my addr 172.21.15.90:0/747447614 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:57.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.604+0000 7fb59de00640 1 --2- 172.21.15.90:0/747447614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5a006b120 0x7fb5a01e4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:57.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.604+0000 7fb59d400640 1 -- 172.21.15.90:0/747447614 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb5a015ca10 msgr2=0x7fb5a01f1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:57.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.604+0000 7fb59d400640 1 --2- 172.21.15.90:0/747447614 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb5a015ca10 0x7fb5a01f1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.604+0000 7fb59d400640 1 -- 172.21.15.90:0/747447614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5a006b120 msgr2=0x7fb5a01e4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:57.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.604+0000 7fb59d400640 1 --2- 172.21.15.90:0/747447614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5a006b120 0x7fb5a01e4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.604+0000 7fb59d400640 1 -- 172.21.15.90:0/747447614 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5a00ab680 con 0x7fb5a00ae730 2024-04-25T01:16:57.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.604+0000 7fb59d400640 1 --2- 172.21.15.90:0/747447614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5a00ae730 0x7fb5a01e3140 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb58c00d800 tx=0x7fb58c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:57.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.604+0000 7fb596000640 1 -- 172.21.15.90:0/747447614 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb58c00b6c0 con 0x7fb5a00ae730 2024-04-25T01:16:57.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.604+0000 7fb596000640 1 -- 172.21.15.90:0/747447614 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb58c019070 con 0x7fb5a00ae730 2024-04-25T01:16:57.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.604+0000 7fb596000640 1 -- 172.21.15.90:0/747447614 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb58c00ba80 con 0x7fb5a00ae730 2024-04-25T01:16:57.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.604+0000 7fb59f200640 1 -- 172.21.15.90:0/747447614 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5a00ab960 con 0x7fb5a00ae730 2024-04-25T01:16:57.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.604+0000 7fb59f200640 1 -- 172.21.15.90:0/747447614 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5a00abf20 con 0x7fb5a00ae730 2024-04-25T01:16:57.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.605+0000 7fb59f200640 1 -- 172.21.15.90:0/747447614 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5a0075350 con 0x7fb5a00ae730 2024-04-25T01:16:57.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.606+0000 7fb596000640 1 -- 172.21.15.90:0/747447614 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb58c00bc20 con 0x7fb5a00ae730 2024-04-25T01:16:57.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.607+0000 7fb596000640 1 --2- 172.21.15.90:0/747447614 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb58006be30 0x7fb58006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:57.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.607+0000 7fb596000640 1 -- 172.21.15.90:0/747447614 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fb58c08b5a0 con 0x7fb5a00ae730 2024-04-25T01:16:57.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.607+0000 7fb59de00640 1 --2- 172.21.15.90:0/747447614 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb58006be30 0x7fb58006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:57.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.608+0000 7fb59de00640 1 --2- 172.21.15.90:0/747447614 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb58006be30 0x7fb58006e2f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb590034000 tx=0x7fb59002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:57.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.612+0000 7fb596000640 1 -- 172.21.15.90:0/747447614 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb58c059ca0 con 0x7fb5a00ae730 2024-04-25T01:16:57.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.789+0000 7fb59f200640 1 -- 172.21.15.90:0/747447614 --> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb5a006ccd0 con 0x7fb58006be30 2024-04-25T01:16:57.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.814+0000 7fb596000640 1 -- 172.21.15.90:0/747447614 <== mgr.4104 v2:172.21.15.90:6832/2198086733 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332533 (secure 0 0 0) 0x7fb5a006ccd0 con 0x7fb58006be30 2024-04-25T01:16:57.800 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:16:57.810 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-25T01:16:57.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.821+0000 7fb59f200640 1 -- 172.21.15.90:0/747447614 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb58006be30 msgr2=0x7fb58006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:57.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.821+0000 7fb59f200640 1 --2- 172.21.15.90:0/747447614 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb58006be30 0x7fb58006e2f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb590034000 tx=0x7fb59002e000 comp rx=0 tx=0).stop 2024-04-25T01:16:57.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.821+0000 7fb59f200640 1 -- 172.21.15.90:0/747447614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5a00ae730 msgr2=0x7fb5a01e3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:57.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.821+0000 7fb59f200640 1 --2- 172.21.15.90:0/747447614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5a00ae730 0x7fb5a01e3140 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb58c00d800 tx=0x7fb58c00dcc0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.821+0000 7fb59f200640 1 -- 172.21.15.90:0/747447614 shutdown_connections 2024-04-25T01:16:57.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.821+0000 7fb59f200640 1 --2- 172.21.15.90:0/747447614 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb58006be30 0x7fb58006e2f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.821+0000 7fb59f200640 1 --2- 172.21.15.90:0/747447614 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb5a015ca10 0x7fb5a01f1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.821+0000 7fb59f200640 1 --2- 172.21.15.90:0/747447614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5a00ae730 0x7fb5a01e3140 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.821+0000 7fb59f200640 1 --2- 172.21.15.90:0/747447614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5a006b120 0x7fb5a01e4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:57.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.821+0000 7fb59f200640 1 -- 172.21.15.90:0/747447614 >> 172.21.15.90:0/747447614 conn(0x7fb5a00b1660 msgr2=0x7fb5a006de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:57.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.821+0000 7fb59f200640 1 -- 172.21.15.90:0/747447614 shutdown_connections 2024-04-25T01:16:57.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:57.821+0000 7fb59f200640 1 -- 172.21.15.90:0/747447614 wait complete. 2024-04-25T01:16:57.823 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-04-25T01:16:55.835317+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219536,"kb_used_data":3368,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755184,"statfs":{"total":773094113280,"available":772869308416,"internally_reserved":0,"allocated":3448832,"data_stored":2055328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004562"},"pg_stats":[{"pgid":"2.1f","version":"21'4","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.698700+0000","last_change":"2024-04-25T01:16:21.622879+0000","last_active":"2024-04-25T01:16:27.698700+0000","last_peered":"2024-04-25T01:16:27.698700+0000","last_clean":"2024-04-25T01:16:27.698700+0000","last_became_active":"2024-04-25T01:16:21.622586+0000","last_became_peered":"2024-04-25T01:16:21.622586+0000","last_unstale":"2024-04-25T01:16:27.698700+0000","last_undegraded":"2024-04-25T01:16:27.698700+0000","last_fullsized":"2024-04-25T01:16:27.698700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:50.882927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699904+0000","last_change":"2024-04-25T01:16:22.615827+0000","last_active":"2024-04-25T01:16:27.699904+0000","last_peered":"2024-04-25T01:16:27.699904+0000","last_clean":"2024-04-25T01:16:27.699904+0000","last_became_active":"2024-04-25T01:16:22.615497+0000","last_became_peered":"2024-04-25T01:16:22.615497+0000","last_unstale":"2024-04-25T01:16:27.699904+0000","last_undegraded":"2024-04-25T01:16:27.699904+0000","last_fullsized":"2024-04-25T01:16:27.699904+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:03:53.876481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.223962+0000","last_change":"2024-04-25T01:16:26.672437+0000","last_active":"2024-04-25T01:16:28.223962+0000","last_peered":"2024-04-25T01:16:28.223962+0000","last_clean":"2024-04-25T01:16:28.223962+0000","last_became_active":"2024-04-25T01:16:26.671999+0000","last_became_peered":"2024-04-25T01:16:26.671999+0000","last_unstale":"2024-04-25T01:16:28.223962+0000","last_undegraded":"2024-04-25T01:16:28.223962+0000","last_fullsized":"2024-04-25T01:16:28.223962+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:56.586900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4,1],"acting":[7,6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.700145+0000","last_change":"2024-04-25T01:16:21.609447+0000","last_active":"2024-04-25T01:16:27.700145+0000","last_peered":"2024-04-25T01:16:27.700145+0000","last_clean":"2024-04-25T01:16:27.700145+0000","last_became_active":"2024-04-25T01:16:21.608490+0000","last_became_peered":"2024-04-25T01:16:21.608490+0000","last_unstale":"2024-04-25T01:16:27.700145+0000","last_undegraded":"2024-04-25T01:16:27.700145+0000","last_fullsized":"2024-04-25T01:16:27.700145+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:15.007939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699632+0000","last_change":"2024-04-25T01:16:22.613891+0000","last_active":"2024-04-25T01:16:27.699632+0000","last_peered":"2024-04-25T01:16:27.699632+0000","last_clean":"2024-04-25T01:16:27.699632+0000","last_became_active":"2024-04-25T01:16:22.613471+0000","last_became_peered":"2024-04-25T01:16:22.613471+0000","last_unstale":"2024-04-25T01:16:27.699632+0000","last_undegraded":"2024-04-25T01:16:27.699632+0000","last_fullsized":"2024-04-25T01:16:27.699632+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:37:00.182534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.874736+0000","last_change":"2024-04-25T01:16:26.653705+0000","last_active":"2024-04-25T01:16:27.874736+0000","last_peered":"2024-04-25T01:16:27.874736+0000","last_clean":"2024-04-25T01:16:27.874736+0000","last_became_active":"2024-04-25T01:16:26.653348+0000","last_became_peered":"2024-04-25T01:16:26.653348+0000","last_unstale":"2024-04-25T01:16:27.874736+0000","last_undegraded":"2024-04-25T01:16:27.874736+0000","last_fullsized":"2024-04-25T01:16:27.874736+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:58:37.794558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4,6],"acting":[2,7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.772478+0000","last_change":"2024-04-25T01:16:21.611644+0000","last_active":"2024-04-25T01:16:29.772478+0000","last_peered":"2024-04-25T01:16:29.772478+0000","last_clean":"2024-04-25T01:16:29.772478+0000","last_became_active":"2024-04-25T01:16:21.611346+0000","last_became_peered":"2024-04-25T01:16:21.611346+0000","last_unstale":"2024-04-25T01:16:29.772478+0000","last_undegraded":"2024-04-25T01:16:29.772478+0000","last_fullsized":"2024-04-25T01:16:29.772478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:44.675184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.971313+0000","last_change":"2024-04-25T01:16:22.627961+0000","last_active":"2024-04-25T01:16:27.971313+0000","last_peered":"2024-04-25T01:16:27.971313+0000","last_clean":"2024-04-25T01:16:27.971313+0000","last_became_active":"2024-04-25T01:16:22.626767+0000","last_became_peered":"2024-04-25T01:16:22.626767+0000","last_unstale":"2024-04-25T01:16:27.971313+0000","last_undegraded":"2024-04-25T01:16:27.971313+0000","last_fullsized":"2024-04-25T01:16:27.971313+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:49:05.483350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.875764+0000","last_change":"2024-04-25T01:16:26.657547+0000","last_active":"2024-04-25T01:16:27.875764+0000","last_peered":"2024-04-25T01:16:27.875764+0000","last_clean":"2024-04-25T01:16:27.875764+0000","last_became_active":"2024-04-25T01:16:26.656643+0000","last_became_peered":"2024-04-25T01:16:26.656643+0000","last_unstale":"2024-04-25T01:16:27.875764+0000","last_undegraded":"2024-04-25T01:16:27.875764+0000","last_fullsized":"2024-04-25T01:16:27.875764+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:05.563072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3,6],"acting":[2,5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.772473+0000","last_change":"2024-04-25T01:16:21.617309+0000","last_active":"2024-04-25T01:16:29.772473+0000","last_peered":"2024-04-25T01:16:29.772473+0000","last_clean":"2024-04-25T01:16:29.772473+0000","last_became_active":"2024-04-25T01:16:21.617021+0000","last_became_peered":"2024-04-25T01:16:21.617021+0000","last_unstale":"2024-04-25T01:16:29.772473+0000","last_undegraded":"2024-04-25T01:16:29.772473+0000","last_fullsized":"2024-04-25T01:16:29.772473+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:06.940097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694375+0000","last_change":"2024-04-25T01:16:22.639034+0000","last_active":"2024-04-25T01:16:27.694375+0000","last_peered":"2024-04-25T01:16:27.694375+0000","last_clean":"2024-04-25T01:16:27.694375+0000","last_became_active":"2024-04-25T01:16:22.638661+0000","last_became_peered":"2024-04-25T01:16:22.638661+0000","last_unstale":"2024-04-25T01:16:27.694375+0000","last_undegraded":"2024-04-25T01:16:27.694375+0000","last_fullsized":"2024-04-25T01:16:27.694375+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:28.044482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.875219+0000","last_change":"2024-04-25T01:16:26.656419+0000","last_active":"2024-04-25T01:16:27.875219+0000","last_peered":"2024-04-25T01:16:27.875219+0000","last_clean":"2024-04-25T01:16:27.875219+0000","last_became_active":"2024-04-25T01:16:26.655743+0000","last_became_peered":"2024-04-25T01:16:26.655743+0000","last_unstale":"2024-04-25T01:16:27.875219+0000","last_undegraded":"2024-04-25T01:16:27.875219+0000","last_fullsized":"2024-04-25T01:16:27.875219+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:41:04.510055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5,7],"acting":[2,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694711+0000","last_change":"2024-04-25T01:16:22.621126+0000","last_active":"2024-04-25T01:16:27.694711+0000","last_peered":"2024-04-25T01:16:27.694711+0000","last_clean":"2024-04-25T01:16:27.694711+0000","last_became_active":"2024-04-25T01:16:22.620692+0000","last_became_peered":"2024-04-25T01:16:22.620692+0000","last_unstale":"2024-04-25T01:16:27.694711+0000","last_undegraded":"2024-04-25T01:16:27.694711+0000","last_fullsized":"2024-04-25T01:16:27.694711+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:28.261961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694834+0000","last_change":"2024-04-25T01:16:21.608564+0000","last_active":"2024-04-25T01:16:27.694834+0000","last_peered":"2024-04-25T01:16:27.694834+0000","last_clean":"2024-04-25T01:16:27.694834+0000","last_became_active":"2024-04-25T01:16:21.608227+0000","last_became_peered":"2024-04-25T01:16:21.608227+0000","last_unstale":"2024-04-25T01:16:27.694834+0000","last_undegraded":"2024-04-25T01:16:27.694834+0000","last_fullsized":"2024-04-25T01:16:27.694834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:38:08.656990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694160+0000","last_change":"2024-04-25T01:16:26.656167+0000","last_active":"2024-04-25T01:16:27.694160+0000","last_peered":"2024-04-25T01:16:27.694160+0000","last_clean":"2024-04-25T01:16:27.694160+0000","last_became_active":"2024-04-25T01:16:26.655643+0000","last_became_peered":"2024-04-25T01:16:26.655643+0000","last_unstale":"2024-04-25T01:16:27.694160+0000","last_undegraded":"2024-04-25T01:16:27.694160+0000","last_fullsized":"2024-04-25T01:16:27.694160+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:17:57.227839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,5],"acting":[3,1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.224130+0000","last_change":"2024-04-25T01:16:22.630130+0000","last_active":"2024-04-25T01:16:28.224130+0000","last_peered":"2024-04-25T01:16:28.224130+0000","last_clean":"2024-04-25T01:16:28.224130+0000","last_became_active":"2024-04-25T01:16:22.629829+0000","last_became_peered":"2024-04-25T01:16:22.629829+0000","last_unstale":"2024-04-25T01:16:28.224130+0000","last_undegraded":"2024-04-25T01:16:28.224130+0000","last_fullsized":"2024-04-25T01:16:28.224130+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:48:42.505862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.086752+0000","last_change":"2024-04-25T01:16:21.615603+0000","last_active":"2024-04-25T01:16:30.086752+0000","last_peered":"2024-04-25T01:16:30.086752+0000","last_clean":"2024-04-25T01:16:30.086752+0000","last_became_active":"2024-04-25T01:16:21.615217+0000","last_became_peered":"2024-04-25T01:16:21.615217+0000","last_unstale":"2024-04-25T01:16:30.086752+0000","last_undegraded":"2024-04-25T01:16:30.086752+0000","last_fullsized":"2024-04-25T01:16:30.086752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:10:00.414303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.773845+0000","last_change":"2024-04-25T01:16:26.677623+0000","last_active":"2024-04-25T01:16:29.773845+0000","last_peered":"2024-04-25T01:16:29.773845+0000","last_clean":"2024-04-25T01:16:29.773845+0000","last_became_active":"2024-04-25T01:16:26.677212+0000","last_became_peered":"2024-04-25T01:16:26.677212+0000","last_unstale":"2024-04-25T01:16:29.773845+0000","last_undegraded":"2024-04-25T01:16:29.773845+0000","last_fullsized":"2024-04-25T01:16:29.773845+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:58.601828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6,1],"acting":[0,4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694468+0000","last_change":"2024-04-25T01:16:22.617565+0000","last_active":"2024-04-25T01:16:27.694468+0000","last_peered":"2024-04-25T01:16:27.694468+0000","last_clean":"2024-04-25T01:16:27.694468+0000","last_became_active":"2024-04-25T01:16:22.617247+0000","last_became_peered":"2024-04-25T01:16:22.617247+0000","last_unstale":"2024-04-25T01:16:27.694468+0000","last_undegraded":"2024-04-25T01:16:27.694468+0000","last_fullsized":"2024-04-25T01:16:27.694468+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:00:17.366512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695722+0000","last_change":"2024-04-25T01:16:21.620221+0000","last_active":"2024-04-25T01:16:27.695722+0000","last_peered":"2024-04-25T01:16:27.695722+0000","last_clean":"2024-04-25T01:16:27.695722+0000","last_became_active":"2024-04-25T01:16:21.619923+0000","last_became_peered":"2024-04-25T01:16:21.619923+0000","last_unstale":"2024-04-25T01:16:27.695722+0000","last_undegraded":"2024-04-25T01:16:27.695722+0000","last_fullsized":"2024-04-25T01:16:27.695722+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:06.940097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.225195+0000","last_change":"2024-04-25T01:16:26.674824+0000","last_active":"2024-04-25T01:16:28.225195+0000","last_peered":"2024-04-25T01:16:28.225195+0000","last_clean":"2024-04-25T01:16:28.225195+0000","last_became_active":"2024-04-25T01:16:26.672608+0000","last_became_peered":"2024-04-25T01:16:26.672608+0000","last_unstale":"2024-04-25T01:16:28.225195+0000","last_undegraded":"2024-04-25T01:16:28.225195+0000","last_fullsized":"2024-04-25T01:16:28.225195+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:46:37.320887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2,3],"acting":[7,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.693656+0000","last_change":"2024-04-25T01:16:22.624853+0000","last_active":"2024-04-25T01:16:27.693656+0000","last_peered":"2024-04-25T01:16:27.693656+0000","last_clean":"2024-04-25T01:16:27.693656+0000","last_became_active":"2024-04-25T01:16:22.624275+0000","last_became_peered":"2024-04-25T01:16:22.624275+0000","last_unstale":"2024-04-25T01:16:27.693656+0000","last_undegraded":"2024-04-25T01:16:27.693656+0000","last_fullsized":"2024-04-25T01:16:27.693656+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:09.176572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.772092+0000","last_change":"2024-04-25T01:16:21.616953+0000","last_active":"2024-04-25T01:16:29.772092+0000","last_peered":"2024-04-25T01:16:29.772092+0000","last_clean":"2024-04-25T01:16:29.772092+0000","last_became_active":"2024-04-25T01:16:21.615656+0000","last_became_peered":"2024-04-25T01:16:21.615656+0000","last_unstale":"2024-04-25T01:16:29.772092+0000","last_undegraded":"2024-04-25T01:16:29.772092+0000","last_fullsized":"2024-04-25T01:16:29.772092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:48:41.491267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.876501+0000","last_change":"2024-04-25T01:16:26.661383+0000","last_active":"2024-04-25T01:16:27.876501+0000","last_peered":"2024-04-25T01:16:27.876501+0000","last_clean":"2024-04-25T01:16:27.876501+0000","last_became_active":"2024-04-25T01:16:26.658181+0000","last_became_peered":"2024-04-25T01:16:26.658181+0000","last_unstale":"2024-04-25T01:16:27.876501+0000","last_undegraded":"2024-04-25T01:16:27.876501+0000","last_fullsized":"2024-04-25T01:16:27.876501+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:23:19.810360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,7],"acting":[2,1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.697025+0000","last_change":"2024-04-25T01:16:22.618495+0000","last_active":"2024-04-25T01:16:27.697025+0000","last_peered":"2024-04-25T01:16:27.697025+0000","last_clean":"2024-04-25T01:16:27.697025+0000","last_became_active":"2024-04-25T01:16:22.618155+0000","last_became_peered":"2024-04-25T01:16:22.618155+0000","last_unstale":"2024-04-25T01:16:27.697025+0000","last_undegraded":"2024-04-25T01:16:27.697025+0000","last_fullsized":"2024-04-25T01:16:27.697025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:53.625025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695692+0000","last_change":"2024-04-25T01:16:21.618714+0000","last_active":"2024-04-25T01:16:27.695692+0000","last_peered":"2024-04-25T01:16:27.695692+0000","last_clean":"2024-04-25T01:16:27.695692+0000","last_became_active":"2024-04-25T01:16:21.618439+0000","last_became_peered":"2024-04-25T01:16:21.618439+0000","last_unstale":"2024-04-25T01:16:27.695692+0000","last_undegraded":"2024-04-25T01:16:27.695692+0000","last_fullsized":"2024-04-25T01:16:27.695692+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:17:15.320611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.876136+0000","last_change":"2024-04-25T01:16:26.649207+0000","last_active":"2024-04-25T01:16:27.876136+0000","last_peered":"2024-04-25T01:16:27.876136+0000","last_clean":"2024-04-25T01:16:27.876136+0000","last_became_active":"2024-04-25T01:16:26.648837+0000","last_became_peered":"2024-04-25T01:16:26.648837+0000","last_unstale":"2024-04-25T01:16:27.876136+0000","last_undegraded":"2024-04-25T01:16:27.876136+0000","last_fullsized":"2024-04-25T01:16:27.876136+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:13.227403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,1],"acting":[2,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.698475+0000","last_change":"2024-04-25T01:16:22.627735+0000","last_active":"2024-04-25T01:16:27.698475+0000","last_peered":"2024-04-25T01:16:27.698475+0000","last_clean":"2024-04-25T01:16:27.698475+0000","last_became_active":"2024-04-25T01:16:22.627430+0000","last_became_peered":"2024-04-25T01:16:22.627430+0000","last_unstale":"2024-04-25T01:16:27.698475+0000","last_undegraded":"2024-04-25T01:16:27.698475+0000","last_fullsized":"2024-04-25T01:16:27.698475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:06:31.257662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.697307+0000","last_change":"2024-04-25T01:16:21.620348+0000","last_active":"2024-04-25T01:16:27.697307+0000","last_peered":"2024-04-25T01:16:27.697307+0000","last_clean":"2024-04-25T01:16:27.697307+0000","last_became_active":"2024-04-25T01:16:21.620049+0000","last_became_peered":"2024-04-25T01:16:21.620049+0000","last_unstale":"2024-04-25T01:16:27.697307+0000","last_undegraded":"2024-04-25T01:16:27.697307+0000","last_fullsized":"2024-04-25T01:16:27.697307+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:17:59.668250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694701+0000","last_change":"2024-04-25T01:16:26.672455+0000","last_active":"2024-04-25T01:16:27.694701+0000","last_peered":"2024-04-25T01:16:27.694701+0000","last_clean":"2024-04-25T01:16:27.694701+0000","last_became_active":"2024-04-25T01:16:26.671952+0000","last_became_peered":"2024-04-25T01:16:26.671952+0000","last_unstale":"2024-04-25T01:16:27.694701+0000","last_undegraded":"2024-04-25T01:16:27.694701+0000","last_fullsized":"2024-04-25T01:16:27.694701+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:13.227403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6,3],"acting":[4,1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.088865+0000","last_change":"2024-04-25T01:16:22.628050+0000","last_active":"2024-04-25T01:16:30.088865+0000","last_peered":"2024-04-25T01:16:30.088865+0000","last_clean":"2024-04-25T01:16:30.088865+0000","last_became_active":"2024-04-25T01:16:22.626875+0000","last_became_peered":"2024-04-25T01:16:22.626875+0000","last_unstale":"2024-04-25T01:16:30.088865+0000","last_undegraded":"2024-04-25T01:16:30.088865+0000","last_fullsized":"2024-04-25T01:16:30.088865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:24:16.000792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695500+0000","last_change":"2024-04-25T01:16:21.621352+0000","last_active":"2024-04-25T01:16:27.695500+0000","last_peered":"2024-04-25T01:16:27.695500+0000","last_clean":"2024-04-25T01:16:27.695500+0000","last_became_active":"2024-04-25T01:16:21.620116+0000","last_became_peered":"2024-04-25T01:16:21.620116+0000","last_unstale":"2024-04-25T01:16:27.695500+0000","last_undegraded":"2024-04-25T01:16:27.695500+0000","last_fullsized":"2024-04-25T01:16:27.695500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:07:31.151855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.087941+0000","last_change":"2024-04-25T01:16:26.663371+0000","last_active":"2024-04-25T01:16:30.087941+0000","last_peered":"2024-04-25T01:16:30.087941+0000","last_clean":"2024-04-25T01:16:30.087941+0000","last_became_active":"2024-04-25T01:16:26.662972+0000","last_became_peered":"2024-04-25T01:16:26.662972+0000","last_unstale":"2024-04-25T01:16:30.087941+0000","last_undegraded":"2024-04-25T01:16:30.087941+0000","last_fullsized":"2024-04-25T01:16:30.087941+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:18.289163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2,7],"acting":[6,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.875472+0000","last_change":"2024-04-25T01:16:22.628735+0000","last_active":"2024-04-25T01:16:27.875472+0000","last_peered":"2024-04-25T01:16:27.875472+0000","last_clean":"2024-04-25T01:16:27.875472+0000","last_became_active":"2024-04-25T01:16:22.628429+0000","last_became_peered":"2024-04-25T01:16:22.628429+0000","last_unstale":"2024-04-25T01:16:27.875472+0000","last_undegraded":"2024-04-25T01:16:27.875472+0000","last_fullsized":"2024-04-25T01:16:27.875472+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:20.635412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.697942+0000","last_change":"2024-04-25T01:16:21.622977+0000","last_active":"2024-04-25T01:16:27.697942+0000","last_peered":"2024-04-25T01:16:27.697942+0000","last_clean":"2024-04-25T01:16:27.697942+0000","last_became_active":"2024-04-25T01:16:21.622298+0000","last_became_peered":"2024-04-25T01:16:21.622298+0000","last_unstale":"2024-04-25T01:16:27.697942+0000","last_undegraded":"2024-04-25T01:16:27.697942+0000","last_fullsized":"2024-04-25T01:16:27.697942+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:34.732380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.087427+0000","last_change":"2024-04-25T01:16:26.664207+0000","last_active":"2024-04-25T01:16:30.087427+0000","last_peered":"2024-04-25T01:16:30.087427+0000","last_clean":"2024-04-25T01:16:30.087427+0000","last_became_active":"2024-04-25T01:16:26.663749+0000","last_became_peered":"2024-04-25T01:16:26.663749+0000","last_unstale":"2024-04-25T01:16:30.087427+0000","last_undegraded":"2024-04-25T01:16:30.087427+0000","last_fullsized":"2024-04-25T01:16:30.087427+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:05.467463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5,7],"acting":[6,3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.773873+0000","last_change":"2024-04-25T01:16:22.635293+0000","last_active":"2024-04-25T01:16:29.773873+0000","last_peered":"2024-04-25T01:16:29.773873+0000","last_clean":"2024-04-25T01:16:29.773873+0000","last_became_active":"2024-04-25T01:16:22.635026+0000","last_became_peered":"2024-04-25T01:16:22.635026+0000","last_unstale":"2024-04-25T01:16:29.773873+0000","last_undegraded":"2024-04-25T01:16:29.773873+0000","last_fullsized":"2024-04-25T01:16:29.773873+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:46:16.608442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696109+0000","last_change":"2024-04-25T01:16:21.623621+0000","last_active":"2024-04-25T01:16:27.696109+0000","last_peered":"2024-04-25T01:16:27.696109+0000","last_clean":"2024-04-25T01:16:27.696109+0000","last_became_active":"2024-04-25T01:16:21.623351+0000","last_became_peered":"2024-04-25T01:16:21.623351+0000","last_unstale":"2024-04-25T01:16:27.696109+0000","last_undegraded":"2024-04-25T01:16:27.696109+0000","last_fullsized":"2024-04-25T01:16:27.696109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:49:04.468754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.088614+0000","last_change":"2024-04-25T01:16:26.656694+0000","last_active":"2024-04-25T01:16:30.088614+0000","last_peered":"2024-04-25T01:16:30.088614+0000","last_clean":"2024-04-25T01:16:30.088614+0000","last_became_active":"2024-04-25T01:16:26.656201+0000","last_became_peered":"2024-04-25T01:16:26.656201+0000","last_unstale":"2024-04-25T01:16:30.088614+0000","last_undegraded":"2024-04-25T01:16:30.088614+0000","last_fullsized":"2024-04-25T01:16:30.088614+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:39:18.953803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0,3],"acting":[6,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694802+0000","last_change":"2024-04-25T01:16:22.615076+0000","last_active":"2024-04-25T01:16:27.694802+0000","last_peered":"2024-04-25T01:16:27.694802+0000","last_clean":"2024-04-25T01:16:27.694802+0000","last_became_active":"2024-04-25T01:16:22.613139+0000","last_became_peered":"2024-04-25T01:16:22.613139+0000","last_unstale":"2024-04-25T01:16:27.694802+0000","last_undegraded":"2024-04-25T01:16:27.694802+0000","last_fullsized":"2024-04-25T01:16:27.694802+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:22:36.726844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.224133+0000","last_change":"2024-04-25T01:16:21.614492+0000","last_active":"2024-04-25T01:16:28.224133+0000","last_peered":"2024-04-25T01:16:28.224133+0000","last_clean":"2024-04-25T01:16:28.224133+0000","last_became_active":"2024-04-25T01:16:21.613490+0000","last_became_peered":"2024-04-25T01:16:21.613490+0000","last_unstale":"2024-04-25T01:16:28.224133+0000","last_undegraded":"2024-04-25T01:16:28.224133+0000","last_fullsized":"2024-04-25T01:16:28.224133+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:41:30.225989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.088458+0000","last_change":"2024-04-25T01:16:26.656061+0000","last_active":"2024-04-25T01:16:30.088458+0000","last_peered":"2024-04-25T01:16:30.088458+0000","last_clean":"2024-04-25T01:16:30.088458+0000","last_became_active":"2024-04-25T01:16:26.655189+0000","last_became_peered":"2024-04-25T01:16:26.655189+0000","last_unstale":"2024-04-25T01:16:30.088458+0000","last_undegraded":"2024-04-25T01:16:30.088458+0000","last_fullsized":"2024-04-25T01:16:30.088458+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:37:04.233365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,7],"acting":[6,0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.087313+0000","last_change":"2024-04-25T01:16:22.634160+0000","last_active":"2024-04-25T01:16:30.087313+0000","last_peered":"2024-04-25T01:16:30.087313+0000","last_clean":"2024-04-25T01:16:30.087313+0000","last_became_active":"2024-04-25T01:16:22.633856+0000","last_became_peered":"2024-04-25T01:16:22.633856+0000","last_unstale":"2024-04-25T01:16:30.087313+0000","last_undegraded":"2024-04-25T01:16:30.087313+0000","last_fullsized":"2024-04-25T01:16:30.087313+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:46:05.184032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695122+0000","last_change":"2024-04-25T01:16:21.617465+0000","last_active":"2024-04-25T01:16:27.695122+0000","last_peered":"2024-04-25T01:16:27.695122+0000","last_clean":"2024-04-25T01:16:27.695122+0000","last_became_active":"2024-04-25T01:16:21.617201+0000","last_became_peered":"2024-04-25T01:16:21.617201+0000","last_unstale":"2024-04-25T01:16:27.695122+0000","last_undegraded":"2024-04-25T01:16:27.695122+0000","last_fullsized":"2024-04-25T01:16:27.695122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:48:41.491267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699450+0000","last_change":"2024-04-25T01:16:26.666150+0000","last_active":"2024-04-25T01:16:27.699450+0000","last_peered":"2024-04-25T01:16:27.699450+0000","last_clean":"2024-04-25T01:16:27.699450+0000","last_became_active":"2024-04-25T01:16:26.665620+0000","last_became_peered":"2024-04-25T01:16:26.665620+0000","last_unstale":"2024-04-25T01:16:27.699450+0000","last_undegraded":"2024-04-25T01:16:27.699450+0000","last_fullsized":"2024-04-25T01:16:27.699450+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:51:23.702657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6,1],"acting":[5,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695397+0000","last_change":"2024-04-25T01:16:22.637692+0000","last_active":"2024-04-25T01:16:27.695397+0000","last_peered":"2024-04-25T01:16:27.695397+0000","last_clean":"2024-04-25T01:16:27.695397+0000","last_became_active":"2024-04-25T01:16:22.635225+0000","last_became_peered":"2024-04-25T01:16:22.635225+0000","last_unstale":"2024-04-25T01:16:27.695397+0000","last_undegraded":"2024-04-25T01:16:27.695397+0000","last_fullsized":"2024-04-25T01:16:27.695397+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:11.514022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695136+0000","last_change":"2024-04-25T01:16:21.608226+0000","last_active":"2024-04-25T01:16:27.695136+0000","last_peered":"2024-04-25T01:16:27.695136+0000","last_clean":"2024-04-25T01:16:27.695136+0000","last_became_active":"2024-04-25T01:16:21.607851+0000","last_became_peered":"2024-04-25T01:16:21.607851+0000","last_unstale":"2024-04-25T01:16:27.695136+0000","last_undegraded":"2024-04-25T01:16:27.695136+0000","last_fullsized":"2024-04-25T01:16:27.695136+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:51.521474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":178,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.970532+0000","last_change":"2024-04-25T01:16:17.466569+0000","last_active":"2024-04-25T01:16:27.970532+0000","last_peered":"2024-04-25T01:16:27.970532+0000","last_clean":"2024-04-25T01:16:27.970532+0000","last_became_active":"2024-04-25T01:16:17.465687+0000","last_became_peered":"2024-04-25T01:16:17.465687+0000","last_unstale":"2024-04-25T01:16:27.970532+0000","last_undegraded":"2024-04-25T01:16:27.970532+0000","last_fullsized":"2024-04-25T01:16:27.970532+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:16.440559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:16.440559+0000","last_clean_scrub_stamp":"2024-04-25T01:16:16.440559+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:18:17.493555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695469+0000","last_change":"2024-04-25T01:16:26.646040+0000","last_active":"2024-04-25T01:16:27.695469+0000","last_peered":"2024-04-25T01:16:27.695469+0000","last_clean":"2024-04-25T01:16:27.695469+0000","last_became_active":"2024-04-25T01:16:26.645500+0000","last_became_peered":"2024-04-25T01:16:26.645500+0000","last_unstale":"2024-04-25T01:16:27.695469+0000","last_undegraded":"2024-04-25T01:16:27.695469+0000","last_fullsized":"2024-04-25T01:16:27.695469+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:01:53.032237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2,0],"acting":[3,1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694653+0000","last_change":"2024-04-25T01:16:22.631511+0000","last_active":"2024-04-25T01:16:27.694653+0000","last_peered":"2024-04-25T01:16:27.694653+0000","last_clean":"2024-04-25T01:16:27.694653+0000","last_became_active":"2024-04-25T01:16:22.631169+0000","last_became_peered":"2024-04-25T01:16:22.631169+0000","last_unstale":"2024-04-25T01:16:27.694653+0000","last_undegraded":"2024-04-25T01:16:27.694653+0000","last_fullsized":"2024-04-25T01:16:27.694653+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:54.550997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.089006+0000","last_change":"2024-04-25T01:16:21.621474+0000","last_active":"2024-04-25T01:16:30.089006+0000","last_peered":"2024-04-25T01:16:30.089006+0000","last_clean":"2024-04-25T01:16:30.089006+0000","last_became_active":"2024-04-25T01:16:21.621139+0000","last_became_peered":"2024-04-25T01:16:21.621139+0000","last_unstale":"2024-04-25T01:16:30.089006+0000","last_undegraded":"2024-04-25T01:16:30.089006+0000","last_fullsized":"2024-04-25T01:16:30.089006+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:35:01.136892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.875567+0000","last_change":"2024-04-25T01:16:26.669916+0000","last_active":"2024-04-25T01:16:27.875567+0000","last_peered":"2024-04-25T01:16:27.875567+0000","last_clean":"2024-04-25T01:16:27.875567+0000","last_became_active":"2024-04-25T01:16:26.668054+0000","last_became_peered":"2024-04-25T01:16:26.668054+0000","last_unstale":"2024-04-25T01:16:27.875567+0000","last_undegraded":"2024-04-25T01:16:27.875567+0000","last_fullsized":"2024-04-25T01:16:27.875567+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:58:10.131890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7,4],"acting":[2,3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774586+0000","last_change":"2024-04-25T01:16:22.617183+0000","last_active":"2024-04-25T01:16:29.774586+0000","last_peered":"2024-04-25T01:16:29.774586+0000","last_clean":"2024-04-25T01:16:29.774586+0000","last_became_active":"2024-04-25T01:16:22.615748+0000","last_became_peered":"2024-04-25T01:16:22.615748+0000","last_unstale":"2024-04-25T01:16:29.774586+0000","last_undegraded":"2024-04-25T01:16:29.774586+0000","last_fullsized":"2024-04-25T01:16:29.774586+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:41:00.459225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699060+0000","last_change":"2024-04-25T01:16:21.617685+0000","last_active":"2024-04-25T01:16:27.699060+0000","last_peered":"2024-04-25T01:16:27.699060+0000","last_clean":"2024-04-25T01:16:27.699060+0000","last_became_active":"2024-04-25T01:16:21.617209+0000","last_became_peered":"2024-04-25T01:16:21.617209+0000","last_unstale":"2024-04-25T01:16:27.699060+0000","last_undegraded":"2024-04-25T01:16:27.699060+0000","last_fullsized":"2024-04-25T01:16:27.699060+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:34:39.714953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696564+0000","last_change":"2024-04-25T01:16:26.661702+0000","last_active":"2024-04-25T01:16:27.696564+0000","last_peered":"2024-04-25T01:16:27.696564+0000","last_clean":"2024-04-25T01:16:27.696564+0000","last_became_active":"2024-04-25T01:16:26.660914+0000","last_became_peered":"2024-04-25T01:16:26.660914+0000","last_unstale":"2024-04-25T01:16:27.696564+0000","last_undegraded":"2024-04-25T01:16:27.696564+0000","last_fullsized":"2024-04-25T01:16:27.696564+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:04:40.070035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1,3],"acting":[5,4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774509+0000","last_change":"2024-04-25T01:16:22.626033+0000","last_active":"2024-04-25T01:16:29.774509+0000","last_peered":"2024-04-25T01:16:29.774509+0000","last_clean":"2024-04-25T01:16:29.774509+0000","last_became_active":"2024-04-25T01:16:22.625646+0000","last_became_peered":"2024-04-25T01:16:22.625646+0000","last_unstale":"2024-04-25T01:16:29.774509+0000","last_undegraded":"2024-04-25T01:16:29.774509+0000","last_fullsized":"2024-04-25T01:16:29.774509+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:28.261961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.697212+0000","last_change":"2024-04-25T01:16:21.614628+0000","last_active":"2024-04-25T01:16:27.697212+0000","last_peered":"2024-04-25T01:16:27.697212+0000","last_clean":"2024-04-25T01:16:27.697212+0000","last_became_active":"2024-04-25T01:16:21.614110+0000","last_became_peered":"2024-04-25T01:16:21.614110+0000","last_unstale":"2024-04-25T01:16:27.697212+0000","last_undegraded":"2024-04-25T01:16:27.697212+0000","last_fullsized":"2024-04-25T01:16:27.697212+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:44.675184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.693414+0000","last_change":"2024-04-25T01:16:26.652814+0000","last_active":"2024-04-25T01:16:27.693414+0000","last_peered":"2024-04-25T01:16:27.693414+0000","last_clean":"2024-04-25T01:16:27.693414+0000","last_became_active":"2024-04-25T01:16:26.652030+0000","last_became_peered":"2024-04-25T01:16:26.652030+0000","last_unstale":"2024-04-25T01:16:27.693414+0000","last_undegraded":"2024-04-25T01:16:27.693414+0000","last_fullsized":"2024-04-25T01:16:27.693414+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:22.783625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.876518+0000","last_change":"2024-04-25T01:16:22.617280+0000","last_active":"2024-04-25T01:16:27.876518+0000","last_peered":"2024-04-25T01:16:27.876518+0000","last_clean":"2024-04-25T01:16:27.876518+0000","last_became_active":"2024-04-25T01:16:22.616985+0000","last_became_peered":"2024-04-25T01:16:22.616985+0000","last_unstale":"2024-04-25T01:16:27.876518+0000","last_undegraded":"2024-04-25T01:16:27.876518+0000","last_fullsized":"2024-04-25T01:16:27.876518+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:17:16.335206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.772458+0000","last_change":"2024-04-25T01:16:21.612374+0000","last_active":"2024-04-25T01:16:29.772458+0000","last_peered":"2024-04-25T01:16:29.772458+0000","last_clean":"2024-04-25T01:16:29.772458+0000","last_became_active":"2024-04-25T01:16:21.610765+0000","last_became_peered":"2024-04-25T01:16:21.610765+0000","last_unstale":"2024-04-25T01:16:29.772458+0000","last_undegraded":"2024-04-25T01:16:29.772458+0000","last_fullsized":"2024-04-25T01:16:29.772458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:17:59.668250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.971020+0000","last_change":"2024-04-25T01:16:26.655048+0000","last_active":"2024-04-25T01:16:27.971020+0000","last_peered":"2024-04-25T01:16:27.971020+0000","last_clean":"2024-04-25T01:16:27.971020+0000","last_became_active":"2024-04-25T01:16:26.654534+0000","last_became_peered":"2024-04-25T01:16:26.654534+0000","last_unstale":"2024-04-25T01:16:27.971020+0000","last_undegraded":"2024-04-25T01:16:27.971020+0000","last_fullsized":"2024-04-25T01:16:27.971020+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:17:20.386037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,2],"acting":[1,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.693580+0000","last_change":"2024-04-25T01:16:22.621853+0000","last_active":"2024-04-25T01:16:27.693580+0000","last_peered":"2024-04-25T01:16:27.693580+0000","last_clean":"2024-04-25T01:16:27.693580+0000","last_became_active":"2024-04-25T01:16:22.621537+0000","last_became_peered":"2024-04-25T01:16:22.621537+0000","last_unstale":"2024-04-25T01:16:27.693580+0000","last_undegraded":"2024-04-25T01:16:27.693580+0000","last_fullsized":"2024-04-25T01:16:27.693580+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:01.512241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694812+0000","last_change":"2024-04-25T01:16:21.607981+0000","last_active":"2024-04-25T01:16:27.694812+0000","last_peered":"2024-04-25T01:16:27.694812+0000","last_clean":"2024-04-25T01:16:27.694812+0000","last_became_active":"2024-04-25T01:16:21.607619+0000","last_became_peered":"2024-04-25T01:16:21.607619+0000","last_unstale":"2024-04-25T01:16:27.694812+0000","last_undegraded":"2024-04-25T01:16:27.694812+0000","last_fullsized":"2024-04-25T01:16:27.694812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:48:41.491267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699998+0000","last_change":"2024-04-25T01:16:26.664351+0000","last_active":"2024-04-25T01:16:27.699998+0000","last_peered":"2024-04-25T01:16:27.699998+0000","last_clean":"2024-04-25T01:16:27.699998+0000","last_became_active":"2024-04-25T01:16:26.663469+0000","last_became_peered":"2024-04-25T01:16:26.663469+0000","last_unstale":"2024-04-25T01:16:27.699998+0000","last_undegraded":"2024-04-25T01:16:27.699998+0000","last_fullsized":"2024-04-25T01:16:27.699998+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:10.152413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1,6],"acting":[5,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.970542+0000","last_change":"2024-04-25T01:16:22.609467+0000","last_active":"2024-04-25T01:16:27.970542+0000","last_peered":"2024-04-25T01:16:27.970542+0000","last_clean":"2024-04-25T01:16:27.970542+0000","last_became_active":"2024-04-25T01:16:22.609021+0000","last_became_peered":"2024-04-25T01:16:22.609021+0000","last_unstale":"2024-04-25T01:16:27.970542+0000","last_undegraded":"2024-04-25T01:16:27.970542+0000","last_fullsized":"2024-04-25T01:16:27.970542+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:45.689780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.772174+0000","last_change":"2024-04-25T01:16:21.618665+0000","last_active":"2024-04-25T01:16:29.772174+0000","last_peered":"2024-04-25T01:16:29.772174+0000","last_clean":"2024-04-25T01:16:29.772174+0000","last_became_active":"2024-04-25T01:16:21.618387+0000","last_became_peered":"2024-04-25T01:16:21.618387+0000","last_unstale":"2024-04-25T01:16:29.772174+0000","last_undegraded":"2024-04-25T01:16:29.772174+0000","last_fullsized":"2024-04-25T01:16:29.772174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:50.882927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699704+0000","last_change":"2024-04-25T01:16:26.676113+0000","last_active":"2024-04-25T01:16:27.699704+0000","last_peered":"2024-04-25T01:16:27.699704+0000","last_clean":"2024-04-25T01:16:27.699704+0000","last_became_active":"2024-04-25T01:16:26.675447+0000","last_became_peered":"2024-04-25T01:16:26.675447+0000","last_unstale":"2024-04-25T01:16:27.699704+0000","last_undegraded":"2024-04-25T01:16:27.699704+0000","last_fullsized":"2024-04-25T01:16:27.699704+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:07:54.641757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6,4],"acting":[5,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694284+0000","last_change":"2024-04-25T01:16:22.633079+0000","last_active":"2024-04-25T01:16:27.694284+0000","last_peered":"2024-04-25T01:16:27.694284+0000","last_clean":"2024-04-25T01:16:27.694284+0000","last_became_active":"2024-04-25T01:16:22.632813+0000","last_became_peered":"2024-04-25T01:16:22.632813+0000","last_unstale":"2024-04-25T01:16:27.694284+0000","last_undegraded":"2024-04-25T01:16:27.694284+0000","last_fullsized":"2024-04-25T01:16:27.694284+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:45:08.158420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695843+0000","last_change":"2024-04-25T01:16:21.610362+0000","last_active":"2024-04-25T01:16:27.695843+0000","last_peered":"2024-04-25T01:16:27.695843+0000","last_clean":"2024-04-25T01:16:27.695843+0000","last_became_active":"2024-04-25T01:16:21.608581+0000","last_became_peered":"2024-04-25T01:16:21.608581+0000","last_unstale":"2024-04-25T01:16:27.695843+0000","last_undegraded":"2024-04-25T01:16:27.695843+0000","last_fullsized":"2024-04-25T01:16:27.695843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:17:59.668250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.088195+0000","last_change":"2024-04-25T01:16:26.678631+0000","last_active":"2024-04-25T01:16:30.088195+0000","last_peered":"2024-04-25T01:16:30.088195+0000","last_clean":"2024-04-25T01:16:30.088195+0000","last_became_active":"2024-04-25T01:16:26.676876+0000","last_became_peered":"2024-04-25T01:16:26.676876+0000","last_unstale":"2024-04-25T01:16:30.088195+0000","last_undegraded":"2024-04-25T01:16:30.088195+0000","last_fullsized":"2024-04-25T01:16:30.088195+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:03:57.927312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0,7],"acting":[6,4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.087763+0000","last_change":"2024-04-25T01:16:22.627367+0000","last_active":"2024-04-25T01:16:30.087763+0000","last_peered":"2024-04-25T01:16:30.087763+0000","last_clean":"2024-04-25T01:16:30.087763+0000","last_became_active":"2024-04-25T01:16:22.626338+0000","last_became_peered":"2024-04-25T01:16:22.626338+0000","last_unstale":"2024-04-25T01:16:30.087763+0000","last_undegraded":"2024-04-25T01:16:30.087763+0000","last_fullsized":"2024-04-25T01:16:30.087763+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:50:15.794188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.224430+0000","last_change":"2024-04-25T01:16:21.613871+0000","last_active":"2024-04-25T01:16:28.224430+0000","last_peered":"2024-04-25T01:16:28.224430+0000","last_clean":"2024-04-25T01:16:28.224430+0000","last_became_active":"2024-04-25T01:16:21.613496+0000","last_became_peered":"2024-04-25T01:16:21.613496+0000","last_unstale":"2024-04-25T01:16:28.224430+0000","last_undegraded":"2024-04-25T01:16:28.224430+0000","last_fullsized":"2024-04-25T01:16:28.224430+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:37:47.297259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694895+0000","last_change":"2024-04-25T01:16:26.653921+0000","last_active":"2024-04-25T01:16:27.694895+0000","last_peered":"2024-04-25T01:16:27.694895+0000","last_clean":"2024-04-25T01:16:27.694895+0000","last_became_active":"2024-04-25T01:16:26.653438+0000","last_became_peered":"2024-04-25T01:16:26.653438+0000","last_unstale":"2024-04-25T01:16:27.694895+0000","last_undegraded":"2024-04-25T01:16:27.694895+0000","last_fullsized":"2024-04-25T01:16:27.694895+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:58:40.113666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5,6],"acting":[3,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774620+0000","last_change":"2024-04-25T01:16:22.631703+0000","last_active":"2024-04-25T01:16:29.774620+0000","last_peered":"2024-04-25T01:16:29.774620+0000","last_clean":"2024-04-25T01:16:29.774620+0000","last_became_active":"2024-04-25T01:16:22.631211+0000","last_became_peered":"2024-04-25T01:16:22.631211+0000","last_unstale":"2024-04-25T01:16:29.774620+0000","last_undegraded":"2024-04-25T01:16:29.774620+0000","last_fullsized":"2024-04-25T01:16:29.774620+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:01.512241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"26'10","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:41.501751+0000","last_change":"2024-04-25T01:16:21.614845+0000","last_active":"2024-04-25T01:16:41.501751+0000","last_peered":"2024-04-25T01:16:41.501751+0000","last_clean":"2024-04-25T01:16:41.501751+0000","last_became_active":"2024-04-25T01:16:21.614378+0000","last_became_peered":"2024-04-25T01:16:21.614378+0000","last_unstale":"2024-04-25T01:16:41.501751+0000","last_undegraded":"2024-04-25T01:16:41.501751+0000","last_fullsized":"2024-04-25T01:16:41.501751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:41:30.225989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695086+0000","last_change":"2024-04-25T01:16:26.676618+0000","last_active":"2024-04-25T01:16:27.695086+0000","last_peered":"2024-04-25T01:16:27.695086+0000","last_clean":"2024-04-25T01:16:27.695086+0000","last_became_active":"2024-04-25T01:16:26.667226+0000","last_became_peered":"2024-04-25T01:16:26.667226+0000","last_unstale":"2024-04-25T01:16:27.695086+0000","last_undegraded":"2024-04-25T01:16:27.695086+0000","last_fullsized":"2024-04-25T01:16:27.695086+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:23:19.810360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5,0],"acting":[4,3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.693968+0000","last_change":"2024-04-25T01:16:22.624757+0000","last_active":"2024-04-25T01:16:27.693968+0000","last_peered":"2024-04-25T01:16:27.693968+0000","last_clean":"2024-04-25T01:16:27.693968+0000","last_became_active":"2024-04-25T01:16:22.619992+0000","last_became_peered":"2024-04-25T01:16:22.619992+0000","last_unstale":"2024-04-25T01:16:27.693968+0000","last_undegraded":"2024-04-25T01:16:27.693968+0000","last_fullsized":"2024-04-25T01:16:27.693968+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:26:11.534238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.225000+0000","last_change":"2024-04-25T01:16:21.615700+0000","last_active":"2024-04-25T01:16:28.225000+0000","last_peered":"2024-04-25T01:16:28.225000+0000","last_clean":"2024-04-25T01:16:28.225000+0000","last_became_active":"2024-04-25T01:16:21.615207+0000","last_became_peered":"2024-04-25T01:16:21.615207+0000","last_unstale":"2024-04-25T01:16:28.225000+0000","last_undegraded":"2024-04-25T01:16:28.225000+0000","last_fullsized":"2024-04-25T01:16:28.225000+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:13.115669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.875349+0000","last_change":"2024-04-25T01:16:26.666133+0000","last_active":"2024-04-25T01:16:27.875349+0000","last_peered":"2024-04-25T01:16:27.875349+0000","last_clean":"2024-04-25T01:16:27.875349+0000","last_became_active":"2024-04-25T01:16:26.665359+0000","last_became_peered":"2024-04-25T01:16:26.665359+0000","last_unstale":"2024-04-25T01:16:27.875349+0000","last_undegraded":"2024-04-25T01:16:27.875349+0000","last_fullsized":"2024-04-25T01:16:27.875349+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:46:20.659273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1,7],"acting":[2,3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699852+0000","last_change":"2024-04-25T01:16:22.626495+0000","last_active":"2024-04-25T01:16:27.699852+0000","last_peered":"2024-04-25T01:16:27.699852+0000","last_clean":"2024-04-25T01:16:27.699852+0000","last_became_active":"2024-04-25T01:16:22.625913+0000","last_became_peered":"2024-04-25T01:16:22.625913+0000","last_unstale":"2024-04-25T01:16:27.699852+0000","last_undegraded":"2024-04-25T01:16:27.699852+0000","last_fullsized":"2024-04-25T01:16:27.699852+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:36.984987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.772343+0000","last_change":"2024-04-25T01:16:21.611655+0000","last_active":"2024-04-25T01:16:29.772343+0000","last_peered":"2024-04-25T01:16:29.772343+0000","last_clean":"2024-04-25T01:16:29.772343+0000","last_became_active":"2024-04-25T01:16:21.611388+0000","last_became_peered":"2024-04-25T01:16:21.611388+0000","last_unstale":"2024-04-25T01:16:29.772343+0000","last_undegraded":"2024-04-25T01:16:29.772343+0000","last_fullsized":"2024-04-25T01:16:29.772343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:39:20.330039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.223876+0000","last_change":"2024-04-25T01:16:26.668770+0000","last_active":"2024-04-25T01:16:28.223876+0000","last_peered":"2024-04-25T01:16:28.223876+0000","last_clean":"2024-04-25T01:16:28.223876+0000","last_became_active":"2024-04-25T01:16:26.668309+0000","last_became_peered":"2024-04-25T01:16:26.668309+0000","last_unstale":"2024-04-25T01:16:28.223876+0000","last_undegraded":"2024-04-25T01:16:28.223876+0000","last_fullsized":"2024-04-25T01:16:28.223876+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:17:20.386037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3,2],"acting":[7,1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.876399+0000","last_change":"2024-04-25T01:16:22.632717+0000","last_active":"2024-04-25T01:16:27.876399+0000","last_peered":"2024-04-25T01:16:27.876399+0000","last_clean":"2024-04-25T01:16:27.876399+0000","last_became_active":"2024-04-25T01:16:22.630174+0000","last_became_peered":"2024-04-25T01:16:22.630174+0000","last_unstale":"2024-04-25T01:16:27.876399+0000","last_undegraded":"2024-04-25T01:16:27.876399+0000","last_fullsized":"2024-04-25T01:16:27.876399+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:52.536070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.971972+0000","last_change":"2024-04-25T01:16:21.619013+0000","last_active":"2024-04-25T01:16:27.971972+0000","last_peered":"2024-04-25T01:16:27.971972+0000","last_clean":"2024-04-25T01:16:27.971972+0000","last_became_active":"2024-04-25T01:16:21.615903+0000","last_became_peered":"2024-04-25T01:16:21.615903+0000","last_unstale":"2024-04-25T01:16:27.971972+0000","last_undegraded":"2024-04-25T01:16:27.971972+0000","last_fullsized":"2024-04-25T01:16:27.971972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:37:47.297259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695313+0000","last_change":"2024-04-25T01:16:26.673940+0000","last_active":"2024-04-25T01:16:27.695313+0000","last_peered":"2024-04-25T01:16:27.695313+0000","last_clean":"2024-04-25T01:16:27.695313+0000","last_became_active":"2024-04-25T01:16:26.667111+0000","last_became_peered":"2024-04-25T01:16:26.667111+0000","last_unstale":"2024-04-25T01:16:27.695313+0000","last_undegraded":"2024-04-25T01:16:27.695313+0000","last_fullsized":"2024-04-25T01:16:27.695313+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:32.312792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3,7],"acting":[4,0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696886+0000","last_change":"2024-04-25T01:16:26.679002+0000","last_active":"2024-04-25T01:16:27.696886+0000","last_peered":"2024-04-25T01:16:27.696886+0000","last_clean":"2024-04-25T01:16:27.696886+0000","last_became_active":"2024-04-25T01:16:26.678398+0000","last_became_peered":"2024-04-25T01:16:26.678398+0000","last_unstale":"2024-04-25T01:16:27.696886+0000","last_undegraded":"2024-04-25T01:16:27.696886+0000","last_fullsized":"2024-04-25T01:16:27.696886+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:45:12.209250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5,3],"acting":[4,1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695447+0000","last_change":"2024-04-25T01:16:22.619871+0000","last_active":"2024-04-25T01:16:27.695447+0000","last_peered":"2024-04-25T01:16:27.695447+0000","last_clean":"2024-04-25T01:16:27.695447+0000","last_became_active":"2024-04-25T01:16:22.619553+0000","last_became_peered":"2024-04-25T01:16:22.619553+0000","last_unstale":"2024-04-25T01:16:27.695447+0000","last_undegraded":"2024-04-25T01:16:27.695447+0000","last_fullsized":"2024-04-25T01:16:27.695447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:46:33.270057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694865+0000","last_change":"2024-04-25T01:16:21.625865+0000","last_active":"2024-04-25T01:16:27.694865+0000","last_peered":"2024-04-25T01:16:27.694865+0000","last_clean":"2024-04-25T01:16:27.694865+0000","last_became_active":"2024-04-25T01:16:21.625566+0000","last_became_peered":"2024-04-25T01:16:21.625566+0000","last_unstale":"2024-04-25T01:16:27.694865+0000","last_undegraded":"2024-04-25T01:16:27.694865+0000","last_fullsized":"2024-04-25T01:16:27.694865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:44.675184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695078+0000","last_change":"2024-04-25T01:16:26.650705+0000","last_active":"2024-04-25T01:16:27.695078+0000","last_peered":"2024-04-25T01:16:27.695078+0000","last_clean":"2024-04-25T01:16:27.695078+0000","last_became_active":"2024-04-25T01:16:26.650332+0000","last_became_peered":"2024-04-25T01:16:26.650332+0000","last_unstale":"2024-04-25T01:16:27.695078+0000","last_undegraded":"2024-04-25T01:16:27.695078+0000","last_fullsized":"2024-04-25T01:16:27.695078+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:47:34.491567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,1],"acting":[3,2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694898+0000","last_change":"2024-04-25T01:16:22.622105+0000","last_active":"2024-04-25T01:16:27.694898+0000","last_peered":"2024-04-25T01:16:27.694898+0000","last_clean":"2024-04-25T01:16:27.694898+0000","last_became_active":"2024-04-25T01:16:22.621798+0000","last_became_peered":"2024-04-25T01:16:22.621798+0000","last_unstale":"2024-04-25T01:16:27.694898+0000","last_undegraded":"2024-04-25T01:16:27.694898+0000","last_fullsized":"2024-04-25T01:16:27.694898+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:47.769440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.087421+0000","last_change":"2024-04-25T01:16:21.621471+0000","last_active":"2024-04-25T01:16:30.087421+0000","last_peered":"2024-04-25T01:16:30.087421+0000","last_clean":"2024-04-25T01:16:30.087421+0000","last_became_active":"2024-04-25T01:16:21.621210+0000","last_became_peered":"2024-04-25T01:16:21.621210+0000","last_unstale":"2024-04-25T01:16:30.087421+0000","last_undegraded":"2024-04-25T01:16:30.087421+0000","last_fullsized":"2024-04-25T01:16:30.087421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:34.732380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695224+0000","last_change":"2024-04-25T01:16:21.619390+0000","last_active":"2024-04-25T01:16:27.695224+0000","last_peered":"2024-04-25T01:16:27.695224+0000","last_clean":"2024-04-25T01:16:27.695224+0000","last_became_active":"2024-04-25T01:16:21.619110+0000","last_became_peered":"2024-04-25T01:16:21.619110+0000","last_unstale":"2024-04-25T01:16:27.695224+0000","last_undegraded":"2024-04-25T01:16:27.695224+0000","last_fullsized":"2024-04-25T01:16:27.695224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:50.882927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696696+0000","last_change":"2024-04-25T01:16:22.638518+0000","last_active":"2024-04-25T01:16:27.696696+0000","last_peered":"2024-04-25T01:16:27.696696+0000","last_clean":"2024-04-25T01:16:27.696696+0000","last_became_active":"2024-04-25T01:16:22.638204+0000","last_became_peered":"2024-04-25T01:16:22.638204+0000","last_unstale":"2024-04-25T01:16:27.696696+0000","last_undegraded":"2024-04-25T01:16:27.696696+0000","last_fullsized":"2024-04-25T01:16:27.696696+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:06:58.102648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.697794+0000","last_change":"2024-04-25T01:16:26.673686+0000","last_active":"2024-04-25T01:16:27.697794+0000","last_peered":"2024-04-25T01:16:27.697794+0000","last_clean":"2024-04-25T01:16:27.697794+0000","last_became_active":"2024-04-25T01:16:26.673316+0000","last_became_peered":"2024-04-25T01:16:26.673316+0000","last_unstale":"2024-04-25T01:16:27.697794+0000","last_undegraded":"2024-04-25T01:16:27.697794+0000","last_fullsized":"2024-04-25T01:16:27.697794+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:26:15.585069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,7],"acting":[4,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696890+0000","last_change":"2024-04-25T01:16:21.623660+0000","last_active":"2024-04-25T01:16:27.696890+0000","last_peered":"2024-04-25T01:16:27.696890+0000","last_clean":"2024-04-25T01:16:27.696890+0000","last_became_active":"2024-04-25T01:16:21.623239+0000","last_became_peered":"2024-04-25T01:16:21.623239+0000","last_unstale":"2024-04-25T01:16:27.696890+0000","last_undegraded":"2024-04-25T01:16:27.696890+0000","last_fullsized":"2024-04-25T01:16:27.696890+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:10:00.414303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.970886+0000","last_change":"2024-04-25T01:16:22.606098+0000","last_active":"2024-04-25T01:16:27.970886+0000","last_peered":"2024-04-25T01:16:27.970886+0000","last_clean":"2024-04-25T01:16:27.970886+0000","last_became_active":"2024-04-25T01:16:22.605632+0000","last_became_peered":"2024-04-25T01:16:22.605632+0000","last_unstale":"2024-04-25T01:16:27.970886+0000","last_undegraded":"2024-04-25T01:16:27.970886+0000","last_fullsized":"2024-04-25T01:16:27.970886+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:18:00.682845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.224854+0000","last_change":"2024-04-25T01:16:26.667852+0000","last_active":"2024-04-25T01:16:28.224854+0000","last_peered":"2024-04-25T01:16:28.224854+0000","last_clean":"2024-04-25T01:16:28.224854+0000","last_became_active":"2024-04-25T01:16:26.667070+0000","last_became_peered":"2024-04-25T01:16:26.667070+0000","last_unstale":"2024-04-25T01:16:28.224854+0000","last_undegraded":"2024-04-25T01:16:28.224854+0000","last_fullsized":"2024-04-25T01:16:28.224854+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:35:37.472927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,0],"acting":[7,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.874985+0000","last_change":"2024-04-25T01:16:22.621505+0000","last_active":"2024-04-25T01:16:27.874985+0000","last_peered":"2024-04-25T01:16:27.874985+0000","last_clean":"2024-04-25T01:16:27.874985+0000","last_became_active":"2024-04-25T01:16:22.621239+0000","last_became_peered":"2024-04-25T01:16:22.621239+0000","last_unstale":"2024-04-25T01:16:27.874985+0000","last_undegraded":"2024-04-25T01:16:27.874985+0000","last_fullsized":"2024-04-25T01:16:27.874985+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:18:00.682845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.223851+0000","last_change":"2024-04-25T01:16:22.619015+0000","last_active":"2024-04-25T01:16:28.223851+0000","last_peered":"2024-04-25T01:16:28.223851+0000","last_clean":"2024-04-25T01:16:28.223851+0000","last_became_active":"2024-04-25T01:16:22.618739+0000","last_became_peered":"2024-04-25T01:16:22.618739+0000","last_unstale":"2024-04-25T01:16:28.223851+0000","last_undegraded":"2024-04-25T01:16:28.223851+0000","last_fullsized":"2024-04-25T01:16:28.223851+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:07.954692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694435+0000","last_change":"2024-04-25T01:16:22.621371+0000","last_active":"2024-04-25T01:16:27.694435+0000","last_peered":"2024-04-25T01:16:27.694435+0000","last_clean":"2024-04-25T01:16:27.694435+0000","last_became_active":"2024-04-25T01:16:22.621094+0000","last_became_peered":"2024-04-25T01:16:22.621094+0000","last_unstale":"2024-04-25T01:16:27.694435+0000","last_undegraded":"2024-04-25T01:16:27.694435+0000","last_fullsized":"2024-04-25T01:16:27.694435+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:23:15.759529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696710+0000","last_change":"2024-04-25T01:16:22.613471+0000","last_active":"2024-04-25T01:16:27.696710+0000","last_peered":"2024-04-25T01:16:27.696710+0000","last_clean":"2024-04-25T01:16:27.696710+0000","last_became_active":"2024-04-25T01:16:22.613036+0000","last_became_peered":"2024-04-25T01:16:22.613036+0000","last_unstale":"2024-04-25T01:16:27.696710+0000","last_undegraded":"2024-04-25T01:16:27.696710+0000","last_fullsized":"2024-04-25T01:16:27.696710+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:24:38.413314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.088877+0000","last_change":"2024-04-25T01:16:22.634780+0000","last_active":"2024-04-25T01:16:30.088877+0000","last_peered":"2024-04-25T01:16:30.088877+0000","last_clean":"2024-04-25T01:16:30.088877+0000","last_became_active":"2024-04-25T01:16:22.634412+0000","last_became_peered":"2024-04-25T01:16:22.634412+0000","last_unstale":"2024-04-25T01:16:30.088877+0000","last_undegraded":"2024-04-25T01:16:30.088877+0000","last_fullsized":"2024-04-25T01:16:30.088877+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:18:36.768908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.088852+0000","last_change":"2024-04-25T01:16:22.629001+0000","last_active":"2024-04-25T01:16:30.088852+0000","last_peered":"2024-04-25T01:16:30.088852+0000","last_clean":"2024-04-25T01:16:30.088852+0000","last_became_active":"2024-04-25T01:16:22.626649+0000","last_became_peered":"2024-04-25T01:16:22.626649+0000","last_unstale":"2024-04-25T01:16:30.088852+0000","last_undegraded":"2024-04-25T01:16:30.088852+0000","last_fullsized":"2024-04-25T01:16:30.088852+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:27:44.210712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.876273+0000","last_change":"2024-04-25T01:16:22.633405+0000","last_active":"2024-04-25T01:16:27.876273+0000","last_peered":"2024-04-25T01:16:27.876273+0000","last_clean":"2024-04-25T01:16:27.876273+0000","last_became_active":"2024-04-25T01:16:22.632539+0000","last_became_peered":"2024-04-25T01:16:22.632539+0000","last_unstale":"2024-04-25T01:16:27.876273+0000","last_undegraded":"2024-04-25T01:16:27.876273+0000","last_fullsized":"2024-04-25T01:16:27.876273+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:38:09.671586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699448+0000","last_change":"2024-04-25T01:16:22.615235+0000","last_active":"2024-04-25T01:16:27.699448+0000","last_peered":"2024-04-25T01:16:27.699448+0000","last_clean":"2024-04-25T01:16:27.699448+0000","last_became_active":"2024-04-25T01:16:22.614864+0000","last_became_peered":"2024-04-25T01:16:22.614864+0000","last_unstale":"2024-04-25T01:16:27.699448+0000","last_undegraded":"2024-04-25T01:16:27.699448+0000","last_fullsized":"2024-04-25T01:16:27.699448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:39:14.902973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.693959+0000","last_change":"2024-04-25T01:16:22.615333+0000","last_active":"2024-04-25T01:16:27.693959+0000","last_peered":"2024-04-25T01:16:27.693959+0000","last_clean":"2024-04-25T01:16:27.693959+0000","last_became_active":"2024-04-25T01:16:22.614510+0000","last_became_peered":"2024-04-25T01:16:22.614510+0000","last_unstale":"2024-04-25T01:16:27.693959+0000","last_undegraded":"2024-04-25T01:16:27.693959+0000","last_fullsized":"2024-04-25T01:16:27.693959+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:58:06.081060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696037+0000","last_change":"2024-04-25T01:16:22.615852+0000","last_active":"2024-04-25T01:16:27.696037+0000","last_peered":"2024-04-25T01:16:27.696037+0000","last_clean":"2024-04-25T01:16:27.696037+0000","last_became_active":"2024-04-25T01:16:22.615517+0000","last_became_peered":"2024-04-25T01:16:22.615517+0000","last_unstale":"2024-04-25T01:16:27.696037+0000","last_undegraded":"2024-04-25T01:16:27.696037+0000","last_fullsized":"2024-04-25T01:16:27.696037+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:31.459176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.970579+0000","last_change":"2024-04-25T01:16:22.627717+0000","last_active":"2024-04-25T01:16:27.970579+0000","last_peered":"2024-04-25T01:16:27.970579+0000","last_clean":"2024-04-25T01:16:27.970579+0000","last_became_active":"2024-04-25T01:16:22.627359+0000","last_became_peered":"2024-04-25T01:16:22.627359+0000","last_unstale":"2024-04-25T01:16:27.970579+0000","last_undegraded":"2024-04-25T01:16:27.970579+0000","last_fullsized":"2024-04-25T01:16:27.970579+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:51.897522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696423+0000","last_change":"2024-04-25T01:16:22.634253+0000","last_active":"2024-04-25T01:16:27.696423+0000","last_peered":"2024-04-25T01:16:27.696423+0000","last_clean":"2024-04-25T01:16:27.696423+0000","last_became_active":"2024-04-25T01:16:22.632643+0000","last_became_peered":"2024-04-25T01:16:22.632643+0000","last_unstale":"2024-04-25T01:16:27.696423+0000","last_undegraded":"2024-04-25T01:16:27.696423+0000","last_fullsized":"2024-04-25T01:16:27.696423+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:14:10.348493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774424+0000","last_change":"2024-04-25T01:16:22.627377+0000","last_active":"2024-04-25T01:16:29.774424+0000","last_peered":"2024-04-25T01:16:29.774424+0000","last_clean":"2024-04-25T01:16:29.774424+0000","last_became_active":"2024-04-25T01:16:22.626141+0000","last_became_peered":"2024-04-25T01:16:22.626141+0000","last_unstale":"2024-04-25T01:16:29.774424+0000","last_undegraded":"2024-04-25T01:16:29.774424+0000","last_fullsized":"2024-04-25T01:16:29.774424+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:58:33.743727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695050+0000","last_change":"2024-04-25T01:16:22.622416+0000","last_active":"2024-04-25T01:16:27.695050+0000","last_peered":"2024-04-25T01:16:27.695050+0000","last_clean":"2024-04-25T01:16:27.695050+0000","last_became_active":"2024-04-25T01:16:22.620963+0000","last_became_peered":"2024-04-25T01:16:22.620963+0000","last_unstale":"2024-04-25T01:16:27.695050+0000","last_undegraded":"2024-04-25T01:16:27.695050+0000","last_fullsized":"2024-04-25T01:16:27.695050+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:58:32.477986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.225310+0000","last_change":"2024-04-25T01:16:22.614902+0000","last_active":"2024-04-25T01:16:28.225310+0000","last_peered":"2024-04-25T01:16:28.225310+0000","last_clean":"2024-04-25T01:16:28.225310+0000","last_became_active":"2024-04-25T01:16:22.614621+0000","last_became_peered":"2024-04-25T01:16:22.614621+0000","last_unstale":"2024-04-25T01:16:28.225310+0000","last_undegraded":"2024-04-25T01:16:28.225310+0000","last_fullsized":"2024-04-25T01:16:28.225310+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:45.689780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695622+0000","last_change":"2024-04-25T01:16:22.637378+0000","last_active":"2024-04-25T01:16:27.695622+0000","last_peered":"2024-04-25T01:16:27.695622+0000","last_clean":"2024-04-25T01:16:27.695622+0000","last_became_active":"2024-04-25T01:16:22.636599+0000","last_became_peered":"2024-04-25T01:16:22.636599+0000","last_unstale":"2024-04-25T01:16:27.695622+0000","last_undegraded":"2024-04-25T01:16:27.695622+0000","last_fullsized":"2024-04-25T01:16:27.695622+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:44:52.172147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.693572+0000","last_change":"2024-04-25T01:16:22.621416+0000","last_active":"2024-04-25T01:16:27.693572+0000","last_peered":"2024-04-25T01:16:27.693572+0000","last_clean":"2024-04-25T01:16:27.693572+0000","last_became_active":"2024-04-25T01:16:22.619180+0000","last_became_peered":"2024-04-25T01:16:22.619180+0000","last_unstale":"2024-04-25T01:16:27.693572+0000","last_undegraded":"2024-04-25T01:16:27.693572+0000","last_fullsized":"2024-04-25T01:16:27.693572+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:46:16.608442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695613+0000","last_change":"2024-04-25T01:16:22.623812+0000","last_active":"2024-04-25T01:16:27.695613+0000","last_peered":"2024-04-25T01:16:27.695613+0000","last_clean":"2024-04-25T01:16:27.695613+0000","last_became_active":"2024-04-25T01:16:22.623468+0000","last_became_peered":"2024-04-25T01:16:22.623468+0000","last_unstale":"2024-04-25T01:16:27.695613+0000","last_undegraded":"2024-04-25T01:16:27.695613+0000","last_fullsized":"2024-04-25T01:16:27.695613+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:07:25.313465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.087421+0000","last_change":"2024-04-25T01:16:22.622280+0000","last_active":"2024-04-25T01:16:30.087421+0000","last_peered":"2024-04-25T01:16:30.087421+0000","last_clean":"2024-04-25T01:16:30.087421+0000","last_became_active":"2024-04-25T01:16:22.621901+0000","last_became_peered":"2024-04-25T01:16:22.621901+0000","last_unstale":"2024-04-25T01:16:30.087421+0000","last_undegraded":"2024-04-25T01:16:30.087421+0000","last_fullsized":"2024-04-25T01:16:30.087421+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:09:42.404928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.087951+0000","last_change":"2024-04-25T01:16:22.635778+0000","last_active":"2024-04-25T01:16:30.087951+0000","last_peered":"2024-04-25T01:16:30.087951+0000","last_clean":"2024-04-25T01:16:30.087951+0000","last_became_active":"2024-04-25T01:16:22.635492+0000","last_became_peered":"2024-04-25T01:16:22.635492+0000","last_unstale":"2024-04-25T01:16:30.087951+0000","last_undegraded":"2024-04-25T01:16:30.087951+0000","last_fullsized":"2024-04-25T01:16:30.087951+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:16:27.803006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774190+0000","last_change":"2024-04-25T01:16:22.627185+0000","last_active":"2024-04-25T01:16:29.774190+0000","last_peered":"2024-04-25T01:16:29.774190+0000","last_clean":"2024-04-25T01:16:29.774190+0000","last_became_active":"2024-04-25T01:16:22.626147+0000","last_became_peered":"2024-04-25T01:16:22.626147+0000","last_unstale":"2024-04-25T01:16:29.774190+0000","last_undegraded":"2024-04-25T01:16:29.774190+0000","last_fullsized":"2024-04-25T01:16:29.774190+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:06:31.257662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774396+0000","last_change":"2024-04-25T01:16:22.627705+0000","last_active":"2024-04-25T01:16:29.774396+0000","last_peered":"2024-04-25T01:16:29.774396+0000","last_clean":"2024-04-25T01:16:29.774396+0000","last_became_active":"2024-04-25T01:16:22.627424+0000","last_became_peered":"2024-04-25T01:16:22.627424+0000","last_unstale":"2024-04-25T01:16:29.774396+0000","last_undegraded":"2024-04-25T01:16:29.774396+0000","last_fullsized":"2024-04-25T01:16:29.774396+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:58:06.081060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774278+0000","last_change":"2024-04-25T01:16:22.625859+0000","last_active":"2024-04-25T01:16:29.774278+0000","last_peered":"2024-04-25T01:16:29.774278+0000","last_clean":"2024-04-25T01:16:29.774278+0000","last_became_active":"2024-04-25T01:16:22.625512+0000","last_became_peered":"2024-04-25T01:16:22.625512+0000","last_unstale":"2024-04-25T01:16:29.774278+0000","last_undegraded":"2024-04-25T01:16:29.774278+0000","last_fullsized":"2024-04-25T01:16:29.774278+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:09.176572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.971952+0000","last_change":"2024-04-25T01:16:22.622906+0000","last_active":"2024-04-25T01:16:27.971952+0000","last_peered":"2024-04-25T01:16:27.971952+0000","last_clean":"2024-04-25T01:16:27.971952+0000","last_became_active":"2024-04-25T01:16:22.622619+0000","last_became_peered":"2024-04-25T01:16:22.622619+0000","last_unstale":"2024-04-25T01:16:27.971952+0000","last_undegraded":"2024-04-25T01:16:27.971952+0000","last_fullsized":"2024-04-25T01:16:27.971952+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:20.635412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694511+0000","last_change":"2024-04-25T01:16:22.623806+0000","last_active":"2024-04-25T01:16:27.694511+0000","last_peered":"2024-04-25T01:16:27.694511+0000","last_clean":"2024-04-25T01:16:27.694511+0000","last_became_active":"2024-04-25T01:16:22.623165+0000","last_became_peered":"2024-04-25T01:16:22.623165+0000","last_unstale":"2024-04-25T01:16:27.694511+0000","last_undegraded":"2024-04-25T01:16:27.694511+0000","last_fullsized":"2024-04-25T01:16:27.694511+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:01:54.012020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.773234+0000","last_change":"2024-04-25T01:16:22.621330+0000","last_active":"2024-04-25T01:16:29.773234+0000","last_peered":"2024-04-25T01:16:29.773234+0000","last_clean":"2024-04-25T01:16:29.773234+0000","last_became_active":"2024-04-25T01:16:22.620993+0000","last_became_peered":"2024-04-25T01:16:22.620993+0000","last_unstale":"2024-04-25T01:16:29.773234+0000","last_undegraded":"2024-04-25T01:16:29.773234+0000","last_fullsized":"2024-04-25T01:16:29.773234+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:53.625025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.875525+0000","last_change":"2024-04-25T01:16:22.623636+0000","last_active":"2024-04-25T01:16:27.875525+0000","last_peered":"2024-04-25T01:16:27.875525+0000","last_clean":"2024-04-25T01:16:27.875525+0000","last_became_active":"2024-04-25T01:16:22.623365+0000","last_became_peered":"2024-04-25T01:16:22.623365+0000","last_unstale":"2024-04-25T01:16:27.875525+0000","last_undegraded":"2024-04-25T01:16:27.875525+0000","last_fullsized":"2024-04-25T01:16:27.875525+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:49:05.483350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.773871+0000","last_change":"2024-04-25T01:16:22.616635+0000","last_active":"2024-04-25T01:16:29.773871+0000","last_peered":"2024-04-25T01:16:29.773871+0000","last_clean":"2024-04-25T01:16:29.773871+0000","last_became_active":"2024-04-25T01:16:22.616342+0000","last_became_peered":"2024-04-25T01:16:22.616342+0000","last_unstale":"2024-04-25T01:16:29.773871+0000","last_undegraded":"2024-04-25T01:16:29.773871+0000","last_fullsized":"2024-04-25T01:16:29.773871+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:09:07.843928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.875763+0000","last_change":"2024-04-25T01:16:22.623875+0000","last_active":"2024-04-25T01:16:27.875763+0000","last_peered":"2024-04-25T01:16:27.875763+0000","last_clean":"2024-04-25T01:16:27.875763+0000","last_became_active":"2024-04-25T01:16:22.623524+0000","last_became_peered":"2024-04-25T01:16:22.623524+0000","last_unstale":"2024-04-25T01:16:27.875763+0000","last_undegraded":"2024-04-25T01:16:27.875763+0000","last_fullsized":"2024-04-25T01:16:27.875763+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:07:32.166451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.698853+0000","last_change":"2024-04-25T01:16:22.620609+0000","last_active":"2024-04-25T01:16:27.698853+0000","last_peered":"2024-04-25T01:16:27.698853+0000","last_clean":"2024-04-25T01:16:27.698853+0000","last_became_active":"2024-04-25T01:16:22.620261+0000","last_became_peered":"2024-04-25T01:16:22.620261+0000","last_unstale":"2024-04-25T01:16:27.698853+0000","last_undegraded":"2024-04-25T01:16:27.698853+0000","last_fullsized":"2024-04-25T01:16:27.698853+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:01.416632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699491+0000","last_change":"2024-04-25T01:16:22.633173+0000","last_active":"2024-04-25T01:16:27.699491+0000","last_peered":"2024-04-25T01:16:27.699491+0000","last_clean":"2024-04-25T01:16:27.699491+0000","last_became_active":"2024-04-25T01:16:22.632857+0000","last_became_peered":"2024-04-25T01:16:22.632857+0000","last_unstale":"2024-04-25T01:16:27.699491+0000","last_undegraded":"2024-04-25T01:16:27.699491+0000","last_fullsized":"2024-04-25T01:16:27.699491+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:14.238333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774391+0000","last_change":"2024-04-25T01:16:22.628348+0000","last_active":"2024-04-25T01:16:29.774391+0000","last_peered":"2024-04-25T01:16:29.774391+0000","last_clean":"2024-04-25T01:16:29.774391+0000","last_became_active":"2024-04-25T01:16:22.627597+0000","last_became_peered":"2024-04-25T01:16:22.627597+0000","last_unstale":"2024-04-25T01:16:29.774391+0000","last_undegraded":"2024-04-25T01:16:29.774391+0000","last_fullsized":"2024-04-25T01:16:29.774391+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:23:15.759529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542155,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":108288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542154,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":108164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542155,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":876544,"data_stored":698624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542154,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":294912,"data_stored":112678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542154,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":108746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542156,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":286720,"data_stored":112006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542154,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":872448,"data_stored":698602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542155,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":108220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:16:57.828 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:16:58.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.081+0000 7fead7400640 1 Processor -- start 2024-04-25T01:16:58.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.081+0000 7fead7400640 1 -- start start 2024-04-25T01:16:58.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.083+0000 7fead7400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fead006fcd0 0x7fead015ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:58.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.083+0000 7fead7400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fead006c840 0x7fead00b3d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:58.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.083+0000 7fead7400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fead00b3560 con 0x7fead00ade10 2024-04-25T01:16:58.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.083+0000 7fead7400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fead01589e0 con 0x7fead006fcd0 2024-04-25T01:16:58.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.083+0000 7fead7400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fead00b3890 con 0x7fead006c840 2024-04-25T01:16:58.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.083+0000 7fead4a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fead006fcd0 0x7fead015ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:58.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.083+0000 7fead4a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fead006fcd0 0x7fead015ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56322/0 (socket says 172.21.15.90:56322) 2024-04-25T01:16:58.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.083+0000 7fead4a00640 1 -- 172.21.15.90:0/4189751652 learned_addr learned my addr 172.21.15.90:0/4189751652 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:58.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.084+0000 7fead5400640 1 --2- 172.21.15.90:0/4189751652 >> v2:172.21.15.103:3300/0 conn(0x7fead006c840 0x7fead00b3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:58.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.084+0000 7fead4a00640 1 -- 172.21.15.90:0/4189751652 >> v2:172.21.15.103:3300/0 conn(0x7fead006c840 msgr2=0x7fead00b3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:58.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.084+0000 7fead4a00640 1 --2- 172.21.15.90:0/4189751652 >> v2:172.21.15.103:3300/0 conn(0x7fead006c840 0x7fead00b3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:58.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.084+0000 7fead4a00640 1 -- 172.21.15.90:0/4189751652 >> v1:172.21.15.103:6789/0 conn(0x7fead00ade10 legacy=0x7fead006c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:58.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.084+0000 7fead4a00640 1 -- 172.21.15.90:0/4189751652 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fead00ae510 con 0x7fead006fcd0 2024-04-25T01:16:58.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.084+0000 7fead5400640 1 --2- 172.21.15.90:0/4189751652 >> v2:172.21.15.103:3300/0 conn(0x7fead006c840 0x7fead00b3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:58.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.085+0000 7fead4a00640 1 --2- 172.21.15.90:0/4189751652 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead006fcd0 0x7fead015ef50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fead0067cb0 tx=0x7feac002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcce4458a09d222e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:58.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.085+0000 7feacfe00640 1 -- 172.21.15.90:0/4189751652 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac0037070 con 0x7fead006fcd0 2024-04-25T01:16:58.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.085+0000 7feacfe00640 1 -- 172.21.15.90:0/4189751652 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feac0032440 con 0x7fead006fcd0 2024-04-25T01:16:58.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.085+0000 7feacfe00640 1 -- 172.21.15.90:0/4189751652 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac0032810 con 0x7fead006fcd0 2024-04-25T01:16:58.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.086+0000 7fead7400640 1 -- 172.21.15.90:0/4189751652 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead006fcd0 msgr2=0x7fead015ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:58.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.086+0000 7fead7400640 1 --2- 172.21.15.90:0/4189751652 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead006fcd0 0x7fead015ef50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fead0067cb0 tx=0x7feac002a180 comp rx=0 tx=0).stop 2024-04-25T01:16:58.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.086+0000 7fead7400640 1 -- 172.21.15.90:0/4189751652 shutdown_connections 2024-04-25T01:16:58.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.086+0000 7fead7400640 1 --2- 172.21.15.90:0/4189751652 >> v2:172.21.15.103:3300/0 conn(0x7fead006c840 0x7fead00b3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:58.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.086+0000 7fead7400640 1 --2- 172.21.15.90:0/4189751652 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead006fcd0 0x7fead015ef50 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:58.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.086+0000 7fead7400640 1 -- 172.21.15.90:0/4189751652 >> 172.21.15.90:0/4189751652 conn(0x7fead0071550 msgr2=0x7fead0071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:58.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.086+0000 7fead7400640 1 -- 172.21.15.90:0/4189751652 shutdown_connections 2024-04-25T01:16:58.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.086+0000 7fead7400640 1 -- 172.21.15.90:0/4189751652 wait complete. 2024-04-25T01:16:58.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.087+0000 7fead7400640 1 Processor -- start 2024-04-25T01:16:58.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.087+0000 7fead7400640 1 -- start start 2024-04-25T01:16:58.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.088+0000 7fead7400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fead006c840 0x7fead01575d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:58.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.088+0000 7fead7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead006fcd0 0x7fead014f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:58.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.088+0000 7fead7400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fead00ade10 0x7fead01f79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:58.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.088+0000 7fead7400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fead00aead0 con 0x7fead006fcd0 2024-04-25T01:16:58.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.088+0000 7fead7400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fead00ae2c0 con 0x7fead00ade10 2024-04-25T01:16:58.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.088+0000 7fead7400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fead00b4260 con 0x7fead006c840 2024-04-25T01:16:58.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.089+0000 7feac7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead006fcd0 0x7fead014f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:58.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.089+0000 7feac7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead006fcd0 0x7fead014f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56334/0 (socket says 172.21.15.90:56334) 2024-04-25T01:16:58.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.089+0000 7feac7e00640 1 -- 172.21.15.90:0/1416945722 learned_addr learned my addr 172.21.15.90:0/1416945722 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:58.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.089+0000 7fead4a00640 1 --2- 172.21.15.90:0/1416945722 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fead006c840 0x7fead01575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:58.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.089+0000 7fead5400640 1 --2- 172.21.15.90:0/1416945722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fead00ade10 0x7fead01f79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:58.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.089+0000 7feac7e00640 1 -- 172.21.15.90:0/1416945722 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fead006c840 msgr2=0x7fead01575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:58.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.089+0000 7feac7e00640 1 --2- 172.21.15.90:0/1416945722 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fead006c840 0x7fead01575d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:58.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.089+0000 7feac7e00640 1 -- 172.21.15.90:0/1416945722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fead00ade10 msgr2=0x7fead01f79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:58.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.089+0000 7feac7e00640 1 --2- 172.21.15.90:0/1416945722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fead00ade10 0x7fead01f79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:58.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.089+0000 7feac7e00640 1 -- 172.21.15.90:0/1416945722 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fead014d510 con 0x7fead006fcd0 2024-04-25T01:16:58.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.089+0000 7feac7e00640 1 --2- 172.21.15.90:0/1416945722 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead006fcd0 0x7fead014f8a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7feab800b3e0 tx=0x7feab800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:58.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.090+0000 7feacd600640 1 -- 172.21.15.90:0/1416945722 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab8015020 con 0x7fead006fcd0 2024-04-25T01:16:58.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.090+0000 7feacd600640 1 -- 172.21.15.90:0/1416945722 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feab8011850 con 0x7fead006fcd0 2024-04-25T01:16:58.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.090+0000 7fead7400640 1 -- 172.21.15.90:0/1416945722 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fead014d7f0 con 0x7fead006fcd0 2024-04-25T01:16:58.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.090+0000 7feacd600640 1 -- 172.21.15.90:0/1416945722 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab8011c40 con 0x7fead006fcd0 2024-04-25T01:16:58.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.090+0000 7fead7400640 1 -- 172.21.15.90:0/1416945722 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fead014ddb0 con 0x7fead006fcd0 2024-04-25T01:16:58.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.091+0000 7fead7400640 1 -- 172.21.15.90:0/1416945722 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fead0068fb0 con 0x7fead006fcd0 2024-04-25T01:16:58.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.092+0000 7feacd600640 1 -- 172.21.15.90:0/1416945722 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7feab8018020 con 0x7fead006fcd0 2024-04-25T01:16:58.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.093+0000 7feacd600640 1 --2- 172.21.15.90:0/1416945722 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7feaac06be30 0x7feaac06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:58.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.093+0000 7feacd600640 1 -- 172.21.15.90:0/1416945722 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7feab808b1c0 con 0x7fead006fcd0 2024-04-25T01:16:58.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.093+0000 7fead4a00640 1 --2- 172.21.15.90:0/1416945722 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7feaac06be30 0x7feaac06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:58.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.094+0000 7fead4a00640 1 --2- 172.21.15.90:0/1416945722 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7feaac06be30 0x7feaac06e2f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7feac0034000 tx=0x7feac002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:58.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.097+0000 7feacd600640 1 -- 172.21.15.90:0/1416945722 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7feab801f020 con 0x7fead006fcd0 2024-04-25T01:16:58.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.282+0000 7fead7400640 1 -- 172.21.15.90:0/1416945722 --> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fead00675a0 con 0x7feaac06be30 2024-04-25T01:16:58.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.306+0000 7feacd600640 1 -- 172.21.15.90:0/1416945722 <== mgr.4104 v2:172.21.15.90:6832/2198086733 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332531 (secure 0 0 0) 0x7fead00675a0 con 0x7feaac06be30 2024-04-25T01:16:58.294 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:16:58.296 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-25T01:16:58.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.312+0000 7fead7400640 1 -- 172.21.15.90:0/1416945722 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7feaac06be30 msgr2=0x7feaac06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:58.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.312+0000 7fead7400640 1 --2- 172.21.15.90:0/1416945722 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7feaac06be30 0x7feaac06e2f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7feac0034000 tx=0x7feac002e000 comp rx=0 tx=0).stop 2024-04-25T01:16:58.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.312+0000 7fead7400640 1 -- 172.21.15.90:0/1416945722 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead006fcd0 msgr2=0x7fead014f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:58.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.312+0000 7fead7400640 1 --2- 172.21.15.90:0/1416945722 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead006fcd0 0x7fead014f8a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7feab800b3e0 tx=0x7feab800b8a0 comp rx=0 tx=0).stop 2024-04-25T01:16:58.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.312+0000 7fead7400640 1 -- 172.21.15.90:0/1416945722 shutdown_connections 2024-04-25T01:16:58.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.312+0000 7fead7400640 1 --2- 172.21.15.90:0/1416945722 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7feaac06be30 0x7feaac06e2f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:58.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.312+0000 7fead7400640 1 --2- 172.21.15.90:0/1416945722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fead00ade10 0x7fead01f79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:58.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.312+0000 7fead7400640 1 --2- 172.21.15.90:0/1416945722 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead006fcd0 0x7fead014f8a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:58.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.312+0000 7fead7400640 1 --2- 172.21.15.90:0/1416945722 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fead006c840 0x7fead01575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:58.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.312+0000 7fead7400640 1 -- 172.21.15.90:0/1416945722 >> 172.21.15.90:0/1416945722 conn(0x7fead0071550 msgr2=0x7fead0153f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:58.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.312+0000 7fead7400640 1 -- 172.21.15.90:0/1416945722 shutdown_connections 2024-04-25T01:16:58.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.312+0000 7fead7400640 1 -- 172.21.15.90:0/1416945722 wait complete. 2024-04-25T01:16:58.315 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-04-25T01:16:57.836420+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219536,"kb_used_data":3368,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755184,"statfs":{"total":773094113280,"available":772869308416,"internally_reserved":0,"allocated":3448832,"data_stored":2055328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005043"},"pg_stats":[{"pgid":"2.1f","version":"21'4","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.698700+0000","last_change":"2024-04-25T01:16:21.622879+0000","last_active":"2024-04-25T01:16:27.698700+0000","last_peered":"2024-04-25T01:16:27.698700+0000","last_clean":"2024-04-25T01:16:27.698700+0000","last_became_active":"2024-04-25T01:16:21.622586+0000","last_became_peered":"2024-04-25T01:16:21.622586+0000","last_unstale":"2024-04-25T01:16:27.698700+0000","last_undegraded":"2024-04-25T01:16:27.698700+0000","last_fullsized":"2024-04-25T01:16:27.698700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:50.882927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699904+0000","last_change":"2024-04-25T01:16:22.615827+0000","last_active":"2024-04-25T01:16:27.699904+0000","last_peered":"2024-04-25T01:16:27.699904+0000","last_clean":"2024-04-25T01:16:27.699904+0000","last_became_active":"2024-04-25T01:16:22.615497+0000","last_became_peered":"2024-04-25T01:16:22.615497+0000","last_unstale":"2024-04-25T01:16:27.699904+0000","last_undegraded":"2024-04-25T01:16:27.699904+0000","last_fullsized":"2024-04-25T01:16:27.699904+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:03:53.876481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.223962+0000","last_change":"2024-04-25T01:16:26.672437+0000","last_active":"2024-04-25T01:16:28.223962+0000","last_peered":"2024-04-25T01:16:28.223962+0000","last_clean":"2024-04-25T01:16:28.223962+0000","last_became_active":"2024-04-25T01:16:26.671999+0000","last_became_peered":"2024-04-25T01:16:26.671999+0000","last_unstale":"2024-04-25T01:16:28.223962+0000","last_undegraded":"2024-04-25T01:16:28.223962+0000","last_fullsized":"2024-04-25T01:16:28.223962+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:56.586900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4,1],"acting":[7,6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.700145+0000","last_change":"2024-04-25T01:16:21.609447+0000","last_active":"2024-04-25T01:16:27.700145+0000","last_peered":"2024-04-25T01:16:27.700145+0000","last_clean":"2024-04-25T01:16:27.700145+0000","last_became_active":"2024-04-25T01:16:21.608490+0000","last_became_peered":"2024-04-25T01:16:21.608490+0000","last_unstale":"2024-04-25T01:16:27.700145+0000","last_undegraded":"2024-04-25T01:16:27.700145+0000","last_fullsized":"2024-04-25T01:16:27.700145+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:15.007939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699632+0000","last_change":"2024-04-25T01:16:22.613891+0000","last_active":"2024-04-25T01:16:27.699632+0000","last_peered":"2024-04-25T01:16:27.699632+0000","last_clean":"2024-04-25T01:16:27.699632+0000","last_became_active":"2024-04-25T01:16:22.613471+0000","last_became_peered":"2024-04-25T01:16:22.613471+0000","last_unstale":"2024-04-25T01:16:27.699632+0000","last_undegraded":"2024-04-25T01:16:27.699632+0000","last_fullsized":"2024-04-25T01:16:27.699632+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:37:00.182534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.874736+0000","last_change":"2024-04-25T01:16:26.653705+0000","last_active":"2024-04-25T01:16:27.874736+0000","last_peered":"2024-04-25T01:16:27.874736+0000","last_clean":"2024-04-25T01:16:27.874736+0000","last_became_active":"2024-04-25T01:16:26.653348+0000","last_became_peered":"2024-04-25T01:16:26.653348+0000","last_unstale":"2024-04-25T01:16:27.874736+0000","last_undegraded":"2024-04-25T01:16:27.874736+0000","last_fullsized":"2024-04-25T01:16:27.874736+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:58:37.794558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4,6],"acting":[2,7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.772478+0000","last_change":"2024-04-25T01:16:21.611644+0000","last_active":"2024-04-25T01:16:29.772478+0000","last_peered":"2024-04-25T01:16:29.772478+0000","last_clean":"2024-04-25T01:16:29.772478+0000","last_became_active":"2024-04-25T01:16:21.611346+0000","last_became_peered":"2024-04-25T01:16:21.611346+0000","last_unstale":"2024-04-25T01:16:29.772478+0000","last_undegraded":"2024-04-25T01:16:29.772478+0000","last_fullsized":"2024-04-25T01:16:29.772478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:44.675184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.971313+0000","last_change":"2024-04-25T01:16:22.627961+0000","last_active":"2024-04-25T01:16:27.971313+0000","last_peered":"2024-04-25T01:16:27.971313+0000","last_clean":"2024-04-25T01:16:27.971313+0000","last_became_active":"2024-04-25T01:16:22.626767+0000","last_became_peered":"2024-04-25T01:16:22.626767+0000","last_unstale":"2024-04-25T01:16:27.971313+0000","last_undegraded":"2024-04-25T01:16:27.971313+0000","last_fullsized":"2024-04-25T01:16:27.971313+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:49:05.483350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.875764+0000","last_change":"2024-04-25T01:16:26.657547+0000","last_active":"2024-04-25T01:16:27.875764+0000","last_peered":"2024-04-25T01:16:27.875764+0000","last_clean":"2024-04-25T01:16:27.875764+0000","last_became_active":"2024-04-25T01:16:26.656643+0000","last_became_peered":"2024-04-25T01:16:26.656643+0000","last_unstale":"2024-04-25T01:16:27.875764+0000","last_undegraded":"2024-04-25T01:16:27.875764+0000","last_fullsized":"2024-04-25T01:16:27.875764+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:05.563072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3,6],"acting":[2,5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.772473+0000","last_change":"2024-04-25T01:16:21.617309+0000","last_active":"2024-04-25T01:16:29.772473+0000","last_peered":"2024-04-25T01:16:29.772473+0000","last_clean":"2024-04-25T01:16:29.772473+0000","last_became_active":"2024-04-25T01:16:21.617021+0000","last_became_peered":"2024-04-25T01:16:21.617021+0000","last_unstale":"2024-04-25T01:16:29.772473+0000","last_undegraded":"2024-04-25T01:16:29.772473+0000","last_fullsized":"2024-04-25T01:16:29.772473+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:06.940097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694375+0000","last_change":"2024-04-25T01:16:22.639034+0000","last_active":"2024-04-25T01:16:27.694375+0000","last_peered":"2024-04-25T01:16:27.694375+0000","last_clean":"2024-04-25T01:16:27.694375+0000","last_became_active":"2024-04-25T01:16:22.638661+0000","last_became_peered":"2024-04-25T01:16:22.638661+0000","last_unstale":"2024-04-25T01:16:27.694375+0000","last_undegraded":"2024-04-25T01:16:27.694375+0000","last_fullsized":"2024-04-25T01:16:27.694375+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:28.044482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.875219+0000","last_change":"2024-04-25T01:16:26.656419+0000","last_active":"2024-04-25T01:16:27.875219+0000","last_peered":"2024-04-25T01:16:27.875219+0000","last_clean":"2024-04-25T01:16:27.875219+0000","last_became_active":"2024-04-25T01:16:26.655743+0000","last_became_peered":"2024-04-25T01:16:26.655743+0000","last_unstale":"2024-04-25T01:16:27.875219+0000","last_undegraded":"2024-04-25T01:16:27.875219+0000","last_fullsized":"2024-04-25T01:16:27.875219+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:41:04.510055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5,7],"acting":[2,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694711+0000","last_change":"2024-04-25T01:16:22.621126+0000","last_active":"2024-04-25T01:16:27.694711+0000","last_peered":"2024-04-25T01:16:27.694711+0000","last_clean":"2024-04-25T01:16:27.694711+0000","last_became_active":"2024-04-25T01:16:22.620692+0000","last_became_peered":"2024-04-25T01:16:22.620692+0000","last_unstale":"2024-04-25T01:16:27.694711+0000","last_undegraded":"2024-04-25T01:16:27.694711+0000","last_fullsized":"2024-04-25T01:16:27.694711+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:28.261961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694834+0000","last_change":"2024-04-25T01:16:21.608564+0000","last_active":"2024-04-25T01:16:27.694834+0000","last_peered":"2024-04-25T01:16:27.694834+0000","last_clean":"2024-04-25T01:16:27.694834+0000","last_became_active":"2024-04-25T01:16:21.608227+0000","last_became_peered":"2024-04-25T01:16:21.608227+0000","last_unstale":"2024-04-25T01:16:27.694834+0000","last_undegraded":"2024-04-25T01:16:27.694834+0000","last_fullsized":"2024-04-25T01:16:27.694834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:38:08.656990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694160+0000","last_change":"2024-04-25T01:16:26.656167+0000","last_active":"2024-04-25T01:16:27.694160+0000","last_peered":"2024-04-25T01:16:27.694160+0000","last_clean":"2024-04-25T01:16:27.694160+0000","last_became_active":"2024-04-25T01:16:26.655643+0000","last_became_peered":"2024-04-25T01:16:26.655643+0000","last_unstale":"2024-04-25T01:16:27.694160+0000","last_undegraded":"2024-04-25T01:16:27.694160+0000","last_fullsized":"2024-04-25T01:16:27.694160+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:17:57.227839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,5],"acting":[3,1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.224130+0000","last_change":"2024-04-25T01:16:22.630130+0000","last_active":"2024-04-25T01:16:28.224130+0000","last_peered":"2024-04-25T01:16:28.224130+0000","last_clean":"2024-04-25T01:16:28.224130+0000","last_became_active":"2024-04-25T01:16:22.629829+0000","last_became_peered":"2024-04-25T01:16:22.629829+0000","last_unstale":"2024-04-25T01:16:28.224130+0000","last_undegraded":"2024-04-25T01:16:28.224130+0000","last_fullsized":"2024-04-25T01:16:28.224130+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:48:42.505862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.086752+0000","last_change":"2024-04-25T01:16:21.615603+0000","last_active":"2024-04-25T01:16:30.086752+0000","last_peered":"2024-04-25T01:16:30.086752+0000","last_clean":"2024-04-25T01:16:30.086752+0000","last_became_active":"2024-04-25T01:16:21.615217+0000","last_became_peered":"2024-04-25T01:16:21.615217+0000","last_unstale":"2024-04-25T01:16:30.086752+0000","last_undegraded":"2024-04-25T01:16:30.086752+0000","last_fullsized":"2024-04-25T01:16:30.086752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:10:00.414303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.773845+0000","last_change":"2024-04-25T01:16:26.677623+0000","last_active":"2024-04-25T01:16:29.773845+0000","last_peered":"2024-04-25T01:16:29.773845+0000","last_clean":"2024-04-25T01:16:29.773845+0000","last_became_active":"2024-04-25T01:16:26.677212+0000","last_became_peered":"2024-04-25T01:16:26.677212+0000","last_unstale":"2024-04-25T01:16:29.773845+0000","last_undegraded":"2024-04-25T01:16:29.773845+0000","last_fullsized":"2024-04-25T01:16:29.773845+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:58.601828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6,1],"acting":[0,4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694468+0000","last_change":"2024-04-25T01:16:22.617565+0000","last_active":"2024-04-25T01:16:27.694468+0000","last_peered":"2024-04-25T01:16:27.694468+0000","last_clean":"2024-04-25T01:16:27.694468+0000","last_became_active":"2024-04-25T01:16:22.617247+0000","last_became_peered":"2024-04-25T01:16:22.617247+0000","last_unstale":"2024-04-25T01:16:27.694468+0000","last_undegraded":"2024-04-25T01:16:27.694468+0000","last_fullsized":"2024-04-25T01:16:27.694468+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:00:17.366512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695722+0000","last_change":"2024-04-25T01:16:21.620221+0000","last_active":"2024-04-25T01:16:27.695722+0000","last_peered":"2024-04-25T01:16:27.695722+0000","last_clean":"2024-04-25T01:16:27.695722+0000","last_became_active":"2024-04-25T01:16:21.619923+0000","last_became_peered":"2024-04-25T01:16:21.619923+0000","last_unstale":"2024-04-25T01:16:27.695722+0000","last_undegraded":"2024-04-25T01:16:27.695722+0000","last_fullsized":"2024-04-25T01:16:27.695722+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:06.940097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.225195+0000","last_change":"2024-04-25T01:16:26.674824+0000","last_active":"2024-04-25T01:16:28.225195+0000","last_peered":"2024-04-25T01:16:28.225195+0000","last_clean":"2024-04-25T01:16:28.225195+0000","last_became_active":"2024-04-25T01:16:26.672608+0000","last_became_peered":"2024-04-25T01:16:26.672608+0000","last_unstale":"2024-04-25T01:16:28.225195+0000","last_undegraded":"2024-04-25T01:16:28.225195+0000","last_fullsized":"2024-04-25T01:16:28.225195+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:46:37.320887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2,3],"acting":[7,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.693656+0000","last_change":"2024-04-25T01:16:22.624853+0000","last_active":"2024-04-25T01:16:27.693656+0000","last_peered":"2024-04-25T01:16:27.693656+0000","last_clean":"2024-04-25T01:16:27.693656+0000","last_became_active":"2024-04-25T01:16:22.624275+0000","last_became_peered":"2024-04-25T01:16:22.624275+0000","last_unstale":"2024-04-25T01:16:27.693656+0000","last_undegraded":"2024-04-25T01:16:27.693656+0000","last_fullsized":"2024-04-25T01:16:27.693656+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:09.176572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.772092+0000","last_change":"2024-04-25T01:16:21.616953+0000","last_active":"2024-04-25T01:16:29.772092+0000","last_peered":"2024-04-25T01:16:29.772092+0000","last_clean":"2024-04-25T01:16:29.772092+0000","last_became_active":"2024-04-25T01:16:21.615656+0000","last_became_peered":"2024-04-25T01:16:21.615656+0000","last_unstale":"2024-04-25T01:16:29.772092+0000","last_undegraded":"2024-04-25T01:16:29.772092+0000","last_fullsized":"2024-04-25T01:16:29.772092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:48:41.491267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.876501+0000","last_change":"2024-04-25T01:16:26.661383+0000","last_active":"2024-04-25T01:16:27.876501+0000","last_peered":"2024-04-25T01:16:27.876501+0000","last_clean":"2024-04-25T01:16:27.876501+0000","last_became_active":"2024-04-25T01:16:26.658181+0000","last_became_peered":"2024-04-25T01:16:26.658181+0000","last_unstale":"2024-04-25T01:16:27.876501+0000","last_undegraded":"2024-04-25T01:16:27.876501+0000","last_fullsized":"2024-04-25T01:16:27.876501+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:23:19.810360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,7],"acting":[2,1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.697025+0000","last_change":"2024-04-25T01:16:22.618495+0000","last_active":"2024-04-25T01:16:27.697025+0000","last_peered":"2024-04-25T01:16:27.697025+0000","last_clean":"2024-04-25T01:16:27.697025+0000","last_became_active":"2024-04-25T01:16:22.618155+0000","last_became_peered":"2024-04-25T01:16:22.618155+0000","last_unstale":"2024-04-25T01:16:27.697025+0000","last_undegraded":"2024-04-25T01:16:27.697025+0000","last_fullsized":"2024-04-25T01:16:27.697025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:53.625025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695692+0000","last_change":"2024-04-25T01:16:21.618714+0000","last_active":"2024-04-25T01:16:27.695692+0000","last_peered":"2024-04-25T01:16:27.695692+0000","last_clean":"2024-04-25T01:16:27.695692+0000","last_became_active":"2024-04-25T01:16:21.618439+0000","last_became_peered":"2024-04-25T01:16:21.618439+0000","last_unstale":"2024-04-25T01:16:27.695692+0000","last_undegraded":"2024-04-25T01:16:27.695692+0000","last_fullsized":"2024-04-25T01:16:27.695692+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:17:15.320611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.876136+0000","last_change":"2024-04-25T01:16:26.649207+0000","last_active":"2024-04-25T01:16:27.876136+0000","last_peered":"2024-04-25T01:16:27.876136+0000","last_clean":"2024-04-25T01:16:27.876136+0000","last_became_active":"2024-04-25T01:16:26.648837+0000","last_became_peered":"2024-04-25T01:16:26.648837+0000","last_unstale":"2024-04-25T01:16:27.876136+0000","last_undegraded":"2024-04-25T01:16:27.876136+0000","last_fullsized":"2024-04-25T01:16:27.876136+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:13.227403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,1],"acting":[2,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.698475+0000","last_change":"2024-04-25T01:16:22.627735+0000","last_active":"2024-04-25T01:16:27.698475+0000","last_peered":"2024-04-25T01:16:27.698475+0000","last_clean":"2024-04-25T01:16:27.698475+0000","last_became_active":"2024-04-25T01:16:22.627430+0000","last_became_peered":"2024-04-25T01:16:22.627430+0000","last_unstale":"2024-04-25T01:16:27.698475+0000","last_undegraded":"2024-04-25T01:16:27.698475+0000","last_fullsized":"2024-04-25T01:16:27.698475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:06:31.257662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.697307+0000","last_change":"2024-04-25T01:16:21.620348+0000","last_active":"2024-04-25T01:16:27.697307+0000","last_peered":"2024-04-25T01:16:27.697307+0000","last_clean":"2024-04-25T01:16:27.697307+0000","last_became_active":"2024-04-25T01:16:21.620049+0000","last_became_peered":"2024-04-25T01:16:21.620049+0000","last_unstale":"2024-04-25T01:16:27.697307+0000","last_undegraded":"2024-04-25T01:16:27.697307+0000","last_fullsized":"2024-04-25T01:16:27.697307+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:17:59.668250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694701+0000","last_change":"2024-04-25T01:16:26.672455+0000","last_active":"2024-04-25T01:16:27.694701+0000","last_peered":"2024-04-25T01:16:27.694701+0000","last_clean":"2024-04-25T01:16:27.694701+0000","last_became_active":"2024-04-25T01:16:26.671952+0000","last_became_peered":"2024-04-25T01:16:26.671952+0000","last_unstale":"2024-04-25T01:16:27.694701+0000","last_undegraded":"2024-04-25T01:16:27.694701+0000","last_fullsized":"2024-04-25T01:16:27.694701+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:13.227403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6,3],"acting":[4,1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.088865+0000","last_change":"2024-04-25T01:16:22.628050+0000","last_active":"2024-04-25T01:16:30.088865+0000","last_peered":"2024-04-25T01:16:30.088865+0000","last_clean":"2024-04-25T01:16:30.088865+0000","last_became_active":"2024-04-25T01:16:22.626875+0000","last_became_peered":"2024-04-25T01:16:22.626875+0000","last_unstale":"2024-04-25T01:16:30.088865+0000","last_undegraded":"2024-04-25T01:16:30.088865+0000","last_fullsized":"2024-04-25T01:16:30.088865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:24:16.000792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695500+0000","last_change":"2024-04-25T01:16:21.621352+0000","last_active":"2024-04-25T01:16:27.695500+0000","last_peered":"2024-04-25T01:16:27.695500+0000","last_clean":"2024-04-25T01:16:27.695500+0000","last_became_active":"2024-04-25T01:16:21.620116+0000","last_became_peered":"2024-04-25T01:16:21.620116+0000","last_unstale":"2024-04-25T01:16:27.695500+0000","last_undegraded":"2024-04-25T01:16:27.695500+0000","last_fullsized":"2024-04-25T01:16:27.695500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:07:31.151855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.087941+0000","last_change":"2024-04-25T01:16:26.663371+0000","last_active":"2024-04-25T01:16:30.087941+0000","last_peered":"2024-04-25T01:16:30.087941+0000","last_clean":"2024-04-25T01:16:30.087941+0000","last_became_active":"2024-04-25T01:16:26.662972+0000","last_became_peered":"2024-04-25T01:16:26.662972+0000","last_unstale":"2024-04-25T01:16:30.087941+0000","last_undegraded":"2024-04-25T01:16:30.087941+0000","last_fullsized":"2024-04-25T01:16:30.087941+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:18.289163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2,7],"acting":[6,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.875472+0000","last_change":"2024-04-25T01:16:22.628735+0000","last_active":"2024-04-25T01:16:27.875472+0000","last_peered":"2024-04-25T01:16:27.875472+0000","last_clean":"2024-04-25T01:16:27.875472+0000","last_became_active":"2024-04-25T01:16:22.628429+0000","last_became_peered":"2024-04-25T01:16:22.628429+0000","last_unstale":"2024-04-25T01:16:27.875472+0000","last_undegraded":"2024-04-25T01:16:27.875472+0000","last_fullsized":"2024-04-25T01:16:27.875472+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:20.635412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.697942+0000","last_change":"2024-04-25T01:16:21.622977+0000","last_active":"2024-04-25T01:16:27.697942+0000","last_peered":"2024-04-25T01:16:27.697942+0000","last_clean":"2024-04-25T01:16:27.697942+0000","last_became_active":"2024-04-25T01:16:21.622298+0000","last_became_peered":"2024-04-25T01:16:21.622298+0000","last_unstale":"2024-04-25T01:16:27.697942+0000","last_undegraded":"2024-04-25T01:16:27.697942+0000","last_fullsized":"2024-04-25T01:16:27.697942+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:34.732380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.087427+0000","last_change":"2024-04-25T01:16:26.664207+0000","last_active":"2024-04-25T01:16:30.087427+0000","last_peered":"2024-04-25T01:16:30.087427+0000","last_clean":"2024-04-25T01:16:30.087427+0000","last_became_active":"2024-04-25T01:16:26.663749+0000","last_became_peered":"2024-04-25T01:16:26.663749+0000","last_unstale":"2024-04-25T01:16:30.087427+0000","last_undegraded":"2024-04-25T01:16:30.087427+0000","last_fullsized":"2024-04-25T01:16:30.087427+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:05.467463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5,7],"acting":[6,3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.773873+0000","last_change":"2024-04-25T01:16:22.635293+0000","last_active":"2024-04-25T01:16:29.773873+0000","last_peered":"2024-04-25T01:16:29.773873+0000","last_clean":"2024-04-25T01:16:29.773873+0000","last_became_active":"2024-04-25T01:16:22.635026+0000","last_became_peered":"2024-04-25T01:16:22.635026+0000","last_unstale":"2024-04-25T01:16:29.773873+0000","last_undegraded":"2024-04-25T01:16:29.773873+0000","last_fullsized":"2024-04-25T01:16:29.773873+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:46:16.608442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696109+0000","last_change":"2024-04-25T01:16:21.623621+0000","last_active":"2024-04-25T01:16:27.696109+0000","last_peered":"2024-04-25T01:16:27.696109+0000","last_clean":"2024-04-25T01:16:27.696109+0000","last_became_active":"2024-04-25T01:16:21.623351+0000","last_became_peered":"2024-04-25T01:16:21.623351+0000","last_unstale":"2024-04-25T01:16:27.696109+0000","last_undegraded":"2024-04-25T01:16:27.696109+0000","last_fullsized":"2024-04-25T01:16:27.696109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:49:04.468754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.088614+0000","last_change":"2024-04-25T01:16:26.656694+0000","last_active":"2024-04-25T01:16:30.088614+0000","last_peered":"2024-04-25T01:16:30.088614+0000","last_clean":"2024-04-25T01:16:30.088614+0000","last_became_active":"2024-04-25T01:16:26.656201+0000","last_became_peered":"2024-04-25T01:16:26.656201+0000","last_unstale":"2024-04-25T01:16:30.088614+0000","last_undegraded":"2024-04-25T01:16:30.088614+0000","last_fullsized":"2024-04-25T01:16:30.088614+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:39:18.953803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0,3],"acting":[6,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694802+0000","last_change":"2024-04-25T01:16:22.615076+0000","last_active":"2024-04-25T01:16:27.694802+0000","last_peered":"2024-04-25T01:16:27.694802+0000","last_clean":"2024-04-25T01:16:27.694802+0000","last_became_active":"2024-04-25T01:16:22.613139+0000","last_became_peered":"2024-04-25T01:16:22.613139+0000","last_unstale":"2024-04-25T01:16:27.694802+0000","last_undegraded":"2024-04-25T01:16:27.694802+0000","last_fullsized":"2024-04-25T01:16:27.694802+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:22:36.726844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.224133+0000","last_change":"2024-04-25T01:16:21.614492+0000","last_active":"2024-04-25T01:16:28.224133+0000","last_peered":"2024-04-25T01:16:28.224133+0000","last_clean":"2024-04-25T01:16:28.224133+0000","last_became_active":"2024-04-25T01:16:21.613490+0000","last_became_peered":"2024-04-25T01:16:21.613490+0000","last_unstale":"2024-04-25T01:16:28.224133+0000","last_undegraded":"2024-04-25T01:16:28.224133+0000","last_fullsized":"2024-04-25T01:16:28.224133+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:41:30.225989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.088458+0000","last_change":"2024-04-25T01:16:26.656061+0000","last_active":"2024-04-25T01:16:30.088458+0000","last_peered":"2024-04-25T01:16:30.088458+0000","last_clean":"2024-04-25T01:16:30.088458+0000","last_became_active":"2024-04-25T01:16:26.655189+0000","last_became_peered":"2024-04-25T01:16:26.655189+0000","last_unstale":"2024-04-25T01:16:30.088458+0000","last_undegraded":"2024-04-25T01:16:30.088458+0000","last_fullsized":"2024-04-25T01:16:30.088458+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:37:04.233365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,7],"acting":[6,0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.087313+0000","last_change":"2024-04-25T01:16:22.634160+0000","last_active":"2024-04-25T01:16:30.087313+0000","last_peered":"2024-04-25T01:16:30.087313+0000","last_clean":"2024-04-25T01:16:30.087313+0000","last_became_active":"2024-04-25T01:16:22.633856+0000","last_became_peered":"2024-04-25T01:16:22.633856+0000","last_unstale":"2024-04-25T01:16:30.087313+0000","last_undegraded":"2024-04-25T01:16:30.087313+0000","last_fullsized":"2024-04-25T01:16:30.087313+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:46:05.184032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695122+0000","last_change":"2024-04-25T01:16:21.617465+0000","last_active":"2024-04-25T01:16:27.695122+0000","last_peered":"2024-04-25T01:16:27.695122+0000","last_clean":"2024-04-25T01:16:27.695122+0000","last_became_active":"2024-04-25T01:16:21.617201+0000","last_became_peered":"2024-04-25T01:16:21.617201+0000","last_unstale":"2024-04-25T01:16:27.695122+0000","last_undegraded":"2024-04-25T01:16:27.695122+0000","last_fullsized":"2024-04-25T01:16:27.695122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:48:41.491267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699450+0000","last_change":"2024-04-25T01:16:26.666150+0000","last_active":"2024-04-25T01:16:27.699450+0000","last_peered":"2024-04-25T01:16:27.699450+0000","last_clean":"2024-04-25T01:16:27.699450+0000","last_became_active":"2024-04-25T01:16:26.665620+0000","last_became_peered":"2024-04-25T01:16:26.665620+0000","last_unstale":"2024-04-25T01:16:27.699450+0000","last_undegraded":"2024-04-25T01:16:27.699450+0000","last_fullsized":"2024-04-25T01:16:27.699450+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:51:23.702657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6,1],"acting":[5,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695397+0000","last_change":"2024-04-25T01:16:22.637692+0000","last_active":"2024-04-25T01:16:27.695397+0000","last_peered":"2024-04-25T01:16:27.695397+0000","last_clean":"2024-04-25T01:16:27.695397+0000","last_became_active":"2024-04-25T01:16:22.635225+0000","last_became_peered":"2024-04-25T01:16:22.635225+0000","last_unstale":"2024-04-25T01:16:27.695397+0000","last_undegraded":"2024-04-25T01:16:27.695397+0000","last_fullsized":"2024-04-25T01:16:27.695397+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:11.514022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695136+0000","last_change":"2024-04-25T01:16:21.608226+0000","last_active":"2024-04-25T01:16:27.695136+0000","last_peered":"2024-04-25T01:16:27.695136+0000","last_clean":"2024-04-25T01:16:27.695136+0000","last_became_active":"2024-04-25T01:16:21.607851+0000","last_became_peered":"2024-04-25T01:16:21.607851+0000","last_unstale":"2024-04-25T01:16:27.695136+0000","last_undegraded":"2024-04-25T01:16:27.695136+0000","last_fullsized":"2024-04-25T01:16:27.695136+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:51.521474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":178,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.970532+0000","last_change":"2024-04-25T01:16:17.466569+0000","last_active":"2024-04-25T01:16:27.970532+0000","last_peered":"2024-04-25T01:16:27.970532+0000","last_clean":"2024-04-25T01:16:27.970532+0000","last_became_active":"2024-04-25T01:16:17.465687+0000","last_became_peered":"2024-04-25T01:16:17.465687+0000","last_unstale":"2024-04-25T01:16:27.970532+0000","last_undegraded":"2024-04-25T01:16:27.970532+0000","last_fullsized":"2024-04-25T01:16:27.970532+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:16.440559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:16.440559+0000","last_clean_scrub_stamp":"2024-04-25T01:16:16.440559+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:18:17.493555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695469+0000","last_change":"2024-04-25T01:16:26.646040+0000","last_active":"2024-04-25T01:16:27.695469+0000","last_peered":"2024-04-25T01:16:27.695469+0000","last_clean":"2024-04-25T01:16:27.695469+0000","last_became_active":"2024-04-25T01:16:26.645500+0000","last_became_peered":"2024-04-25T01:16:26.645500+0000","last_unstale":"2024-04-25T01:16:27.695469+0000","last_undegraded":"2024-04-25T01:16:27.695469+0000","last_fullsized":"2024-04-25T01:16:27.695469+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:01:53.032237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2,0],"acting":[3,1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694653+0000","last_change":"2024-04-25T01:16:22.631511+0000","last_active":"2024-04-25T01:16:27.694653+0000","last_peered":"2024-04-25T01:16:27.694653+0000","last_clean":"2024-04-25T01:16:27.694653+0000","last_became_active":"2024-04-25T01:16:22.631169+0000","last_became_peered":"2024-04-25T01:16:22.631169+0000","last_unstale":"2024-04-25T01:16:27.694653+0000","last_undegraded":"2024-04-25T01:16:27.694653+0000","last_fullsized":"2024-04-25T01:16:27.694653+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:54.550997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.089006+0000","last_change":"2024-04-25T01:16:21.621474+0000","last_active":"2024-04-25T01:16:30.089006+0000","last_peered":"2024-04-25T01:16:30.089006+0000","last_clean":"2024-04-25T01:16:30.089006+0000","last_became_active":"2024-04-25T01:16:21.621139+0000","last_became_peered":"2024-04-25T01:16:21.621139+0000","last_unstale":"2024-04-25T01:16:30.089006+0000","last_undegraded":"2024-04-25T01:16:30.089006+0000","last_fullsized":"2024-04-25T01:16:30.089006+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:35:01.136892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.875567+0000","last_change":"2024-04-25T01:16:26.669916+0000","last_active":"2024-04-25T01:16:27.875567+0000","last_peered":"2024-04-25T01:16:27.875567+0000","last_clean":"2024-04-25T01:16:27.875567+0000","last_became_active":"2024-04-25T01:16:26.668054+0000","last_became_peered":"2024-04-25T01:16:26.668054+0000","last_unstale":"2024-04-25T01:16:27.875567+0000","last_undegraded":"2024-04-25T01:16:27.875567+0000","last_fullsized":"2024-04-25T01:16:27.875567+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:58:10.131890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7,4],"acting":[2,3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774586+0000","last_change":"2024-04-25T01:16:22.617183+0000","last_active":"2024-04-25T01:16:29.774586+0000","last_peered":"2024-04-25T01:16:29.774586+0000","last_clean":"2024-04-25T01:16:29.774586+0000","last_became_active":"2024-04-25T01:16:22.615748+0000","last_became_peered":"2024-04-25T01:16:22.615748+0000","last_unstale":"2024-04-25T01:16:29.774586+0000","last_undegraded":"2024-04-25T01:16:29.774586+0000","last_fullsized":"2024-04-25T01:16:29.774586+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:41:00.459225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699060+0000","last_change":"2024-04-25T01:16:21.617685+0000","last_active":"2024-04-25T01:16:27.699060+0000","last_peered":"2024-04-25T01:16:27.699060+0000","last_clean":"2024-04-25T01:16:27.699060+0000","last_became_active":"2024-04-25T01:16:21.617209+0000","last_became_peered":"2024-04-25T01:16:21.617209+0000","last_unstale":"2024-04-25T01:16:27.699060+0000","last_undegraded":"2024-04-25T01:16:27.699060+0000","last_fullsized":"2024-04-25T01:16:27.699060+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:34:39.714953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696564+0000","last_change":"2024-04-25T01:16:26.661702+0000","last_active":"2024-04-25T01:16:27.696564+0000","last_peered":"2024-04-25T01:16:27.696564+0000","last_clean":"2024-04-25T01:16:27.696564+0000","last_became_active":"2024-04-25T01:16:26.660914+0000","last_became_peered":"2024-04-25T01:16:26.660914+0000","last_unstale":"2024-04-25T01:16:27.696564+0000","last_undegraded":"2024-04-25T01:16:27.696564+0000","last_fullsized":"2024-04-25T01:16:27.696564+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:04:40.070035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1,3],"acting":[5,4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774509+0000","last_change":"2024-04-25T01:16:22.626033+0000","last_active":"2024-04-25T01:16:29.774509+0000","last_peered":"2024-04-25T01:16:29.774509+0000","last_clean":"2024-04-25T01:16:29.774509+0000","last_became_active":"2024-04-25T01:16:22.625646+0000","last_became_peered":"2024-04-25T01:16:22.625646+0000","last_unstale":"2024-04-25T01:16:29.774509+0000","last_undegraded":"2024-04-25T01:16:29.774509+0000","last_fullsized":"2024-04-25T01:16:29.774509+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:28.261961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.697212+0000","last_change":"2024-04-25T01:16:21.614628+0000","last_active":"2024-04-25T01:16:27.697212+0000","last_peered":"2024-04-25T01:16:27.697212+0000","last_clean":"2024-04-25T01:16:27.697212+0000","last_became_active":"2024-04-25T01:16:21.614110+0000","last_became_peered":"2024-04-25T01:16:21.614110+0000","last_unstale":"2024-04-25T01:16:27.697212+0000","last_undegraded":"2024-04-25T01:16:27.697212+0000","last_fullsized":"2024-04-25T01:16:27.697212+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:44.675184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.693414+0000","last_change":"2024-04-25T01:16:26.652814+0000","last_active":"2024-04-25T01:16:27.693414+0000","last_peered":"2024-04-25T01:16:27.693414+0000","last_clean":"2024-04-25T01:16:27.693414+0000","last_became_active":"2024-04-25T01:16:26.652030+0000","last_became_peered":"2024-04-25T01:16:26.652030+0000","last_unstale":"2024-04-25T01:16:27.693414+0000","last_undegraded":"2024-04-25T01:16:27.693414+0000","last_fullsized":"2024-04-25T01:16:27.693414+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:22.783625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.876518+0000","last_change":"2024-04-25T01:16:22.617280+0000","last_active":"2024-04-25T01:16:27.876518+0000","last_peered":"2024-04-25T01:16:27.876518+0000","last_clean":"2024-04-25T01:16:27.876518+0000","last_became_active":"2024-04-25T01:16:22.616985+0000","last_became_peered":"2024-04-25T01:16:22.616985+0000","last_unstale":"2024-04-25T01:16:27.876518+0000","last_undegraded":"2024-04-25T01:16:27.876518+0000","last_fullsized":"2024-04-25T01:16:27.876518+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:17:16.335206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.772458+0000","last_change":"2024-04-25T01:16:21.612374+0000","last_active":"2024-04-25T01:16:29.772458+0000","last_peered":"2024-04-25T01:16:29.772458+0000","last_clean":"2024-04-25T01:16:29.772458+0000","last_became_active":"2024-04-25T01:16:21.610765+0000","last_became_peered":"2024-04-25T01:16:21.610765+0000","last_unstale":"2024-04-25T01:16:29.772458+0000","last_undegraded":"2024-04-25T01:16:29.772458+0000","last_fullsized":"2024-04-25T01:16:29.772458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:17:59.668250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.971020+0000","last_change":"2024-04-25T01:16:26.655048+0000","last_active":"2024-04-25T01:16:27.971020+0000","last_peered":"2024-04-25T01:16:27.971020+0000","last_clean":"2024-04-25T01:16:27.971020+0000","last_became_active":"2024-04-25T01:16:26.654534+0000","last_became_peered":"2024-04-25T01:16:26.654534+0000","last_unstale":"2024-04-25T01:16:27.971020+0000","last_undegraded":"2024-04-25T01:16:27.971020+0000","last_fullsized":"2024-04-25T01:16:27.971020+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:17:20.386037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,2],"acting":[1,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.693580+0000","last_change":"2024-04-25T01:16:22.621853+0000","last_active":"2024-04-25T01:16:27.693580+0000","last_peered":"2024-04-25T01:16:27.693580+0000","last_clean":"2024-04-25T01:16:27.693580+0000","last_became_active":"2024-04-25T01:16:22.621537+0000","last_became_peered":"2024-04-25T01:16:22.621537+0000","last_unstale":"2024-04-25T01:16:27.693580+0000","last_undegraded":"2024-04-25T01:16:27.693580+0000","last_fullsized":"2024-04-25T01:16:27.693580+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:01.512241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694812+0000","last_change":"2024-04-25T01:16:21.607981+0000","last_active":"2024-04-25T01:16:27.694812+0000","last_peered":"2024-04-25T01:16:27.694812+0000","last_clean":"2024-04-25T01:16:27.694812+0000","last_became_active":"2024-04-25T01:16:21.607619+0000","last_became_peered":"2024-04-25T01:16:21.607619+0000","last_unstale":"2024-04-25T01:16:27.694812+0000","last_undegraded":"2024-04-25T01:16:27.694812+0000","last_fullsized":"2024-04-25T01:16:27.694812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:48:41.491267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699998+0000","last_change":"2024-04-25T01:16:26.664351+0000","last_active":"2024-04-25T01:16:27.699998+0000","last_peered":"2024-04-25T01:16:27.699998+0000","last_clean":"2024-04-25T01:16:27.699998+0000","last_became_active":"2024-04-25T01:16:26.663469+0000","last_became_peered":"2024-04-25T01:16:26.663469+0000","last_unstale":"2024-04-25T01:16:27.699998+0000","last_undegraded":"2024-04-25T01:16:27.699998+0000","last_fullsized":"2024-04-25T01:16:27.699998+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:10.152413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1,6],"acting":[5,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.970542+0000","last_change":"2024-04-25T01:16:22.609467+0000","last_active":"2024-04-25T01:16:27.970542+0000","last_peered":"2024-04-25T01:16:27.970542+0000","last_clean":"2024-04-25T01:16:27.970542+0000","last_became_active":"2024-04-25T01:16:22.609021+0000","last_became_peered":"2024-04-25T01:16:22.609021+0000","last_unstale":"2024-04-25T01:16:27.970542+0000","last_undegraded":"2024-04-25T01:16:27.970542+0000","last_fullsized":"2024-04-25T01:16:27.970542+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:45.689780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.772174+0000","last_change":"2024-04-25T01:16:21.618665+0000","last_active":"2024-04-25T01:16:29.772174+0000","last_peered":"2024-04-25T01:16:29.772174+0000","last_clean":"2024-04-25T01:16:29.772174+0000","last_became_active":"2024-04-25T01:16:21.618387+0000","last_became_peered":"2024-04-25T01:16:21.618387+0000","last_unstale":"2024-04-25T01:16:29.772174+0000","last_undegraded":"2024-04-25T01:16:29.772174+0000","last_fullsized":"2024-04-25T01:16:29.772174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:50.882927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699704+0000","last_change":"2024-04-25T01:16:26.676113+0000","last_active":"2024-04-25T01:16:27.699704+0000","last_peered":"2024-04-25T01:16:27.699704+0000","last_clean":"2024-04-25T01:16:27.699704+0000","last_became_active":"2024-04-25T01:16:26.675447+0000","last_became_peered":"2024-04-25T01:16:26.675447+0000","last_unstale":"2024-04-25T01:16:27.699704+0000","last_undegraded":"2024-04-25T01:16:27.699704+0000","last_fullsized":"2024-04-25T01:16:27.699704+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:07:54.641757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6,4],"acting":[5,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694284+0000","last_change":"2024-04-25T01:16:22.633079+0000","last_active":"2024-04-25T01:16:27.694284+0000","last_peered":"2024-04-25T01:16:27.694284+0000","last_clean":"2024-04-25T01:16:27.694284+0000","last_became_active":"2024-04-25T01:16:22.632813+0000","last_became_peered":"2024-04-25T01:16:22.632813+0000","last_unstale":"2024-04-25T01:16:27.694284+0000","last_undegraded":"2024-04-25T01:16:27.694284+0000","last_fullsized":"2024-04-25T01:16:27.694284+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:45:08.158420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695843+0000","last_change":"2024-04-25T01:16:21.610362+0000","last_active":"2024-04-25T01:16:27.695843+0000","last_peered":"2024-04-25T01:16:27.695843+0000","last_clean":"2024-04-25T01:16:27.695843+0000","last_became_active":"2024-04-25T01:16:21.608581+0000","last_became_peered":"2024-04-25T01:16:21.608581+0000","last_unstale":"2024-04-25T01:16:27.695843+0000","last_undegraded":"2024-04-25T01:16:27.695843+0000","last_fullsized":"2024-04-25T01:16:27.695843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:17:59.668250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.088195+0000","last_change":"2024-04-25T01:16:26.678631+0000","last_active":"2024-04-25T01:16:30.088195+0000","last_peered":"2024-04-25T01:16:30.088195+0000","last_clean":"2024-04-25T01:16:30.088195+0000","last_became_active":"2024-04-25T01:16:26.676876+0000","last_became_peered":"2024-04-25T01:16:26.676876+0000","last_unstale":"2024-04-25T01:16:30.088195+0000","last_undegraded":"2024-04-25T01:16:30.088195+0000","last_fullsized":"2024-04-25T01:16:30.088195+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:03:57.927312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0,7],"acting":[6,4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.087763+0000","last_change":"2024-04-25T01:16:22.627367+0000","last_active":"2024-04-25T01:16:30.087763+0000","last_peered":"2024-04-25T01:16:30.087763+0000","last_clean":"2024-04-25T01:16:30.087763+0000","last_became_active":"2024-04-25T01:16:22.626338+0000","last_became_peered":"2024-04-25T01:16:22.626338+0000","last_unstale":"2024-04-25T01:16:30.087763+0000","last_undegraded":"2024-04-25T01:16:30.087763+0000","last_fullsized":"2024-04-25T01:16:30.087763+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:50:15.794188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.224430+0000","last_change":"2024-04-25T01:16:21.613871+0000","last_active":"2024-04-25T01:16:28.224430+0000","last_peered":"2024-04-25T01:16:28.224430+0000","last_clean":"2024-04-25T01:16:28.224430+0000","last_became_active":"2024-04-25T01:16:21.613496+0000","last_became_peered":"2024-04-25T01:16:21.613496+0000","last_unstale":"2024-04-25T01:16:28.224430+0000","last_undegraded":"2024-04-25T01:16:28.224430+0000","last_fullsized":"2024-04-25T01:16:28.224430+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:37:47.297259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694895+0000","last_change":"2024-04-25T01:16:26.653921+0000","last_active":"2024-04-25T01:16:27.694895+0000","last_peered":"2024-04-25T01:16:27.694895+0000","last_clean":"2024-04-25T01:16:27.694895+0000","last_became_active":"2024-04-25T01:16:26.653438+0000","last_became_peered":"2024-04-25T01:16:26.653438+0000","last_unstale":"2024-04-25T01:16:27.694895+0000","last_undegraded":"2024-04-25T01:16:27.694895+0000","last_fullsized":"2024-04-25T01:16:27.694895+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:58:40.113666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5,6],"acting":[3,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774620+0000","last_change":"2024-04-25T01:16:22.631703+0000","last_active":"2024-04-25T01:16:29.774620+0000","last_peered":"2024-04-25T01:16:29.774620+0000","last_clean":"2024-04-25T01:16:29.774620+0000","last_became_active":"2024-04-25T01:16:22.631211+0000","last_became_peered":"2024-04-25T01:16:22.631211+0000","last_unstale":"2024-04-25T01:16:29.774620+0000","last_undegraded":"2024-04-25T01:16:29.774620+0000","last_fullsized":"2024-04-25T01:16:29.774620+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:01.512241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"26'10","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:41.501751+0000","last_change":"2024-04-25T01:16:21.614845+0000","last_active":"2024-04-25T01:16:41.501751+0000","last_peered":"2024-04-25T01:16:41.501751+0000","last_clean":"2024-04-25T01:16:41.501751+0000","last_became_active":"2024-04-25T01:16:21.614378+0000","last_became_peered":"2024-04-25T01:16:21.614378+0000","last_unstale":"2024-04-25T01:16:41.501751+0000","last_undegraded":"2024-04-25T01:16:41.501751+0000","last_fullsized":"2024-04-25T01:16:41.501751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:41:30.225989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695086+0000","last_change":"2024-04-25T01:16:26.676618+0000","last_active":"2024-04-25T01:16:27.695086+0000","last_peered":"2024-04-25T01:16:27.695086+0000","last_clean":"2024-04-25T01:16:27.695086+0000","last_became_active":"2024-04-25T01:16:26.667226+0000","last_became_peered":"2024-04-25T01:16:26.667226+0000","last_unstale":"2024-04-25T01:16:27.695086+0000","last_undegraded":"2024-04-25T01:16:27.695086+0000","last_fullsized":"2024-04-25T01:16:27.695086+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:23:19.810360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5,0],"acting":[4,3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.693968+0000","last_change":"2024-04-25T01:16:22.624757+0000","last_active":"2024-04-25T01:16:27.693968+0000","last_peered":"2024-04-25T01:16:27.693968+0000","last_clean":"2024-04-25T01:16:27.693968+0000","last_became_active":"2024-04-25T01:16:22.619992+0000","last_became_peered":"2024-04-25T01:16:22.619992+0000","last_unstale":"2024-04-25T01:16:27.693968+0000","last_undegraded":"2024-04-25T01:16:27.693968+0000","last_fullsized":"2024-04-25T01:16:27.693968+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:26:11.534238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.225000+0000","last_change":"2024-04-25T01:16:21.615700+0000","last_active":"2024-04-25T01:16:28.225000+0000","last_peered":"2024-04-25T01:16:28.225000+0000","last_clean":"2024-04-25T01:16:28.225000+0000","last_became_active":"2024-04-25T01:16:21.615207+0000","last_became_peered":"2024-04-25T01:16:21.615207+0000","last_unstale":"2024-04-25T01:16:28.225000+0000","last_undegraded":"2024-04-25T01:16:28.225000+0000","last_fullsized":"2024-04-25T01:16:28.225000+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:13.115669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.875349+0000","last_change":"2024-04-25T01:16:26.666133+0000","last_active":"2024-04-25T01:16:27.875349+0000","last_peered":"2024-04-25T01:16:27.875349+0000","last_clean":"2024-04-25T01:16:27.875349+0000","last_became_active":"2024-04-25T01:16:26.665359+0000","last_became_peered":"2024-04-25T01:16:26.665359+0000","last_unstale":"2024-04-25T01:16:27.875349+0000","last_undegraded":"2024-04-25T01:16:27.875349+0000","last_fullsized":"2024-04-25T01:16:27.875349+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:46:20.659273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1,7],"acting":[2,3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699852+0000","last_change":"2024-04-25T01:16:22.626495+0000","last_active":"2024-04-25T01:16:27.699852+0000","last_peered":"2024-04-25T01:16:27.699852+0000","last_clean":"2024-04-25T01:16:27.699852+0000","last_became_active":"2024-04-25T01:16:22.625913+0000","last_became_peered":"2024-04-25T01:16:22.625913+0000","last_unstale":"2024-04-25T01:16:27.699852+0000","last_undegraded":"2024-04-25T01:16:27.699852+0000","last_fullsized":"2024-04-25T01:16:27.699852+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:36.984987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.772343+0000","last_change":"2024-04-25T01:16:21.611655+0000","last_active":"2024-04-25T01:16:29.772343+0000","last_peered":"2024-04-25T01:16:29.772343+0000","last_clean":"2024-04-25T01:16:29.772343+0000","last_became_active":"2024-04-25T01:16:21.611388+0000","last_became_peered":"2024-04-25T01:16:21.611388+0000","last_unstale":"2024-04-25T01:16:29.772343+0000","last_undegraded":"2024-04-25T01:16:29.772343+0000","last_fullsized":"2024-04-25T01:16:29.772343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:39:20.330039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.223876+0000","last_change":"2024-04-25T01:16:26.668770+0000","last_active":"2024-04-25T01:16:28.223876+0000","last_peered":"2024-04-25T01:16:28.223876+0000","last_clean":"2024-04-25T01:16:28.223876+0000","last_became_active":"2024-04-25T01:16:26.668309+0000","last_became_peered":"2024-04-25T01:16:26.668309+0000","last_unstale":"2024-04-25T01:16:28.223876+0000","last_undegraded":"2024-04-25T01:16:28.223876+0000","last_fullsized":"2024-04-25T01:16:28.223876+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:17:20.386037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3,2],"acting":[7,1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.876399+0000","last_change":"2024-04-25T01:16:22.632717+0000","last_active":"2024-04-25T01:16:27.876399+0000","last_peered":"2024-04-25T01:16:27.876399+0000","last_clean":"2024-04-25T01:16:27.876399+0000","last_became_active":"2024-04-25T01:16:22.630174+0000","last_became_peered":"2024-04-25T01:16:22.630174+0000","last_unstale":"2024-04-25T01:16:27.876399+0000","last_undegraded":"2024-04-25T01:16:27.876399+0000","last_fullsized":"2024-04-25T01:16:27.876399+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:52.536070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.971972+0000","last_change":"2024-04-25T01:16:21.619013+0000","last_active":"2024-04-25T01:16:27.971972+0000","last_peered":"2024-04-25T01:16:27.971972+0000","last_clean":"2024-04-25T01:16:27.971972+0000","last_became_active":"2024-04-25T01:16:21.615903+0000","last_became_peered":"2024-04-25T01:16:21.615903+0000","last_unstale":"2024-04-25T01:16:27.971972+0000","last_undegraded":"2024-04-25T01:16:27.971972+0000","last_fullsized":"2024-04-25T01:16:27.971972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:37:47.297259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695313+0000","last_change":"2024-04-25T01:16:26.673940+0000","last_active":"2024-04-25T01:16:27.695313+0000","last_peered":"2024-04-25T01:16:27.695313+0000","last_clean":"2024-04-25T01:16:27.695313+0000","last_became_active":"2024-04-25T01:16:26.667111+0000","last_became_peered":"2024-04-25T01:16:26.667111+0000","last_unstale":"2024-04-25T01:16:27.695313+0000","last_undegraded":"2024-04-25T01:16:27.695313+0000","last_fullsized":"2024-04-25T01:16:27.695313+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:32.312792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3,7],"acting":[4,0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696886+0000","last_change":"2024-04-25T01:16:26.679002+0000","last_active":"2024-04-25T01:16:27.696886+0000","last_peered":"2024-04-25T01:16:27.696886+0000","last_clean":"2024-04-25T01:16:27.696886+0000","last_became_active":"2024-04-25T01:16:26.678398+0000","last_became_peered":"2024-04-25T01:16:26.678398+0000","last_unstale":"2024-04-25T01:16:27.696886+0000","last_undegraded":"2024-04-25T01:16:27.696886+0000","last_fullsized":"2024-04-25T01:16:27.696886+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:45:12.209250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5,3],"acting":[4,1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695447+0000","last_change":"2024-04-25T01:16:22.619871+0000","last_active":"2024-04-25T01:16:27.695447+0000","last_peered":"2024-04-25T01:16:27.695447+0000","last_clean":"2024-04-25T01:16:27.695447+0000","last_became_active":"2024-04-25T01:16:22.619553+0000","last_became_peered":"2024-04-25T01:16:22.619553+0000","last_unstale":"2024-04-25T01:16:27.695447+0000","last_undegraded":"2024-04-25T01:16:27.695447+0000","last_fullsized":"2024-04-25T01:16:27.695447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:46:33.270057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694865+0000","last_change":"2024-04-25T01:16:21.625865+0000","last_active":"2024-04-25T01:16:27.694865+0000","last_peered":"2024-04-25T01:16:27.694865+0000","last_clean":"2024-04-25T01:16:27.694865+0000","last_became_active":"2024-04-25T01:16:21.625566+0000","last_became_peered":"2024-04-25T01:16:21.625566+0000","last_unstale":"2024-04-25T01:16:27.694865+0000","last_undegraded":"2024-04-25T01:16:27.694865+0000","last_fullsized":"2024-04-25T01:16:27.694865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:44.675184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695078+0000","last_change":"2024-04-25T01:16:26.650705+0000","last_active":"2024-04-25T01:16:27.695078+0000","last_peered":"2024-04-25T01:16:27.695078+0000","last_clean":"2024-04-25T01:16:27.695078+0000","last_became_active":"2024-04-25T01:16:26.650332+0000","last_became_peered":"2024-04-25T01:16:26.650332+0000","last_unstale":"2024-04-25T01:16:27.695078+0000","last_undegraded":"2024-04-25T01:16:27.695078+0000","last_fullsized":"2024-04-25T01:16:27.695078+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:47:34.491567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,1],"acting":[3,2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694898+0000","last_change":"2024-04-25T01:16:22.622105+0000","last_active":"2024-04-25T01:16:27.694898+0000","last_peered":"2024-04-25T01:16:27.694898+0000","last_clean":"2024-04-25T01:16:27.694898+0000","last_became_active":"2024-04-25T01:16:22.621798+0000","last_became_peered":"2024-04-25T01:16:22.621798+0000","last_unstale":"2024-04-25T01:16:27.694898+0000","last_undegraded":"2024-04-25T01:16:27.694898+0000","last_fullsized":"2024-04-25T01:16:27.694898+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:47.769440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.087421+0000","last_change":"2024-04-25T01:16:21.621471+0000","last_active":"2024-04-25T01:16:30.087421+0000","last_peered":"2024-04-25T01:16:30.087421+0000","last_clean":"2024-04-25T01:16:30.087421+0000","last_became_active":"2024-04-25T01:16:21.621210+0000","last_became_peered":"2024-04-25T01:16:21.621210+0000","last_unstale":"2024-04-25T01:16:30.087421+0000","last_undegraded":"2024-04-25T01:16:30.087421+0000","last_fullsized":"2024-04-25T01:16:30.087421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:34.732380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695224+0000","last_change":"2024-04-25T01:16:21.619390+0000","last_active":"2024-04-25T01:16:27.695224+0000","last_peered":"2024-04-25T01:16:27.695224+0000","last_clean":"2024-04-25T01:16:27.695224+0000","last_became_active":"2024-04-25T01:16:21.619110+0000","last_became_peered":"2024-04-25T01:16:21.619110+0000","last_unstale":"2024-04-25T01:16:27.695224+0000","last_undegraded":"2024-04-25T01:16:27.695224+0000","last_fullsized":"2024-04-25T01:16:27.695224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:50.882927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696696+0000","last_change":"2024-04-25T01:16:22.638518+0000","last_active":"2024-04-25T01:16:27.696696+0000","last_peered":"2024-04-25T01:16:27.696696+0000","last_clean":"2024-04-25T01:16:27.696696+0000","last_became_active":"2024-04-25T01:16:22.638204+0000","last_became_peered":"2024-04-25T01:16:22.638204+0000","last_unstale":"2024-04-25T01:16:27.696696+0000","last_undegraded":"2024-04-25T01:16:27.696696+0000","last_fullsized":"2024-04-25T01:16:27.696696+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:06:58.102648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.697794+0000","last_change":"2024-04-25T01:16:26.673686+0000","last_active":"2024-04-25T01:16:27.697794+0000","last_peered":"2024-04-25T01:16:27.697794+0000","last_clean":"2024-04-25T01:16:27.697794+0000","last_became_active":"2024-04-25T01:16:26.673316+0000","last_became_peered":"2024-04-25T01:16:26.673316+0000","last_unstale":"2024-04-25T01:16:27.697794+0000","last_undegraded":"2024-04-25T01:16:27.697794+0000","last_fullsized":"2024-04-25T01:16:27.697794+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:26:15.585069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,7],"acting":[4,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696890+0000","last_change":"2024-04-25T01:16:21.623660+0000","last_active":"2024-04-25T01:16:27.696890+0000","last_peered":"2024-04-25T01:16:27.696890+0000","last_clean":"2024-04-25T01:16:27.696890+0000","last_became_active":"2024-04-25T01:16:21.623239+0000","last_became_peered":"2024-04-25T01:16:21.623239+0000","last_unstale":"2024-04-25T01:16:27.696890+0000","last_undegraded":"2024-04-25T01:16:27.696890+0000","last_fullsized":"2024-04-25T01:16:27.696890+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:20.547519+0000","last_clean_scrub_stamp":"2024-04-25T01:16:20.547519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:10:00.414303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.970886+0000","last_change":"2024-04-25T01:16:22.606098+0000","last_active":"2024-04-25T01:16:27.970886+0000","last_peered":"2024-04-25T01:16:27.970886+0000","last_clean":"2024-04-25T01:16:27.970886+0000","last_became_active":"2024-04-25T01:16:22.605632+0000","last_became_peered":"2024-04-25T01:16:22.605632+0000","last_unstale":"2024-04-25T01:16:27.970886+0000","last_undegraded":"2024-04-25T01:16:27.970886+0000","last_fullsized":"2024-04-25T01:16:27.970886+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:18:00.682845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.224854+0000","last_change":"2024-04-25T01:16:26.667852+0000","last_active":"2024-04-25T01:16:28.224854+0000","last_peered":"2024-04-25T01:16:28.224854+0000","last_clean":"2024-04-25T01:16:28.224854+0000","last_became_active":"2024-04-25T01:16:26.667070+0000","last_became_peered":"2024-04-25T01:16:26.667070+0000","last_unstale":"2024-04-25T01:16:28.224854+0000","last_undegraded":"2024-04-25T01:16:28.224854+0000","last_fullsized":"2024-04-25T01:16:28.224854+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:25.612945+0000","last_clean_scrub_stamp":"2024-04-25T01:16:25.612945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:35:37.472927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,0],"acting":[7,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.874985+0000","last_change":"2024-04-25T01:16:22.621505+0000","last_active":"2024-04-25T01:16:27.874985+0000","last_peered":"2024-04-25T01:16:27.874985+0000","last_clean":"2024-04-25T01:16:27.874985+0000","last_became_active":"2024-04-25T01:16:22.621239+0000","last_became_peered":"2024-04-25T01:16:22.621239+0000","last_unstale":"2024-04-25T01:16:27.874985+0000","last_undegraded":"2024-04-25T01:16:27.874985+0000","last_fullsized":"2024-04-25T01:16:27.874985+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:18:00.682845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.223851+0000","last_change":"2024-04-25T01:16:22.619015+0000","last_active":"2024-04-25T01:16:28.223851+0000","last_peered":"2024-04-25T01:16:28.223851+0000","last_clean":"2024-04-25T01:16:28.223851+0000","last_became_active":"2024-04-25T01:16:22.618739+0000","last_became_peered":"2024-04-25T01:16:22.618739+0000","last_unstale":"2024-04-25T01:16:28.223851+0000","last_undegraded":"2024-04-25T01:16:28.223851+0000","last_fullsized":"2024-04-25T01:16:28.223851+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:07.954692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694435+0000","last_change":"2024-04-25T01:16:22.621371+0000","last_active":"2024-04-25T01:16:27.694435+0000","last_peered":"2024-04-25T01:16:27.694435+0000","last_clean":"2024-04-25T01:16:27.694435+0000","last_became_active":"2024-04-25T01:16:22.621094+0000","last_became_peered":"2024-04-25T01:16:22.621094+0000","last_unstale":"2024-04-25T01:16:27.694435+0000","last_undegraded":"2024-04-25T01:16:27.694435+0000","last_fullsized":"2024-04-25T01:16:27.694435+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:23:15.759529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696710+0000","last_change":"2024-04-25T01:16:22.613471+0000","last_active":"2024-04-25T01:16:27.696710+0000","last_peered":"2024-04-25T01:16:27.696710+0000","last_clean":"2024-04-25T01:16:27.696710+0000","last_became_active":"2024-04-25T01:16:22.613036+0000","last_became_peered":"2024-04-25T01:16:22.613036+0000","last_unstale":"2024-04-25T01:16:27.696710+0000","last_undegraded":"2024-04-25T01:16:27.696710+0000","last_fullsized":"2024-04-25T01:16:27.696710+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:24:38.413314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.088877+0000","last_change":"2024-04-25T01:16:22.634780+0000","last_active":"2024-04-25T01:16:30.088877+0000","last_peered":"2024-04-25T01:16:30.088877+0000","last_clean":"2024-04-25T01:16:30.088877+0000","last_became_active":"2024-04-25T01:16:22.634412+0000","last_became_peered":"2024-04-25T01:16:22.634412+0000","last_unstale":"2024-04-25T01:16:30.088877+0000","last_undegraded":"2024-04-25T01:16:30.088877+0000","last_fullsized":"2024-04-25T01:16:30.088877+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:18:36.768908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.088852+0000","last_change":"2024-04-25T01:16:22.629001+0000","last_active":"2024-04-25T01:16:30.088852+0000","last_peered":"2024-04-25T01:16:30.088852+0000","last_clean":"2024-04-25T01:16:30.088852+0000","last_became_active":"2024-04-25T01:16:22.626649+0000","last_became_peered":"2024-04-25T01:16:22.626649+0000","last_unstale":"2024-04-25T01:16:30.088852+0000","last_undegraded":"2024-04-25T01:16:30.088852+0000","last_fullsized":"2024-04-25T01:16:30.088852+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:27:44.210712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.876273+0000","last_change":"2024-04-25T01:16:22.633405+0000","last_active":"2024-04-25T01:16:27.876273+0000","last_peered":"2024-04-25T01:16:27.876273+0000","last_clean":"2024-04-25T01:16:27.876273+0000","last_became_active":"2024-04-25T01:16:22.632539+0000","last_became_peered":"2024-04-25T01:16:22.632539+0000","last_unstale":"2024-04-25T01:16:27.876273+0000","last_undegraded":"2024-04-25T01:16:27.876273+0000","last_fullsized":"2024-04-25T01:16:27.876273+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:38:09.671586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699448+0000","last_change":"2024-04-25T01:16:22.615235+0000","last_active":"2024-04-25T01:16:27.699448+0000","last_peered":"2024-04-25T01:16:27.699448+0000","last_clean":"2024-04-25T01:16:27.699448+0000","last_became_active":"2024-04-25T01:16:22.614864+0000","last_became_peered":"2024-04-25T01:16:22.614864+0000","last_unstale":"2024-04-25T01:16:27.699448+0000","last_undegraded":"2024-04-25T01:16:27.699448+0000","last_fullsized":"2024-04-25T01:16:27.699448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:39:14.902973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.693959+0000","last_change":"2024-04-25T01:16:22.615333+0000","last_active":"2024-04-25T01:16:27.693959+0000","last_peered":"2024-04-25T01:16:27.693959+0000","last_clean":"2024-04-25T01:16:27.693959+0000","last_became_active":"2024-04-25T01:16:22.614510+0000","last_became_peered":"2024-04-25T01:16:22.614510+0000","last_unstale":"2024-04-25T01:16:27.693959+0000","last_undegraded":"2024-04-25T01:16:27.693959+0000","last_fullsized":"2024-04-25T01:16:27.693959+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:58:06.081060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696037+0000","last_change":"2024-04-25T01:16:22.615852+0000","last_active":"2024-04-25T01:16:27.696037+0000","last_peered":"2024-04-25T01:16:27.696037+0000","last_clean":"2024-04-25T01:16:27.696037+0000","last_became_active":"2024-04-25T01:16:22.615517+0000","last_became_peered":"2024-04-25T01:16:22.615517+0000","last_unstale":"2024-04-25T01:16:27.696037+0000","last_undegraded":"2024-04-25T01:16:27.696037+0000","last_fullsized":"2024-04-25T01:16:27.696037+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:31.459176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.970579+0000","last_change":"2024-04-25T01:16:22.627717+0000","last_active":"2024-04-25T01:16:27.970579+0000","last_peered":"2024-04-25T01:16:27.970579+0000","last_clean":"2024-04-25T01:16:27.970579+0000","last_became_active":"2024-04-25T01:16:22.627359+0000","last_became_peered":"2024-04-25T01:16:22.627359+0000","last_unstale":"2024-04-25T01:16:27.970579+0000","last_undegraded":"2024-04-25T01:16:27.970579+0000","last_fullsized":"2024-04-25T01:16:27.970579+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:51.897522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.696423+0000","last_change":"2024-04-25T01:16:22.634253+0000","last_active":"2024-04-25T01:16:27.696423+0000","last_peered":"2024-04-25T01:16:27.696423+0000","last_clean":"2024-04-25T01:16:27.696423+0000","last_became_active":"2024-04-25T01:16:22.632643+0000","last_became_peered":"2024-04-25T01:16:22.632643+0000","last_unstale":"2024-04-25T01:16:27.696423+0000","last_undegraded":"2024-04-25T01:16:27.696423+0000","last_fullsized":"2024-04-25T01:16:27.696423+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:14:10.348493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774424+0000","last_change":"2024-04-25T01:16:22.627377+0000","last_active":"2024-04-25T01:16:29.774424+0000","last_peered":"2024-04-25T01:16:29.774424+0000","last_clean":"2024-04-25T01:16:29.774424+0000","last_became_active":"2024-04-25T01:16:22.626141+0000","last_became_peered":"2024-04-25T01:16:22.626141+0000","last_unstale":"2024-04-25T01:16:29.774424+0000","last_undegraded":"2024-04-25T01:16:29.774424+0000","last_fullsized":"2024-04-25T01:16:29.774424+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:58:33.743727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695050+0000","last_change":"2024-04-25T01:16:22.622416+0000","last_active":"2024-04-25T01:16:27.695050+0000","last_peered":"2024-04-25T01:16:27.695050+0000","last_clean":"2024-04-25T01:16:27.695050+0000","last_became_active":"2024-04-25T01:16:22.620963+0000","last_became_peered":"2024-04-25T01:16:22.620963+0000","last_unstale":"2024-04-25T01:16:27.695050+0000","last_undegraded":"2024-04-25T01:16:27.695050+0000","last_fullsized":"2024-04-25T01:16:27.695050+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:58:32.477986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:28.225310+0000","last_change":"2024-04-25T01:16:22.614902+0000","last_active":"2024-04-25T01:16:28.225310+0000","last_peered":"2024-04-25T01:16:28.225310+0000","last_clean":"2024-04-25T01:16:28.225310+0000","last_became_active":"2024-04-25T01:16:22.614621+0000","last_became_peered":"2024-04-25T01:16:22.614621+0000","last_unstale":"2024-04-25T01:16:28.225310+0000","last_undegraded":"2024-04-25T01:16:28.225310+0000","last_fullsized":"2024-04-25T01:16:28.225310+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:45.689780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695622+0000","last_change":"2024-04-25T01:16:22.637378+0000","last_active":"2024-04-25T01:16:27.695622+0000","last_peered":"2024-04-25T01:16:27.695622+0000","last_clean":"2024-04-25T01:16:27.695622+0000","last_became_active":"2024-04-25T01:16:22.636599+0000","last_became_peered":"2024-04-25T01:16:22.636599+0000","last_unstale":"2024-04-25T01:16:27.695622+0000","last_undegraded":"2024-04-25T01:16:27.695622+0000","last_fullsized":"2024-04-25T01:16:27.695622+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:44:52.172147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.693572+0000","last_change":"2024-04-25T01:16:22.621416+0000","last_active":"2024-04-25T01:16:27.693572+0000","last_peered":"2024-04-25T01:16:27.693572+0000","last_clean":"2024-04-25T01:16:27.693572+0000","last_became_active":"2024-04-25T01:16:22.619180+0000","last_became_peered":"2024-04-25T01:16:22.619180+0000","last_unstale":"2024-04-25T01:16:27.693572+0000","last_undegraded":"2024-04-25T01:16:27.693572+0000","last_fullsized":"2024-04-25T01:16:27.693572+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:46:16.608442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.695613+0000","last_change":"2024-04-25T01:16:22.623812+0000","last_active":"2024-04-25T01:16:27.695613+0000","last_peered":"2024-04-25T01:16:27.695613+0000","last_clean":"2024-04-25T01:16:27.695613+0000","last_became_active":"2024-04-25T01:16:22.623468+0000","last_became_peered":"2024-04-25T01:16:22.623468+0000","last_unstale":"2024-04-25T01:16:27.695613+0000","last_undegraded":"2024-04-25T01:16:27.695613+0000","last_fullsized":"2024-04-25T01:16:27.695613+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:07:25.313465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.087421+0000","last_change":"2024-04-25T01:16:22.622280+0000","last_active":"2024-04-25T01:16:30.087421+0000","last_peered":"2024-04-25T01:16:30.087421+0000","last_clean":"2024-04-25T01:16:30.087421+0000","last_became_active":"2024-04-25T01:16:22.621901+0000","last_became_peered":"2024-04-25T01:16:22.621901+0000","last_unstale":"2024-04-25T01:16:30.087421+0000","last_undegraded":"2024-04-25T01:16:30.087421+0000","last_fullsized":"2024-04-25T01:16:30.087421+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:09:42.404928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:30.087951+0000","last_change":"2024-04-25T01:16:22.635778+0000","last_active":"2024-04-25T01:16:30.087951+0000","last_peered":"2024-04-25T01:16:30.087951+0000","last_clean":"2024-04-25T01:16:30.087951+0000","last_became_active":"2024-04-25T01:16:22.635492+0000","last_became_peered":"2024-04-25T01:16:22.635492+0000","last_unstale":"2024-04-25T01:16:30.087951+0000","last_undegraded":"2024-04-25T01:16:30.087951+0000","last_fullsized":"2024-04-25T01:16:30.087951+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:16:27.803006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774190+0000","last_change":"2024-04-25T01:16:22.627185+0000","last_active":"2024-04-25T01:16:29.774190+0000","last_peered":"2024-04-25T01:16:29.774190+0000","last_clean":"2024-04-25T01:16:29.774190+0000","last_became_active":"2024-04-25T01:16:22.626147+0000","last_became_peered":"2024-04-25T01:16:22.626147+0000","last_unstale":"2024-04-25T01:16:29.774190+0000","last_undegraded":"2024-04-25T01:16:29.774190+0000","last_fullsized":"2024-04-25T01:16:29.774190+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:06:31.257662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774396+0000","last_change":"2024-04-25T01:16:22.627705+0000","last_active":"2024-04-25T01:16:29.774396+0000","last_peered":"2024-04-25T01:16:29.774396+0000","last_clean":"2024-04-25T01:16:29.774396+0000","last_became_active":"2024-04-25T01:16:22.627424+0000","last_became_peered":"2024-04-25T01:16:22.627424+0000","last_unstale":"2024-04-25T01:16:29.774396+0000","last_undegraded":"2024-04-25T01:16:29.774396+0000","last_fullsized":"2024-04-25T01:16:29.774396+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:58:06.081060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774278+0000","last_change":"2024-04-25T01:16:22.625859+0000","last_active":"2024-04-25T01:16:29.774278+0000","last_peered":"2024-04-25T01:16:29.774278+0000","last_clean":"2024-04-25T01:16:29.774278+0000","last_became_active":"2024-04-25T01:16:22.625512+0000","last_became_peered":"2024-04-25T01:16:22.625512+0000","last_unstale":"2024-04-25T01:16:29.774278+0000","last_undegraded":"2024-04-25T01:16:29.774278+0000","last_fullsized":"2024-04-25T01:16:29.774278+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:09.176572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.971952+0000","last_change":"2024-04-25T01:16:22.622906+0000","last_active":"2024-04-25T01:16:27.971952+0000","last_peered":"2024-04-25T01:16:27.971952+0000","last_clean":"2024-04-25T01:16:27.971952+0000","last_became_active":"2024-04-25T01:16:22.622619+0000","last_became_peered":"2024-04-25T01:16:22.622619+0000","last_unstale":"2024-04-25T01:16:27.971952+0000","last_undegraded":"2024-04-25T01:16:27.971952+0000","last_fullsized":"2024-04-25T01:16:27.971952+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:20.635412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.694511+0000","last_change":"2024-04-25T01:16:22.623806+0000","last_active":"2024-04-25T01:16:27.694511+0000","last_peered":"2024-04-25T01:16:27.694511+0000","last_clean":"2024-04-25T01:16:27.694511+0000","last_became_active":"2024-04-25T01:16:22.623165+0000","last_became_peered":"2024-04-25T01:16:22.623165+0000","last_unstale":"2024-04-25T01:16:27.694511+0000","last_undegraded":"2024-04-25T01:16:27.694511+0000","last_fullsized":"2024-04-25T01:16:27.694511+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:01:54.012020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.773234+0000","last_change":"2024-04-25T01:16:22.621330+0000","last_active":"2024-04-25T01:16:29.773234+0000","last_peered":"2024-04-25T01:16:29.773234+0000","last_clean":"2024-04-25T01:16:29.773234+0000","last_became_active":"2024-04-25T01:16:22.620993+0000","last_became_peered":"2024-04-25T01:16:22.620993+0000","last_unstale":"2024-04-25T01:16:29.773234+0000","last_undegraded":"2024-04-25T01:16:29.773234+0000","last_fullsized":"2024-04-25T01:16:29.773234+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:53.625025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.875525+0000","last_change":"2024-04-25T01:16:22.623636+0000","last_active":"2024-04-25T01:16:27.875525+0000","last_peered":"2024-04-25T01:16:27.875525+0000","last_clean":"2024-04-25T01:16:27.875525+0000","last_became_active":"2024-04-25T01:16:22.623365+0000","last_became_peered":"2024-04-25T01:16:22.623365+0000","last_unstale":"2024-04-25T01:16:27.875525+0000","last_undegraded":"2024-04-25T01:16:27.875525+0000","last_fullsized":"2024-04-25T01:16:27.875525+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:49:05.483350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.773871+0000","last_change":"2024-04-25T01:16:22.616635+0000","last_active":"2024-04-25T01:16:29.773871+0000","last_peered":"2024-04-25T01:16:29.773871+0000","last_clean":"2024-04-25T01:16:29.773871+0000","last_became_active":"2024-04-25T01:16:22.616342+0000","last_became_peered":"2024-04-25T01:16:22.616342+0000","last_unstale":"2024-04-25T01:16:29.773871+0000","last_undegraded":"2024-04-25T01:16:29.773871+0000","last_fullsized":"2024-04-25T01:16:29.773871+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:09:07.843928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.875763+0000","last_change":"2024-04-25T01:16:22.623875+0000","last_active":"2024-04-25T01:16:27.875763+0000","last_peered":"2024-04-25T01:16:27.875763+0000","last_clean":"2024-04-25T01:16:27.875763+0000","last_became_active":"2024-04-25T01:16:22.623524+0000","last_became_peered":"2024-04-25T01:16:22.623524+0000","last_unstale":"2024-04-25T01:16:27.875763+0000","last_undegraded":"2024-04-25T01:16:27.875763+0000","last_fullsized":"2024-04-25T01:16:27.875763+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:07:32.166451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.698853+0000","last_change":"2024-04-25T01:16:22.620609+0000","last_active":"2024-04-25T01:16:27.698853+0000","last_peered":"2024-04-25T01:16:27.698853+0000","last_clean":"2024-04-25T01:16:27.698853+0000","last_became_active":"2024-04-25T01:16:22.620261+0000","last_became_peered":"2024-04-25T01:16:22.620261+0000","last_unstale":"2024-04-25T01:16:27.698853+0000","last_undegraded":"2024-04-25T01:16:27.698853+0000","last_fullsized":"2024-04-25T01:16:27.698853+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:01.416632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:27.699491+0000","last_change":"2024-04-25T01:16:22.633173+0000","last_active":"2024-04-25T01:16:27.699491+0000","last_peered":"2024-04-25T01:16:27.699491+0000","last_clean":"2024-04-25T01:16:27.699491+0000","last_became_active":"2024-04-25T01:16:22.632857+0000","last_became_peered":"2024-04-25T01:16:22.632857+0000","last_unstale":"2024-04-25T01:16:27.699491+0000","last_undegraded":"2024-04-25T01:16:27.699491+0000","last_fullsized":"2024-04-25T01:16:27.699491+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:14.238333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T01:16:29.774391+0000","last_change":"2024-04-25T01:16:22.628348+0000","last_active":"2024-04-25T01:16:29.774391+0000","last_peered":"2024-04-25T01:16:29.774391+0000","last_clean":"2024-04-25T01:16:29.774391+0000","last_became_active":"2024-04-25T01:16:22.627597+0000","last_became_peered":"2024-04-25T01:16:22.627597+0000","last_unstale":"2024-04-25T01:16:29.774391+0000","last_undegraded":"2024-04-25T01:16:29.774391+0000","last_fullsized":"2024-04-25T01:16:29.774391+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:21.562114+0000","last_clean_scrub_stamp":"2024-04-25T01:16:21.562114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:23:15.759529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542155,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":108288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542154,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":108164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542155,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":876544,"data_stored":698624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542154,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":294912,"data_stored":112678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542154,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":108746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542156,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":286720,"data_stored":112006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542154,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":872448,"data_stored":698602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542155,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":108220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:16:58.320 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-25T01:16:58.320 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-25T01:16:58.320 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-25T01:16:58.321 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-25T01:16:58.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.567+0000 7fdec7e00640 1 Processor -- start 2024-04-25T01:16:58.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.567+0000 7fdec7e00640 1 -- start start 2024-04-25T01:16:58.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.569+0000 7fdec7e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdec81443a0 0x7fdec8071f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:58.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.569+0000 7fdec7e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdec80cfde0 0x7fdec80d0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:58.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.569+0000 7fdec7e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdec80711b0 con 0x7fdec80b2c40 2024-04-25T01:16:58.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.569+0000 7fdec7e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fdec815c200 con 0x7fdec80cfde0 2024-04-25T01:16:58.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.569+0000 7fdec7e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fdec8071450 con 0x7fdec81443a0 2024-04-25T01:16:58.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.569+0000 7fdec6000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdec80cfde0 0x7fdec80d0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:58.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.569+0000 7fdec6000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdec80cfde0 0x7fdec80d0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56342/0 (socket says 172.21.15.90:56342) 2024-04-25T01:16:58.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.569+0000 7fdec6000640 1 -- 172.21.15.90:0/1613191716 learned_addr learned my addr 172.21.15.90:0/1613191716 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:58.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.570+0000 7fdec6a00640 1 --2- 172.21.15.90:0/1613191716 >> v2:172.21.15.103:3300/0 conn(0x7fdec81443a0 0x7fdec8071f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:58.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.570+0000 7fdec6000640 1 -- 172.21.15.90:0/1613191716 >> v2:172.21.15.103:3300/0 conn(0x7fdec81443a0 msgr2=0x7fdec8071f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:58.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.570+0000 7fdec6000640 1 --2- 172.21.15.90:0/1613191716 >> v2:172.21.15.103:3300/0 conn(0x7fdec81443a0 0x7fdec8071f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:58.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.570+0000 7fdec6000640 1 -- 172.21.15.90:0/1613191716 >> v1:172.21.15.90:6789/0 conn(0x7fdec80b2c40 legacy=0x7fdec80b3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:58.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.570+0000 7fdec6000640 1 -- 172.21.15.90:0/1613191716 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdec8071730 con 0x7fdec80cfde0 2024-04-25T01:16:58.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.570+0000 7fdec6a00640 1 --2- 172.21.15.90:0/1613191716 >> v2:172.21.15.103:3300/0 conn(0x7fdec81443a0 0x7fdec8071f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:58.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.571+0000 7fdec6000640 1 --2- 172.21.15.90:0/1613191716 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdec80cfde0 0x7fdec80d0290 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdec80cd5e0 tx=0x7fdeb402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30304a5a240240be server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:58.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.571+0000 7fdec5600640 1 -- 172.21.15.90:0/1613191716 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb4037070 con 0x7fdec80cfde0 2024-04-25T01:16:58.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.571+0000 7fdec5600640 1 -- 172.21.15.90:0/1613191716 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdeb4032440 con 0x7fdec80cfde0 2024-04-25T01:16:58.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.571+0000 7fdec5600640 1 -- 172.21.15.90:0/1613191716 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb4032810 con 0x7fdec80cfde0 2024-04-25T01:16:58.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.572+0000 7fdec7e00640 1 -- 172.21.15.90:0/1613191716 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdec80cfde0 msgr2=0x7fdec80d0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:58.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.572+0000 7fdec7e00640 1 --2- 172.21.15.90:0/1613191716 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdec80cfde0 0x7fdec80d0290 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdec80cd5e0 tx=0x7fdeb402a270 comp rx=0 tx=0).stop 2024-04-25T01:16:58.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.572+0000 7fdec7e00640 1 -- 172.21.15.90:0/1613191716 shutdown_connections 2024-04-25T01:16:58.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.572+0000 7fdec7e00640 1 --2- 172.21.15.90:0/1613191716 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdec80cfde0 0x7fdec80d0290 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:58.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.572+0000 7fdec7e00640 1 --2- 172.21.15.90:0/1613191716 >> v2:172.21.15.103:3300/0 conn(0x7fdec81443a0 0x7fdec8071f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:58.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.572+0000 7fdec7e00640 1 -- 172.21.15.90:0/1613191716 >> 172.21.15.90:0/1613191716 conn(0x7fdec80b1ac0 msgr2=0x7fdec80b1ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:58.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.572+0000 7fdec7e00640 1 -- 172.21.15.90:0/1613191716 shutdown_connections 2024-04-25T01:16:58.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.572+0000 7fdec7e00640 1 -- 172.21.15.90:0/1613191716 wait complete. 2024-04-25T01:16:58.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.573+0000 7fdec7e00640 1 Processor -- start 2024-04-25T01:16:58.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.574+0000 7fdec7e00640 1 -- start start 2024-04-25T01:16:58.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.574+0000 7fdec7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdec80b2c40 0x7fdec80cca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:58.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.574+0000 7fdec7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdec80cfde0 0x7fdec80c5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:58.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.574+0000 7fdec7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdec81443a0 0x7fdec80c60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:58.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.574+0000 7fdec7e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdec80b39c0 con 0x7fdec80b2c40 2024-04-25T01:16:58.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.574+0000 7fdec7e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdec80b3840 con 0x7fdec81443a0 2024-04-25T01:16:58.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.574+0000 7fdec7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdec8144780 con 0x7fdec80cfde0 2024-04-25T01:16:58.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.575+0000 7fdec6a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdec80b2c40 0x7fdec80cca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:58.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.575+0000 7fdec6a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdec80b2c40 0x7fdec80cca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56348/0 (socket says 172.21.15.90:56348) 2024-04-25T01:16:58.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.575+0000 7fdec6a00640 1 -- 172.21.15.90:0/1335400445 learned_addr learned my addr 172.21.15.90:0/1335400445 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:58.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.575+0000 7fdec6000640 1 --2- 172.21.15.90:0/1335400445 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdec80cfde0 0x7fdec80c5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:58.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.575+0000 7fdec7400640 1 --2- 172.21.15.90:0/1335400445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdec81443a0 0x7fdec80c60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:58.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.575+0000 7fdec6a00640 1 -- 172.21.15.90:0/1335400445 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdec80cfde0 msgr2=0x7fdec80c5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:58.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.575+0000 7fdec6a00640 1 --2- 172.21.15.90:0/1335400445 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdec80cfde0 0x7fdec80c5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:58.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.575+0000 7fdec6a00640 1 -- 172.21.15.90:0/1335400445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdec81443a0 msgr2=0x7fdec80c60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:58.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.575+0000 7fdec6a00640 1 --2- 172.21.15.90:0/1335400445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdec81443a0 0x7fdec80c60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:58.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.575+0000 7fdec6a00640 1 -- 172.21.15.90:0/1335400445 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdec81f4b50 con 0x7fdec80b2c40 2024-04-25T01:16:58.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.575+0000 7fdec6a00640 1 --2- 172.21.15.90:0/1335400445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdec80b2c40 0x7fdec80cca80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdeb800c820 tx=0x7fdeb800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:58.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.575+0000 7fdebea00640 1 -- 172.21.15.90:0/1335400445 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb8015070 con 0x7fdec80b2c40 2024-04-25T01:16:58.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.575+0000 7fdebea00640 1 -- 172.21.15.90:0/1335400445 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdeb80079d0 con 0x7fdec80b2c40 2024-04-25T01:16:58.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.575+0000 7fdec7e00640 1 -- 172.21.15.90:0/1335400445 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdec80c7700 con 0x7fdec80b2c40 2024-04-25T01:16:58.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.576+0000 7fdebea00640 1 -- 172.21.15.90:0/1335400445 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb8007da0 con 0x7fdec80b2c40 2024-04-25T01:16:58.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.576+0000 7fdec7e00640 1 -- 172.21.15.90:0/1335400445 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdec81f4070 con 0x7fdec80b2c40 2024-04-25T01:16:58.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.577+0000 7fdec7e00640 1 -- 172.21.15.90:0/1335400445 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdec8068fb0 con 0x7fdec80b2c40 2024-04-25T01:16:58.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.577+0000 7fdebea00640 1 -- 172.21.15.90:0/1335400445 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fdeb8018070 con 0x7fdec80b2c40 2024-04-25T01:16:58.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.579+0000 7fdebea00640 1 --2- 172.21.15.90:0/1335400445 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdea806be30 0x7fdea806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:58.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.579+0000 7fdebea00640 1 -- 172.21.15.90:0/1335400445 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fdeb808b0d0 con 0x7fdec80b2c40 2024-04-25T01:16:58.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.579+0000 7fdec6000640 1 --2- 172.21.15.90:0/1335400445 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdea806be30 0x7fdea806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:58.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.580+0000 7fdec6000640 1 --2- 172.21.15.90:0/1335400445 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdea806be30 0x7fdea806e2f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4034000 tx=0x7fdeb402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:58.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.583+0000 7fdebea00640 1 -- 172.21.15.90:0/1335400445 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fdeb8059740 con 0x7fdec80b2c40 2024-04-25T01:16:58.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.813+0000 7fdec7e00640 1 -- 172.21.15.90:0/1335400445 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fdec80cc2f0 con 0x7fdec80b2c40 2024-04-25T01:16:58.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.813+0000 7fdebea00640 1 -- 172.21.15.90:0/1335400445 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fdeb8059040 con 0x7fdec80b2c40 2024-04-25T01:16:58.798 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:16:58.798 INFO:teuthology.orchestra.run.smithi090.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-25T01:16:58.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.816+0000 7fdec7e00640 1 -- 172.21.15.90:0/1335400445 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdea806be30 msgr2=0x7fdea806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:58.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.816+0000 7fdec7e00640 1 --2- 172.21.15.90:0/1335400445 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdea806be30 0x7fdea806e2f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4034000 tx=0x7fdeb402e000 comp rx=0 tx=0).stop 2024-04-25T01:16:58.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.816+0000 7fdec7e00640 1 -- 172.21.15.90:0/1335400445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdec80b2c40 msgr2=0x7fdec80cca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:58.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.816+0000 7fdec7e00640 1 --2- 172.21.15.90:0/1335400445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdec80b2c40 0x7fdec80cca80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdeb800c820 tx=0x7fdeb800cc20 comp rx=0 tx=0).stop 2024-04-25T01:16:58.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.816+0000 7fdec7e00640 1 -- 172.21.15.90:0/1335400445 shutdown_connections 2024-04-25T01:16:58.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.816+0000 7fdec7e00640 1 --2- 172.21.15.90:0/1335400445 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdea806be30 0x7fdea806e2f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:58.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.816+0000 7fdec7e00640 1 --2- 172.21.15.90:0/1335400445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdec81443a0 0x7fdec80c60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:58.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.816+0000 7fdec7e00640 1 --2- 172.21.15.90:0/1335400445 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdec80cfde0 0x7fdec80c5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:58.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.816+0000 7fdec7e00640 1 --2- 172.21.15.90:0/1335400445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdec80b2c40 0x7fdec80cca80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:58.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.816+0000 7fdec7e00640 1 -- 172.21.15.90:0/1335400445 >> 172.21.15.90:0/1335400445 conn(0x7fdec80b1ac0 msgr2=0x7fdec806ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:58.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.816+0000 7fdec7e00640 1 -- 172.21.15.90:0/1335400445 shutdown_connections 2024-04-25T01:16:58.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:58.816+0000 7fdec7e00640 1 -- 172.21.15.90:0/1335400445 wait complete. 2024-04-25T01:16:58.817 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-25T01:16:58.818 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:16:58.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.015+0000 7f017a200640 1 Processor -- start 2024-04-25T01:16:58.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.015+0000 7f017a200640 1 -- start start 2024-04-25T01:16:59.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.016+0000 7f017a200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0174143d20 0x7f017415ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:59.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.016+0000 7f017a200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f017406ed60 0x7f017415f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:59.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.016+0000 7f017a200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f017415f770 con 0x7f0174143940 2024-04-25T01:16:59.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.016+0000 7f017a200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f0174070e60 con 0x7f017406ed60 2024-04-25T01:16:59.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.016+0000 7f017a200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0174071140 con 0x7f0174143d20 2024-04-25T01:16:59.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.016+0000 7f0173e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f017406ed60 0x7f017415f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:59.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.017+0000 7f0173e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f017406ed60 0x7f017415f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56362/0 (socket says 172.21.15.90:56362) 2024-04-25T01:16:59.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.017+0000 7f0173e00640 1 -- 172.21.15.90:0/1460364379 learned_addr learned my addr 172.21.15.90:0/1460364379 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:59.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.017+0000 7f0173e00640 1 -- 172.21.15.90:0/1460364379 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0174143d20 msgr2=0x7f017415ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:59.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.017+0000 7f0173e00640 1 --2- 172.21.15.90:0/1460364379 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0174143d20 0x7f017415ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:59.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.017+0000 7f0173e00640 1 -- 172.21.15.90:0/1460364379 >> v1:172.21.15.90:6789/0 conn(0x7f0174143940 legacy=0x7f017406e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:59.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.017+0000 7f0173e00640 1 -- 172.21.15.90:0/1460364379 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0174071420 con 0x7f017406ed60 2024-04-25T01:16:59.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.018+0000 7f0173e00640 1 --2- 172.21.15.90:0/1460364379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f017406ed60 0x7f017415f230 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f016c009e80 tx=0x7f016c02a750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a48b9a8d007389bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:59.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.018+0000 7f0172a00640 1 -- 172.21.15.90:0/1460364379 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016c036070 con 0x7f017406ed60 2024-04-25T01:16:59.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.018+0000 7f0172a00640 1 -- 172.21.15.90:0/1460364379 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f016c031600 con 0x7f017406ed60 2024-04-25T01:16:59.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.018+0000 7f0172a00640 1 -- 172.21.15.90:0/1460364379 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016c0319d0 con 0x7f017406ed60 2024-04-25T01:16:59.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.019+0000 7f017a200640 1 -- 172.21.15.90:0/1460364379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f017406ed60 msgr2=0x7f017415f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:59.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.019+0000 7f017a200640 1 --2- 172.21.15.90:0/1460364379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f017406ed60 0x7f017415f230 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f016c009e80 tx=0x7f016c02a750 comp rx=0 tx=0).stop 2024-04-25T01:16:59.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.019+0000 7f017a200640 1 -- 172.21.15.90:0/1460364379 shutdown_connections 2024-04-25T01:16:59.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.019+0000 7f017a200640 1 --2- 172.21.15.90:0/1460364379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f017406ed60 0x7f017415f230 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:59.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.019+0000 7f017a200640 1 --2- 172.21.15.90:0/1460364379 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0174143d20 0x7f017415ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:59.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.019+0000 7f017a200640 1 -- 172.21.15.90:0/1460364379 >> 172.21.15.90:0/1460364379 conn(0x7f01740b1820 msgr2=0x7f01740b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:59.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.019+0000 7f017a200640 1 -- 172.21.15.90:0/1460364379 shutdown_connections 2024-04-25T01:16:59.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.019+0000 7f017a200640 1 -- 172.21.15.90:0/1460364379 wait complete. 2024-04-25T01:16:59.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.021+0000 7f017a200640 1 Processor -- start 2024-04-25T01:16:59.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.021+0000 7f017a200640 1 -- start start 2024-04-25T01:16:59.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.022+0000 7f017a200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f017406ed60 0x7f0174142420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:59.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.022+0000 7f017a200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0174143940 0x7f017413f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:59.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.022+0000 7f017a200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0174143d20 0x7f017413f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:59.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.022+0000 7f017a200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0174071da0 con 0x7f017406ed60 2024-04-25T01:16:59.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.022+0000 7f017a200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0174071c20 con 0x7f0174143940 2024-04-25T01:16:59.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.022+0000 7f017a200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0174071f20 con 0x7f0174143d20 2024-04-25T01:16:59.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.022+0000 7f0173400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f017406ed60 0x7f0174142420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:59.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.022+0000 7f0173400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f017406ed60 0x7f0174142420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56370/0 (socket says 172.21.15.90:56370) 2024-04-25T01:16:59.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.022+0000 7f0173400640 1 -- 172.21.15.90:0/1251580710 learned_addr learned my addr 172.21.15.90:0/1251580710 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:59.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.023+0000 7f0173400640 1 -- 172.21.15.90:0/1251580710 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0174143d20 msgr2=0x7f017413f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:59.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.022+0000 7f016be00640 1 --2- 172.21.15.90:0/1251580710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0174143940 0x7f017413f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:59.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.023+0000 7f0173e00640 1 --2- 172.21.15.90:0/1251580710 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0174143d20 0x7f017413f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:59.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.023+0000 7f0173400640 1 --2- 172.21.15.90:0/1251580710 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0174143d20 0x7f017413f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:59.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.023+0000 7f0173400640 1 -- 172.21.15.90:0/1251580710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0174143940 msgr2=0x7f017413f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:59.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.023+0000 7f0173400640 1 --2- 172.21.15.90:0/1251580710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0174143940 0x7f017413f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:59.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.023+0000 7f0173400640 1 -- 172.21.15.90:0/1251580710 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01741e48b0 con 0x7f017406ed60 2024-04-25T01:16:59.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.023+0000 7f0173e00640 1 --2- 172.21.15.90:0/1251580710 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0174143d20 0x7f017413f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:59.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.023+0000 7f016be00640 1 --2- 172.21.15.90:0/1251580710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0174143940 0x7f017413f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:59.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.023+0000 7f0173400640 1 --2- 172.21.15.90:0/1251580710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f017406ed60 0x7f0174142420 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f016400c400 tx=0x7f016400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:59.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.023+0000 7f016b400640 1 -- 172.21.15.90:0/1251580710 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01640052d0 con 0x7f017406ed60 2024-04-25T01:16:59.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.023+0000 7f016b400640 1 -- 172.21.15.90:0/1251580710 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0164005490 con 0x7f017406ed60 2024-04-25T01:16:59.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.023+0000 7f016b400640 1 -- 172.21.15.90:0/1251580710 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0164007260 con 0x7f017406ed60 2024-04-25T01:16:59.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.023+0000 7f017a200640 1 -- 172.21.15.90:0/1251580710 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01741f82d0 con 0x7f017406ed60 2024-04-25T01:16:59.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.023+0000 7f017a200640 1 -- 172.21.15.90:0/1251580710 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01741e4e50 con 0x7f017406ed60 2024-04-25T01:16:59.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.024+0000 7f017a200640 1 -- 172.21.15.90:0/1251580710 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0174068fb0 con 0x7f017406ed60 2024-04-25T01:16:59.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.025+0000 7f016b400640 1 -- 172.21.15.90:0/1251580710 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0164007400 con 0x7f017406ed60 2024-04-25T01:16:59.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.026+0000 7f016b400640 1 --2- 172.21.15.90:0/1251580710 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f015006be30 0x7f015006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:59.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.026+0000 7f016b400640 1 -- 172.21.15.90:0/1251580710 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f016408a650 con 0x7f017406ed60 2024-04-25T01:16:59.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.026+0000 7f016be00640 1 --2- 172.21.15.90:0/1251580710 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f015006be30 0x7f015006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:59.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.027+0000 7f016be00640 1 --2- 172.21.15.90:0/1251580710 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f015006be30 0x7f015006e2f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f017415de40 tx=0x7f015c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:59.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.030+0000 7f016b400640 1 -- 172.21.15.90:0/1251580710 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0164058d50 con 0x7f017406ed60 2024-04-25T01:16:59.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.213+0000 7f017a200640 1 -- 172.21.15.90:0/1251580710 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f017406d150 con 0x7f017406ed60 2024-04-25T01:16:59.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.217+0000 7f016b400640 1 -- 172.21.15.90:0/1251580710 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18482 (secure 0 0 0) 0x7f0164058650 con 0x7f017406ed60 2024-04-25T01:16:59.201 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:16:59.201 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":26,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:16:27.092855+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:16:20.094512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:16:21.078268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:16:24.670642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:16:59.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.219+0000 7f017a200640 1 -- 172.21.15.90:0/1251580710 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f015006be30 msgr2=0x7f015006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:59.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.219+0000 7f017a200640 1 --2- 172.21.15.90:0/1251580710 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f015006be30 0x7f015006e2f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f017415de40 tx=0x7f015c008040 comp rx=0 tx=0).stop 2024-04-25T01:16:59.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.219+0000 7f017a200640 1 -- 172.21.15.90:0/1251580710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f017406ed60 msgr2=0x7f0174142420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:59.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.219+0000 7f017a200640 1 --2- 172.21.15.90:0/1251580710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f017406ed60 0x7f0174142420 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f016400c400 tx=0x7f016400c8c0 comp rx=0 tx=0).stop 2024-04-25T01:16:59.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.219+0000 7f017a200640 1 -- 172.21.15.90:0/1251580710 shutdown_connections 2024-04-25T01:16:59.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.219+0000 7f017a200640 1 --2- 172.21.15.90:0/1251580710 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f015006be30 0x7f015006e2f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:59.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.219+0000 7f017a200640 1 --2- 172.21.15.90:0/1251580710 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0174143d20 0x7f017413f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:59.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.219+0000 7f017a200640 1 --2- 172.21.15.90:0/1251580710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0174143940 0x7f017413f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:59.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.219+0000 7f017a200640 1 --2- 172.21.15.90:0/1251580710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f017406ed60 0x7f0174142420 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:59.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.219+0000 7f017a200640 1 -- 172.21.15.90:0/1251580710 >> 172.21.15.90:0/1251580710 conn(0x7f01740b1820 msgr2=0x7f01740727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:59.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.219+0000 7f017a200640 1 -- 172.21.15.90:0/1251580710 shutdown_connections 2024-04-25T01:16:59.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.219+0000 7f017a200640 1 -- 172.21.15.90:0/1251580710 wait complete. 2024-04-25T01:16:59.220 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:16:15.894126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:16:20.094512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:16:21.078268+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:16:24.670642+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:16:59.221 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:16:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.433+0000 7f5461c00640 1 Processor -- start 2024-04-25T01:16:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.433+0000 7f5461c00640 1 -- start start 2024-04-25T01:16:59.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.435+0000 7f5461c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f545c15fc40 0x7f545c1600a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:59.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.435+0000 7f5461c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f545c06d110 con 0x7f545c0cedc0 2024-04-25T01:16:59.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.435+0000 7f5461c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f545c06d3f0 con 0x7f545c0722e0 2024-04-25T01:16:59.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.435+0000 7f5461c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f545c160aa0 con 0x7f545c15fc40 2024-04-25T01:16:59.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.435+0000 7f545b400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f545c0cedc0 0x7f545c0cf1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:35706/0 (socket says 172.21.15.90:35706) 2024-04-25T01:16:59.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.435+0000 7f545b400640 1 -- 172.21.15.90:0/3860860857 learned_addr learned my addr 172.21.15.90:0/3860860857 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:59.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.435+0000 7f5453e00640 1 --2- 172.21.15.90:0/3860860857 >> v2:172.21.15.90:3300/0 conn(0x7f545c15fc40 0x7f545c1600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:59.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.436+0000 7f545a000640 1 -- 172.21.15.90:0/3860860857 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3567356263 0 0) 0x7f545c06d110 con 0x7f545c0cedc0 2024-04-25T01:16:59.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.436+0000 7f545a000640 1 -- 172.21.15.90:0/3860860857 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5440003520 con 0x7f545c0cedc0 2024-04-25T01:16:59.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.436+0000 7f5453e00640 1 -- 172.21.15.90:0/3860860857 >> v1:172.21.15.103:6789/0 conn(0x7f545c0722e0 legacy=0x7f545c06ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:59.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.436+0000 7f5453e00640 1 -- 172.21.15.90:0/3860860857 >> v1:172.21.15.90:6789/0 conn(0x7f545c0cedc0 legacy=0x7f545c0cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:59.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.436+0000 7f5453e00640 1 -- 172.21.15.90:0/3860860857 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f545c06d6d0 con 0x7f545c15fc40 2024-04-25T01:16:59.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.437+0000 7f5453e00640 1 --2- 172.21.15.90:0/3860860857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f545c15fc40 0x7f545c1600a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f54440097a0 tx=0x7f544402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7114b87bca668415 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:59.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.437+0000 7f545a000640 1 -- 172.21.15.90:0/3860860857 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5444037070 con 0x7f545c15fc40 2024-04-25T01:16:59.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.437+0000 7f545a000640 1 -- 172.21.15.90:0/3860860857 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f544402ae50 con 0x7f545c15fc40 2024-04-25T01:16:59.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.437+0000 7f545a000640 1 -- 172.21.15.90:0/3860860857 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5444032660 con 0x7f545c15fc40 2024-04-25T01:16:59.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.438+0000 7f5461c00640 1 -- 172.21.15.90:0/3860860857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f545c15fc40 msgr2=0x7f545c1600a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:59.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.438+0000 7f5461c00640 1 --2- 172.21.15.90:0/3860860857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f545c15fc40 0x7f545c1600a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f54440097a0 tx=0x7f544402a140 comp rx=0 tx=0).stop 2024-04-25T01:16:59.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.438+0000 7f5461c00640 1 -- 172.21.15.90:0/3860860857 shutdown_connections 2024-04-25T01:16:59.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.438+0000 7f5461c00640 1 --2- 172.21.15.90:0/3860860857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f545c15fc40 0x7f545c1600a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:59.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.438+0000 7f5461c00640 1 -- 172.21.15.90:0/3860860857 >> 172.21.15.90:0/3860860857 conn(0x7f545c06b720 msgr2=0x7f545c0b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:59.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.438+0000 7f5461c00640 1 -- 172.21.15.90:0/3860860857 shutdown_connections 2024-04-25T01:16:59.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.438+0000 7f5461c00640 1 -- 172.21.15.90:0/3860860857 wait complete. 2024-04-25T01:16:59.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.440+0000 7f5461c00640 1 Processor -- start 2024-04-25T01:16:59.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.440+0000 7f5461c00640 1 -- start start 2024-04-25T01:16:59.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.441+0000 7f5461c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f545c0722e0 0x7f545c1465d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:59.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.441+0000 7f5461c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f545c0cedc0 0x7f545c0b4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:59.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.441+0000 7f5461c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f545c15fc40 0x7f545c15e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:59.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.441+0000 7f5461c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f545c070f60 con 0x7f545c15fc40 2024-04-25T01:16:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.441+0000 7f5461c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f545c070de0 con 0x7f545c0cedc0 2024-04-25T01:16:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.441+0000 7f5461c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f545c0710e0 con 0x7f545c0722e0 2024-04-25T01:16:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.441+0000 7f545b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f545c15fc40 0x7f545c15e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.441+0000 7f545b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f545c15fc40 0x7f545c15e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56382/0 (socket says 172.21.15.90:56382) 2024-04-25T01:16:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.441+0000 7f545b400640 1 -- 172.21.15.90:0/1190848880 learned_addr learned my addr 172.21.15.90:0/1190848880 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.441+0000 7f545aa00640 1 --2- 172.21.15.90:0/1190848880 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f545c0722e0 0x7f545c1465d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:59.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.441+0000 7f545b400640 1 -- 172.21.15.90:0/1190848880 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f545c0722e0 msgr2=0x7f545c1465d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:59.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.441+0000 7f545b400640 1 --2- 172.21.15.90:0/1190848880 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f545c0722e0 0x7f545c1465d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:59.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.441+0000 7f5453e00640 1 --2- 172.21.15.90:0/1190848880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f545c0cedc0 0x7f545c0b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:59.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.441+0000 7f545b400640 1 -- 172.21.15.90:0/1190848880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f545c0cedc0 msgr2=0x7f545c0b4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:59.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.441+0000 7f545b400640 1 --2- 172.21.15.90:0/1190848880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f545c0cedc0 0x7f545c0b4330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:59.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.441+0000 7f545b400640 1 -- 172.21.15.90:0/1190848880 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f545c146b10 con 0x7f545c15fc40 2024-04-25T01:16:59.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.441+0000 7f545aa00640 1 --2- 172.21.15.90:0/1190848880 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f545c0722e0 0x7f545c1465d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:59.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.442+0000 7f545b400640 1 --2- 172.21.15.90:0/1190848880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f545c15fc40 0x7f545c15e520 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f545400cb70 tx=0x7f545400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:59.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.442+0000 7f5452a00640 1 -- 172.21.15.90:0/1190848880 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5454017070 con 0x7f545c15fc40 2024-04-25T01:16:59.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.442+0000 7f5452a00640 1 -- 172.21.15.90:0/1190848880 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f545400ad40 con 0x7f545c15fc40 2024-04-25T01:16:59.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.442+0000 7f5452a00640 1 -- 172.21.15.90:0/1190848880 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5454012620 con 0x7f545c15fc40 2024-04-25T01:16:59.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.442+0000 7f5461c00640 1 -- 172.21.15.90:0/1190848880 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f545c0b4870 con 0x7f545c15fc40 2024-04-25T01:16:59.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.442+0000 7f5461c00640 1 -- 172.21.15.90:0/1190848880 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f545c157540 con 0x7f545c15fc40 2024-04-25T01:16:59.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.443+0000 7f5461c00640 1 -- 172.21.15.90:0/1190848880 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f545c06cc00 con 0x7f545c15fc40 2024-04-25T01:16:59.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.444+0000 7f5452a00640 1 -- 172.21.15.90:0/1190848880 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f545401a070 con 0x7f545c15fc40 2024-04-25T01:16:59.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.445+0000 7f5452a00640 1 --2- 172.21.15.90:0/1190848880 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f543806be30 0x7f543806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:59.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.445+0000 7f5452a00640 1 -- 172.21.15.90:0/1190848880 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f545408afe0 con 0x7f545c15fc40 2024-04-25T01:16:59.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.445+0000 7f545aa00640 1 --2- 172.21.15.90:0/1190848880 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f543806be30 0x7f543806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:59.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.446+0000 7f545aa00640 1 --2- 172.21.15.90:0/1190848880 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f543806be30 0x7f543806e2f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f544c003eb0 tx=0x7f544c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:59.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.449+0000 7f5452a00640 1 -- 172.21.15.90:0/1190848880 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f545c06cc00 con 0x7f545c15fc40 2024-04-25T01:16:59.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.645+0000 7f5461c00640 1 -- 172.21.15.90:0/1190848880 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f545c072c90 con 0x7f545c15fc40 2024-04-25T01:16:59.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.646+0000 7f5452a00640 1 -- 172.21.15.90:0/1190848880 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f5454059680 con 0x7f545c15fc40 2024-04-25T01:16:59.630 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T01:16:59.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.648+0000 7f5461c00640 1 -- 172.21.15.90:0/1190848880 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f543806be30 msgr2=0x7f543806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:59.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.648+0000 7f5461c00640 1 --2- 172.21.15.90:0/1190848880 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f543806be30 0x7f543806e2f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f544c003eb0 tx=0x7f544c009040 comp rx=0 tx=0).stop 2024-04-25T01:16:59.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.648+0000 7f5461c00640 1 -- 172.21.15.90:0/1190848880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f545c15fc40 msgr2=0x7f545c15e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:59.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.648+0000 7f5461c00640 1 --2- 172.21.15.90:0/1190848880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f545c15fc40 0x7f545c15e520 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f545400cb70 tx=0x7f545400a430 comp rx=0 tx=0).stop 2024-04-25T01:16:59.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.648+0000 7f5461c00640 1 -- 172.21.15.90:0/1190848880 shutdown_connections 2024-04-25T01:16:59.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.648+0000 7f5461c00640 1 --2- 172.21.15.90:0/1190848880 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f543806be30 0x7f543806e2f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:59.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.648+0000 7f5461c00640 1 --2- 172.21.15.90:0/1190848880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f545c15fc40 0x7f545c15e520 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:59.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.649+0000 7f5461c00640 1 --2- 172.21.15.90:0/1190848880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f545c0cedc0 0x7f545c0b4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:59.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.649+0000 7f5461c00640 1 --2- 172.21.15.90:0/1190848880 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f545c0722e0 0x7f545c1465d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:59.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.649+0000 7f5461c00640 1 -- 172.21.15.90:0/1190848880 >> 172.21.15.90:0/1190848880 conn(0x7f545c06b720 msgr2=0x7f545c074540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:59.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.649+0000 7f5461c00640 1 -- 172.21.15.90:0/1190848880 shutdown_connections 2024-04-25T01:16:59.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.649+0000 7f5461c00640 1 -- 172.21.15.90:0/1190848880 wait complete. 2024-04-25T01:16:59.651 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:16:59.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.874+0000 7f4685400640 1 Processor -- start 2024-04-25T01:16:59.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.874+0000 7f4685400640 1 -- start start 2024-04-25T01:16:59.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.875+0000 7f4685400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f46800cedb0 0x7f468015e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:59.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.875+0000 7f4685400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46800d00d0 con 0x7f46800700c0 2024-04-25T01:16:59.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.875+0000 7f4685400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46800d03b0 con 0x7f46800cdc50 2024-04-25T01:16:59.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.875+0000 7f4685400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f46800cd350 con 0x7f46800cedb0 2024-04-25T01:16:59.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.876+0000 7f4677e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f46800700c0 0x7f468015d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:35716/0 (socket says 172.21.15.90:35716) 2024-04-25T01:16:59.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.876+0000 7f4677e00640 1 -- 172.21.15.90:0/2059752563 learned_addr learned my addr 172.21.15.90:0/2059752563 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:16:59.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.876+0000 7f467f200640 1 --2- 172.21.15.90:0/2059752563 >> v2:172.21.15.90:3300/0 conn(0x7f46800cedb0 0x7f468015e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:59.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.876+0000 7f467f200640 1 -- 172.21.15.90:0/2059752563 >> v1:172.21.15.103:6789/0 conn(0x7f46800cdc50 legacy=0x7f46800ce6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:16:59.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.876+0000 7f467f200640 1 -- 172.21.15.90:0/2059752563 >> v1:172.21.15.90:6789/0 conn(0x7f46800700c0 legacy=0x7f468015d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:59.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.876+0000 7f467f200640 1 -- 172.21.15.90:0/2059752563 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46800d0690 con 0x7f46800cedb0 2024-04-25T01:16:59.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.877+0000 7f467f200640 1 --2- 172.21.15.90:0/2059752563 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46800cedb0 0x7f468015e070 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4678009d90 tx=0x7f467802a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12cff0a1d83ec9c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:59.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.878+0000 7f467de00640 1 -- 172.21.15.90:0/2059752563 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4678007740 con 0x7f46800cedb0 2024-04-25T01:16:59.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.878+0000 7f467de00640 1 -- 172.21.15.90:0/2059752563 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46780078e0 con 0x7f46800cedb0 2024-04-25T01:16:59.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.878+0000 7f467de00640 1 -- 172.21.15.90:0/2059752563 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4678007cb0 con 0x7f46800cedb0 2024-04-25T01:16:59.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.878+0000 7f4685400640 1 -- 172.21.15.90:0/2059752563 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46800cedb0 msgr2=0x7f468015e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:59.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.878+0000 7f4685400640 1 --2- 172.21.15.90:0/2059752563 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46800cedb0 0x7f468015e070 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4678009d90 tx=0x7f467802a560 comp rx=0 tx=0).stop 2024-04-25T01:16:59.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.878+0000 7f4685400640 1 -- 172.21.15.90:0/2059752563 shutdown_connections 2024-04-25T01:16:59.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.878+0000 7f4685400640 1 --2- 172.21.15.90:0/2059752563 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46800cedb0 0x7f468015e070 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:59.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.878+0000 7f4685400640 1 -- 172.21.15.90:0/2059752563 >> 172.21.15.90:0/2059752563 conn(0x7f46800d1cf0 msgr2=0x7f46800d2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:16:59.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.878+0000 7f4685400640 1 -- 172.21.15.90:0/2059752563 shutdown_connections 2024-04-25T01:16:59.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.879+0000 7f4685400640 1 -- 172.21.15.90:0/2059752563 wait complete. 2024-04-25T01:16:59.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.880+0000 7f4685400640 1 Processor -- start 2024-04-25T01:16:59.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.880+0000 7f4685400640 1 -- start start 2024-04-25T01:16:59.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.881+0000 7f4685400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46800700c0 0x7f46801e7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:59.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.881+0000 7f4685400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46800cdc50 0x7f468009f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:59.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.881+0000 7f4685400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46800cedb0 0x7f468014eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:59.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.881+0000 7f4685400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f46800d1170 con 0x7f46800cedb0 2024-04-25T01:16:59.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.881+0000 7f4685400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f46800d0c50 con 0x7f46800700c0 2024-04-25T01:16:59.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.881+0000 7f4685400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f468014f530 con 0x7f46800cdc50 2024-04-25T01:16:59.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.882+0000 7f467f200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46800cedb0 0x7f468014eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:59.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.882+0000 7f467f200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46800cedb0 0x7f468014eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56402/0 (socket says 172.21.15.90:56402) 2024-04-25T01:16:59.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.882+0000 7f467f200640 1 -- 172.21.15.90:0/1611979575 learned_addr learned my addr 172.21.15.90:0/1611979575 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:16:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.882+0000 7f467e800640 1 --2- 172.21.15.90:0/1611979575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46800700c0 0x7f46801e7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.882+0000 7f4677e00640 1 --2- 172.21.15.90:0/1611979575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46800cdc50 0x7f468009f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.882+0000 7f467f200640 1 -- 172.21.15.90:0/1611979575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46800cdc50 msgr2=0x7f468009f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.882+0000 7f467f200640 1 --2- 172.21.15.90:0/1611979575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46800cdc50 0x7f468009f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.882+0000 7f467f200640 1 -- 172.21.15.90:0/1611979575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46800700c0 msgr2=0x7f46801e7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:16:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.882+0000 7f467f200640 1 --2- 172.21.15.90:0/1611979575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46800700c0 0x7f46801e7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:16:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.882+0000 7f467f200640 1 -- 172.21.15.90:0/1611979575 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f468009ba90 con 0x7f46800cedb0 2024-04-25T01:16:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.882+0000 7f4677e00640 1 --2- 172.21.15.90:0/1611979575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46800cdc50 0x7f468009f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:16:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.882+0000 7f467f200640 1 --2- 172.21.15.90:0/1611979575 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46800cedb0 0x7f468014eff0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4678009d90 tx=0x7f4678004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.883+0000 7f4676a00640 1 -- 172.21.15.90:0/1611979575 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f467803f070 con 0x7f46800cedb0 2024-04-25T01:16:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.883+0000 7f4676a00640 1 -- 172.21.15.90:0/1611979575 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4678031c80 con 0x7f46800cedb0 2024-04-25T01:16:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.883+0000 7f4676a00640 1 -- 172.21.15.90:0/1611979575 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4678003b90 con 0x7f46800cedb0 2024-04-25T01:16:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.883+0000 7f4685400640 1 -- 172.21.15.90:0/1611979575 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46801e8300 con 0x7f46800cedb0 2024-04-25T01:16:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.883+0000 7f4685400640 1 -- 172.21.15.90:0/1611979575 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46801e88c0 con 0x7f46800cedb0 2024-04-25T01:16:59.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.884+0000 7f4685400640 1 -- 172.21.15.90:0/1611979575 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4680068fb0 con 0x7f46800cedb0 2024-04-25T01:16:59.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.885+0000 7f4676a00640 1 -- 172.21.15.90:0/1611979575 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4678004030 con 0x7f46800cedb0 2024-04-25T01:16:59.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.886+0000 7f4676a00640 1 --2- 172.21.15.90:0/1611979575 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f465c06be30 0x7f465c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:16:59.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.886+0000 7f4676a00640 1 -- 172.21.15.90:0/1611979575 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f46780ac460 con 0x7f46800cedb0 2024-04-25T01:16:59.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.886+0000 7f467e800640 1 --2- 172.21.15.90:0/1611979575 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f465c06be30 0x7f465c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:16:59.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.887+0000 7f467e800640 1 --2- 172.21.15.90:0/1611979575 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f465c06be30 0x7f465c06e2f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f46700015a0 tx=0x7f4670009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:16:59.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:16:59.890+0000 7f4676a00640 1 -- 172.21.15.90:0/1611979575 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f467807aac0 con 0x7f46800cedb0 2024-04-25T01:17:00.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.090+0000 7f4685400640 1 -- 172.21.15.90:0/1611979575 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f46800b7c70 con 0x7f46800cedb0 2024-04-25T01:17:00.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.091+0000 7f4676a00640 1 -- 172.21.15.90:0/1611979575 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f467807a3c0 con 0x7f46800cedb0 2024-04-25T01:17:00.075 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:17:00.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.093+0000 7f4685400640 1 -- 172.21.15.90:0/1611979575 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f465c06be30 msgr2=0x7f465c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:00.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.093+0000 7f4685400640 1 --2- 172.21.15.90:0/1611979575 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f465c06be30 0x7f465c06e2f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f46700015a0 tx=0x7f4670009040 comp rx=0 tx=0).stop 2024-04-25T01:17:00.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.093+0000 7f4685400640 1 -- 172.21.15.90:0/1611979575 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46800cedb0 msgr2=0x7f468014eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:00.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.093+0000 7f4685400640 1 --2- 172.21.15.90:0/1611979575 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46800cedb0 0x7f468014eff0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4678009d90 tx=0x7f4678004a40 comp rx=0 tx=0).stop 2024-04-25T01:17:00.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.094+0000 7f4685400640 1 -- 172.21.15.90:0/1611979575 shutdown_connections 2024-04-25T01:17:00.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.094+0000 7f4685400640 1 --2- 172.21.15.90:0/1611979575 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f465c06be30 0x7f465c06e2f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:00.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.094+0000 7f4685400640 1 --2- 172.21.15.90:0/1611979575 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46800cedb0 0x7f468014eff0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:00.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.094+0000 7f4685400640 1 --2- 172.21.15.90:0/1611979575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f46800cdc50 0x7f468009f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:00.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.094+0000 7f4685400640 1 --2- 172.21.15.90:0/1611979575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46800700c0 0x7f46801e7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:00.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.094+0000 7f4685400640 1 -- 172.21.15.90:0/1611979575 >> 172.21.15.90:0/1611979575 conn(0x7f46800d1cf0 msgr2=0x7f46800b0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:00.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.094+0000 7f4685400640 1 -- 172.21.15.90:0/1611979575 shutdown_connections 2024-04-25T01:17:00.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.094+0000 7f4685400640 1 -- 172.21.15.90:0/1611979575 wait complete. 2024-04-25T01:17:00.097 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:17:00.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.319+0000 7fe983200640 1 Processor -- start 2024-04-25T01:17:00.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.319+0000 7fe983200640 1 -- start start 2024-04-25T01:17:00.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.321+0000 7fe983200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe98415e0e0 0x7fe9840d0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:00.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.321+0000 7fe983200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9840d49b0 0x7fe9840d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:00.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.321+0000 7fe983200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe9840cf980 0x7fe9840d27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:00.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.321+0000 7fe983200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fe9840cf4f0 con 0x7fe9840cf980 2024-04-25T01:17:00.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.321+0000 7fe983200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe98415e4c0 con 0x7fe98415e0e0 2024-04-25T01:17:00.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.321+0000 7fe983200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe9840cebc0 con 0x7fe9840d49b0 2024-04-25T01:17:00.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.321+0000 7fe982800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe9840cf980 0x7fe9840d27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:00.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.321+0000 7fe982800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe9840cf980 0x7fe9840d27f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56416/0 (socket says 172.21.15.90:56416) 2024-04-25T01:17:00.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.321+0000 7fe982800640 1 -- 172.21.15.90:0/4236083465 learned_addr learned my addr 172.21.15.90:0/4236083465 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:00.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.322+0000 7fe981e00640 1 --2- 172.21.15.90:0/4236083465 >> v2:172.21.15.103:3300/0 conn(0x7fe98415e0e0 0x7fe9840d0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:00.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.322+0000 7fe981400640 1 --2- 172.21.15.90:0/4236083465 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9840d49b0 0x7fe9840d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:00.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.322+0000 7fe982800640 1 -- 172.21.15.90:0/4236083465 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9840d49b0 msgr2=0x7fe9840d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:00.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.322+0000 7fe982800640 1 --2- 172.21.15.90:0/4236083465 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9840d49b0 0x7fe9840d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:00.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.322+0000 7fe982800640 1 -- 172.21.15.90:0/4236083465 >> v2:172.21.15.103:3300/0 conn(0x7fe98415e0e0 msgr2=0x7fe9840d0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:00.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.322+0000 7fe982800640 1 --2- 172.21.15.90:0/4236083465 >> v2:172.21.15.103:3300/0 conn(0x7fe98415e0e0 0x7fe9840d0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:00.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.322+0000 7fe982800640 1 -- 172.21.15.90:0/4236083465 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9840ceea0 con 0x7fe9840cf980 2024-04-25T01:17:00.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.322+0000 7fe981e00640 1 --2- 172.21.15.90:0/4236083465 >> v2:172.21.15.103:3300/0 conn(0x7fe98415e0e0 0x7fe9840d0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:17:00.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.323+0000 7fe982800640 1 --2- 172.21.15.90:0/4236083465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9840cf980 0x7fe9840d27f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe98408d620 tx=0x7fe97c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b15c149e632dd46f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:00.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.323+0000 7fe980a00640 1 -- 172.21.15.90:0/4236083465 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97c037070 con 0x7fe9840cf980 2024-04-25T01:17:00.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.323+0000 7fe980a00640 1 -- 172.21.15.90:0/4236083465 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe97c032440 con 0x7fe9840cf980 2024-04-25T01:17:00.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.323+0000 7fe980a00640 1 -- 172.21.15.90:0/4236083465 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97c032810 con 0x7fe9840cf980 2024-04-25T01:17:00.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.324+0000 7fe983200640 1 -- 172.21.15.90:0/4236083465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9840cf980 msgr2=0x7fe9840d27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:00.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.324+0000 7fe983200640 1 --2- 172.21.15.90:0/4236083465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9840cf980 0x7fe9840d27f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe98408d620 tx=0x7fe97c02a1e0 comp rx=0 tx=0).stop 2024-04-25T01:17:00.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.324+0000 7fe983200640 1 -- 172.21.15.90:0/4236083465 shutdown_connections 2024-04-25T01:17:00.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.324+0000 7fe983200640 1 --2- 172.21.15.90:0/4236083465 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9840d49b0 0x7fe9840d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:00.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.324+0000 7fe983200640 1 --2- 172.21.15.90:0/4236083465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9840cf980 0x7fe9840d27f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:00.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.324+0000 7fe983200640 1 --2- 172.21.15.90:0/4236083465 >> v2:172.21.15.103:3300/0 conn(0x7fe98415e0e0 0x7fe9840d0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:00.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.324+0000 7fe983200640 1 -- 172.21.15.90:0/4236083465 >> 172.21.15.90:0/4236083465 conn(0x7fe9840d3cd0 msgr2=0x7fe9840d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:00.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.324+0000 7fe983200640 1 -- 172.21.15.90:0/4236083465 shutdown_connections 2024-04-25T01:17:00.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.324+0000 7fe983200640 1 -- 172.21.15.90:0/4236083465 wait complete. 2024-04-25T01:17:00.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.326+0000 7fe983200640 1 Processor -- start 2024-04-25T01:17:00.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.326+0000 7fe983200640 1 -- start start 2024-04-25T01:17:00.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.327+0000 7fe983200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9840cf980 0x7fe9841f4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:00.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.327+0000 7fe983200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9840d49b0 0x7fe9840ca030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:00.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.327+0000 7fe983200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe98415e0e0 0x7fe9840b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:00.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.327+0000 7fe983200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe9840ce960 con 0x7fe9840d49b0 2024-04-25T01:17:00.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.327+0000 7fe983200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe9840ce7e0 con 0x7fe9840cf980 2024-04-25T01:17:00.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.327+0000 7fe983200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe9840d1250 con 0x7fe98415e0e0 2024-04-25T01:17:00.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.327+0000 7fe981400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9840d49b0 0x7fe9840ca030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:00.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.327+0000 7fe981400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9840d49b0 0x7fe9840ca030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56424/0 (socket says 172.21.15.90:56424) 2024-04-25T01:17:00.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.327+0000 7fe981400640 1 -- 172.21.15.90:0/701567033 learned_addr learned my addr 172.21.15.90:0/701567033 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:00.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.328+0000 7fe981e00640 1 --2- 172.21.15.90:0/701567033 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9840cf980 0x7fe9841f4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:00.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.328+0000 7fe982800640 1 --2- 172.21.15.90:0/701567033 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe98415e0e0 0x7fe9840b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:00.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.328+0000 7fe981400640 1 -- 172.21.15.90:0/701567033 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe98415e0e0 msgr2=0x7fe9840b4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:00.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.328+0000 7fe981400640 1 --2- 172.21.15.90:0/701567033 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe98415e0e0 0x7fe9840b4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:00.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.328+0000 7fe981400640 1 -- 172.21.15.90:0/701567033 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9840cf980 msgr2=0x7fe9841f4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:00.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.328+0000 7fe981400640 1 --2- 172.21.15.90:0/701567033 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9840cf980 0x7fe9841f4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:00.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.328+0000 7fe981400640 1 -- 172.21.15.90:0/701567033 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9840b7c20 con 0x7fe9840d49b0 2024-04-25T01:17:00.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.328+0000 7fe982800640 1 --2- 172.21.15.90:0/701567033 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe98415e0e0 0x7fe9840b4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:00.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.328+0000 7fe981400640 1 --2- 172.21.15.90:0/701567033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9840d49b0 0x7fe9840ca030 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe96c007e90 tx=0x7fe96c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:00.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.328+0000 7fe97a000640 1 -- 172.21.15.90:0/701567033 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c014620 con 0x7fe9840d49b0 2024-04-25T01:17:00.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.328+0000 7fe97a000640 1 -- 172.21.15.90:0/701567033 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe96c019070 con 0x7fe9840d49b0 2024-04-25T01:17:00.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.328+0000 7fe983200640 1 -- 172.21.15.90:0/701567033 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9840b7f00 con 0x7fe9840d49b0 2024-04-25T01:17:00.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.328+0000 7fe97a000640 1 -- 172.21.15.90:0/701567033 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c014a00 con 0x7fe9840d49b0 2024-04-25T01:17:00.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.329+0000 7fe983200640 1 -- 172.21.15.90:0/701567033 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9840b84c0 con 0x7fe9840d49b0 2024-04-25T01:17:00.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.330+0000 7fe983200640 1 -- 172.21.15.90:0/701567033 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe984075350 con 0x7fe9840d49b0 2024-04-25T01:17:00.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.331+0000 7fe97a000640 1 -- 172.21.15.90:0/701567033 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe96c014ba0 con 0x7fe9840d49b0 2024-04-25T01:17:00.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.332+0000 7fe97a000640 1 --2- 172.21.15.90:0/701567033 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe96006be30 0x7fe96006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:00.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.332+0000 7fe97a000640 1 -- 172.21.15.90:0/701567033 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fe96c08ba80 con 0x7fe9840d49b0 2024-04-25T01:17:00.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.332+0000 7fe981e00640 1 --2- 172.21.15.90:0/701567033 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe96006be30 0x7fe96006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:00.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.332+0000 7fe981e00640 1 --2- 172.21.15.90:0/701567033 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe96006be30 0x7fe96006e2f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe974006fd0 tx=0x7fe974008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:00.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.336+0000 7fe97a000640 1 -- 172.21.15.90:0/701567033 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe96c08de20 con 0x7fe9840d49b0 2024-04-25T01:17:00.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.527+0000 7fe983200640 1 -- 172.21.15.90:0/701567033 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe9840b3330 con 0x7fe9840d49b0 2024-04-25T01:17:00.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.528+0000 7fe97a000640 1 -- 172.21.15.90:0/701567033 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7fe96c05a0e0 con 0x7fe9840d49b0 2024-04-25T01:17:00.512 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-25T01:17:00.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.531+0000 7fe983200640 1 -- 172.21.15.90:0/701567033 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe96006be30 msgr2=0x7fe96006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:00.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.531+0000 7fe983200640 1 --2- 172.21.15.90:0/701567033 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe96006be30 0x7fe96006e2f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe974006fd0 tx=0x7fe974008040 comp rx=0 tx=0).stop 2024-04-25T01:17:00.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.531+0000 7fe983200640 1 -- 172.21.15.90:0/701567033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9840d49b0 msgr2=0x7fe9840ca030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:00.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.531+0000 7fe983200640 1 --2- 172.21.15.90:0/701567033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9840d49b0 0x7fe9840ca030 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe96c007e90 tx=0x7fe96c00e4b0 comp rx=0 tx=0).stop 2024-04-25T01:17:00.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.531+0000 7fe983200640 1 -- 172.21.15.90:0/701567033 shutdown_connections 2024-04-25T01:17:00.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.531+0000 7fe983200640 1 --2- 172.21.15.90:0/701567033 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe96006be30 0x7fe96006e2f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:00.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.531+0000 7fe983200640 1 --2- 172.21.15.90:0/701567033 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe98415e0e0 0x7fe9840b4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:00.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.531+0000 7fe983200640 1 --2- 172.21.15.90:0/701567033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9840d49b0 0x7fe9840ca030 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:00.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.531+0000 7fe983200640 1 --2- 172.21.15.90:0/701567033 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9840cf980 0x7fe9841f4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:00.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.531+0000 7fe983200640 1 -- 172.21.15.90:0/701567033 >> 172.21.15.90:0/701567033 conn(0x7fe9840d3cd0 msgr2=0x7fe9840b28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:00.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.531+0000 7fe983200640 1 -- 172.21.15.90:0/701567033 shutdown_connections 2024-04-25T01:17:00.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.531+0000 7fe983200640 1 -- 172.21.15.90:0/701567033 wait complete. 2024-04-25T01:17:00.534 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:17:00.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.749+0000 7f6d87000640 1 Processor -- start 2024-04-25T01:17:00.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.749+0000 7f6d87000640 1 -- start start 2024-04-25T01:17:00.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.751+0000 7f6d87000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d80158680 0x7f6d8006df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:00.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.751+0000 7f6d87000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6d8015f4c0 0x7f6d800b3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:00.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.751+0000 7f6d87000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d8006d140 con 0x7f6d80158a60 2024-04-25T01:17:00.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.751+0000 7f6d87000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f6d8006d420 con 0x7f6d8015f4c0 2024-04-25T01:17:00.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.751+0000 7f6d87000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6d8006d700 con 0x7f6d80158680 2024-04-25T01:17:00.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.751+0000 7f6d85000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6d8015f4c0 0x7f6d800b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:00.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.751+0000 7f6d7f400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f6d80158a60 0x7f6d8006e6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:35722/0 (socket says 172.21.15.90:35722) 2024-04-25T01:17:00.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.751+0000 7f6d7f400640 1 -- 172.21.15.90:0/3548137977 learned_addr learned my addr 172.21.15.90:0/3548137977 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:00.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.752+0000 7f6d7fe00640 1 --2- 172.21.15.90:0/3548137977 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d80158680 0x7f6d8006df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:00.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.752+0000 7f6d85000640 1 -- 172.21.15.90:0/3548137977 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d80158680 msgr2=0x7f6d8006df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:00.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.752+0000 7f6d85000640 1 --2- 172.21.15.90:0/3548137977 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d80158680 0x7f6d8006df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:00.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.752+0000 7f6d85000640 1 -- 172.21.15.90:0/3548137977 >> v1:172.21.15.90:6789/0 conn(0x7f6d80158a60 legacy=0x7f6d8006e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:00.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.752+0000 7f6d85000640 1 -- 172.21.15.90:0/3548137977 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d8006d9e0 con 0x7f6d8015f4c0 2024-04-25T01:17:00.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.752+0000 7f6d7ea00640 1 -- 172.21.15.90:0/3548137977 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 412286621 0 0) 0x7f6d8006d140 con 0x7f6d80158a60 2024-04-25T01:17:00.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.753+0000 7f6d85000640 1 --2- 172.21.15.90:0/3548137977 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d8015f4c0 0x7f6d800b3550 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6d78009d90 tx=0x7f6d7802a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86361647790b5c61 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:00.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.753+0000 7f6d7ea00640 1 -- 172.21.15.90:0/3548137977 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d78031600 con 0x7f6d8015f4c0 2024-04-25T01:17:00.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.753+0000 7f6d7ea00640 1 -- 172.21.15.90:0/3548137977 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d780317a0 con 0x7f6d8015f4c0 2024-04-25T01:17:00.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.753+0000 7f6d7ea00640 1 -- 172.21.15.90:0/3548137977 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d78031b70 con 0x7f6d8015f4c0 2024-04-25T01:17:00.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.754+0000 7f6d87000640 1 -- 172.21.15.90:0/3548137977 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d8015f4c0 msgr2=0x7f6d800b3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:00.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.754+0000 7f6d87000640 1 --2- 172.21.15.90:0/3548137977 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d8015f4c0 0x7f6d800b3550 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6d78009d90 tx=0x7f6d7802a5d0 comp rx=0 tx=0).stop 2024-04-25T01:17:00.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.754+0000 7f6d87000640 1 -- 172.21.15.90:0/3548137977 shutdown_connections 2024-04-25T01:17:00.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.754+0000 7f6d87000640 1 --2- 172.21.15.90:0/3548137977 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d8015f4c0 0x7f6d800b3550 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:00.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.754+0000 7f6d87000640 1 --2- 172.21.15.90:0/3548137977 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d80158680 0x7f6d8006df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:00.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.754+0000 7f6d87000640 1 -- 172.21.15.90:0/3548137977 >> 172.21.15.90:0/3548137977 conn(0x7f6d8006b720 msgr2=0x7f6d800b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:00.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.754+0000 7f6d87000640 1 -- 172.21.15.90:0/3548137977 shutdown_connections 2024-04-25T01:17:00.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.754+0000 7f6d87000640 1 -- 172.21.15.90:0/3548137977 wait complete. 2024-04-25T01:17:00.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.756+0000 7f6d87000640 1 Processor -- start 2024-04-25T01:17:00.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.756+0000 7f6d87000640 1 -- start start 2024-04-25T01:17:00.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.757+0000 7f6d87000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d80158680 0x7f6d80160bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:00.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.757+0000 7f6d87000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d80158a60 0x7f6d8014cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:00.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.757+0000 7f6d87000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d8015f4c0 0x7f6d8014f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:00.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.757+0000 7f6d87000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6d800b3c10 con 0x7f6d8015f4c0 2024-04-25T01:17:00.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.757+0000 7f6d87000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6d800b3a90 con 0x7f6d80158680 2024-04-25T01:17:00.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.757+0000 7f6d87000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6d800b3d90 con 0x7f6d80158a60 2024-04-25T01:17:00.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.757+0000 7f6d85000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d8015f4c0 0x7f6d8014f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:00.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.757+0000 7f6d85000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d8015f4c0 0x7f6d8014f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56434/0 (socket says 172.21.15.90:56434) 2024-04-25T01:17:00.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.757+0000 7f6d85000640 1 -- 172.21.15.90:0/1557019894 learned_addr learned my addr 172.21.15.90:0/1557019894 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:00.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.757+0000 7f6d7fe00640 1 --2- 172.21.15.90:0/1557019894 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d80158680 0x7f6d80160bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:00.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.757+0000 7f6d7f400640 1 --2- 172.21.15.90:0/1557019894 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d80158a60 0x7f6d8014cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:00.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.758+0000 7f6d85000640 1 -- 172.21.15.90:0/1557019894 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d80158a60 msgr2=0x7f6d8014cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:00.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.758+0000 7f6d85000640 1 --2- 172.21.15.90:0/1557019894 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d80158a60 0x7f6d8014cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:00.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.758+0000 7f6d85000640 1 -- 172.21.15.90:0/1557019894 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d80158680 msgr2=0x7f6d80160bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:00.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.758+0000 7f6d85000640 1 --2- 172.21.15.90:0/1557019894 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d80158680 0x7f6d80160bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:00.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.758+0000 7f6d85000640 1 -- 172.21.15.90:0/1557019894 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d80156ea0 con 0x7f6d8015f4c0 2024-04-25T01:17:00.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.758+0000 7f6d7fe00640 1 --2- 172.21.15.90:0/1557019894 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d80158680 0x7f6d80160bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:00.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.758+0000 7f6d85000640 1 --2- 172.21.15.90:0/1557019894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d8015f4c0 0x7f6d8014f750 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6d78009d90 tx=0x7f6d78004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:00.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.758+0000 7f6d77e00640 1 -- 172.21.15.90:0/1557019894 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d7803f070 con 0x7f6d8015f4c0 2024-04-25T01:17:00.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.758+0000 7f6d77e00640 1 -- 172.21.15.90:0/1557019894 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d78030be0 con 0x7f6d8015f4c0 2024-04-25T01:17:00.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.758+0000 7f6d77e00640 1 -- 172.21.15.90:0/1557019894 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d78009450 con 0x7f6d8015f4c0 2024-04-25T01:17:00.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.758+0000 7f6d87000640 1 -- 172.21.15.90:0/1557019894 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d80157180 con 0x7f6d8015f4c0 2024-04-25T01:17:00.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.758+0000 7f6d87000640 1 -- 172.21.15.90:0/1557019894 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d8014fc90 con 0x7f6d8015f4c0 2024-04-25T01:17:00.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.759+0000 7f6d87000640 1 -- 172.21.15.90:0/1557019894 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d80068fb0 con 0x7f6d8015f4c0 2024-04-25T01:17:00.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.760+0000 7f6d77e00640 1 -- 172.21.15.90:0/1557019894 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6d78003990 con 0x7f6d8015f4c0 2024-04-25T01:17:00.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.762+0000 7f6d77e00640 1 --2- 172.21.15.90:0/1557019894 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f6d6006be60 0x7f6d6006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:00.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.762+0000 7f6d77e00640 1 -- 172.21.15.90:0/1557019894 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f6d780ac0e0 con 0x7f6d8015f4c0 2024-04-25T01:17:00.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.762+0000 7f6d7fe00640 1 --2- 172.21.15.90:0/1557019894 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f6d6006be60 0x7f6d6006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:00.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.762+0000 7f6d7fe00640 1 --2- 172.21.15.90:0/1557019894 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f6d6006be60 0x7f6d6006e320 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6d70006fd0 tx=0x7f6d70008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:00.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.766+0000 7f6d77e00640 1 -- 172.21.15.90:0/1557019894 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6d7807a7e0 con 0x7f6d8015f4c0 2024-04-25T01:17:00.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.965+0000 7f6d87000640 1 -- 172.21.15.90:0/1557019894 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6d80151c80 con 0x7f6d8015f4c0 2024-04-25T01:17:00.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.966+0000 7f6d77e00640 1 -- 172.21.15.90:0/1557019894 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f6d7807a0e0 con 0x7f6d8015f4c0 2024-04-25T01:17:00.950 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:17:00.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.969+0000 7f6d87000640 1 -- 172.21.15.90:0/1557019894 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f6d6006be60 msgr2=0x7f6d6006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:00.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.969+0000 7f6d87000640 1 --2- 172.21.15.90:0/1557019894 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f6d6006be60 0x7f6d6006e320 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6d70006fd0 tx=0x7f6d70008040 comp rx=0 tx=0).stop 2024-04-25T01:17:00.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.969+0000 7f6d87000640 1 -- 172.21.15.90:0/1557019894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d8015f4c0 msgr2=0x7f6d8014f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:00.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.969+0000 7f6d87000640 1 --2- 172.21.15.90:0/1557019894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d8015f4c0 0x7f6d8014f750 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6d78009d90 tx=0x7f6d78004a40 comp rx=0 tx=0).stop 2024-04-25T01:17:00.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.969+0000 7f6d87000640 1 -- 172.21.15.90:0/1557019894 shutdown_connections 2024-04-25T01:17:00.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.969+0000 7f6d87000640 1 --2- 172.21.15.90:0/1557019894 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f6d6006be60 0x7f6d6006e320 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:00.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.969+0000 7f6d87000640 1 --2- 172.21.15.90:0/1557019894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d8015f4c0 0x7f6d8014f750 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:00.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.969+0000 7f6d87000640 1 --2- 172.21.15.90:0/1557019894 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6d80158a60 0x7f6d8014cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:00.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.969+0000 7f6d87000640 1 --2- 172.21.15.90:0/1557019894 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d80158680 0x7f6d80160bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:00.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.969+0000 7f6d87000640 1 -- 172.21.15.90:0/1557019894 >> 172.21.15.90:0/1557019894 conn(0x7f6d8006b720 msgr2=0x7f6d800b1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:00.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.969+0000 7f6d87000640 1 -- 172.21.15.90:0/1557019894 shutdown_connections 2024-04-25T01:17:00.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:00.969+0000 7f6d87000640 1 -- 172.21.15.90:0/1557019894 wait complete. 2024-04-25T01:17:00.970 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:17:01.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.191+0000 7f3d73e00640 1 Processor -- start 2024-04-25T01:17:01.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.191+0000 7f3d73e00640 1 -- start start 2024-04-25T01:17:01.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.192+0000 7f3d73e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3d7406b120 0x7f3d7406b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:01.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.193+0000 7f3d73e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d7406ba40 con 0x7f3d740ae730 2024-04-25T01:17:01.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.193+0000 7f3d73e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d7415c220 con 0x7f3d7415ca10 2024-04-25T01:17:01.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.193+0000 7f3d73e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f3d740cef40 con 0x7f3d7406b120 2024-04-25T01:17:01.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.193+0000 7f3d6be00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f3d740ae730 0x7f3d740cd900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:35730/0 (socket says 172.21.15.90:35730) 2024-04-25T01:17:01.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.193+0000 7f3d6be00640 1 -- 172.21.15.90:0/1412478312 learned_addr learned my addr 172.21.15.90:0/1412478312 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:01.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.193+0000 7f3d72a00640 1 --2- 172.21.15.90:0/1412478312 >> v2:172.21.15.103:3300/0 conn(0x7f3d7406b120 0x7f3d7406b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:01.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.193+0000 7f3d72000640 1 -- 172.21.15.90:0/1412478312 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1322075033 0 0) 0x7f3d7406ba40 con 0x7f3d740ae730 2024-04-25T01:17:01.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.193+0000 7f3d72000640 1 -- 172.21.15.90:0/1412478312 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d58003520 con 0x7f3d740ae730 2024-04-25T01:17:01.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.194+0000 7f3d72000640 1 -- 172.21.15.90:0/1412478312 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 629602399 0 0) 0x7f3d58003520 con 0x7f3d740ae730 2024-04-25T01:17:01.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.194+0000 7f3d72000640 1 -- 172.21.15.90:0/1412478312 >> v2:172.21.15.103:3300/0 conn(0x7f3d7406b120 msgr2=0x7f3d7406b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:01.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.194+0000 7f3d72000640 1 --2- 172.21.15.90:0/1412478312 >> v2:172.21.15.103:3300/0 conn(0x7f3d7406b120 0x7f3d7406b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:01.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.194+0000 7f3d72000640 1 -- 172.21.15.90:0/1412478312 >> v1:172.21.15.103:6789/0 conn(0x7f3d7415ca10 legacy=0x7f3d740cc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:01.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.194+0000 7f3d72000640 1 -- 172.21.15.90:0/1412478312 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d740aec80 con 0x7f3d740ae730 2024-04-25T01:17:01.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.194+0000 7f3d72000640 1 -- 172.21.15.90:0/1412478312 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f3d5c001670 con 0x7f3d740ae730 2024-04-25T01:17:01.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.194+0000 7f3d72000640 1 -- 172.21.15.90:0/1412478312 >> v1:172.21.15.90:6789/0 conn(0x7f3d740ae730 legacy=0x7f3d740cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:01.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.194+0000 7f3d72000640 1 --2- 172.21.15.90:0/1412478312 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3d580039b0 0x7f3d58003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:01.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.194+0000 7f3d72000640 1 --2- 172.21.15.90:0/1412478312 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d58004800 0x7f3d58004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:01.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.194+0000 7f3d72000640 1 --2- 172.21.15.90:0/1412478312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d580051c0 0x7f3d5800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:01.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.194+0000 7f3d72000640 1 -- 172.21.15.90:0/1412478312 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5800c030 con 0x7f3d58004800 2024-04-25T01:17:01.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.194+0000 7f3d72000640 1 -- 172.21.15.90:0/1412478312 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5800c1e0 con 0x7f3d580051c0 2024-04-25T01:17:01.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.194+0000 7f3d72000640 1 -- 172.21.15.90:0/1412478312 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3d5800c3e0 con 0x7f3d580039b0 2024-04-25T01:17:01.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.194+0000 7f3d6be00640 1 --2- 172.21.15.90:0/1412478312 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d58004800 0x7f3d58004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:01.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.194+0000 7f3d6be00640 1 -- 172.21.15.90:0/1412478312 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3d580039b0 msgr2=0x7f3d58003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:01.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.194+0000 7f3d6be00640 1 --2- 172.21.15.90:0/1412478312 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3d580039b0 0x7f3d58003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:01.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.194+0000 7f3d6be00640 1 -- 172.21.15.90:0/1412478312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d580051c0 msgr2=0x7f3d5800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:01.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.194+0000 7f3d73400640 1 --2- 172.21.15.90:0/1412478312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d580051c0 0x7f3d5800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:01.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.195+0000 7f3d6be00640 1 --2- 172.21.15.90:0/1412478312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d580051c0 0x7f3d5800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:01.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.195+0000 7f3d6be00640 1 -- 172.21.15.90:0/1412478312 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d58003520 con 0x7f3d58004800 2024-04-25T01:17:01.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.195+0000 7f3d73400640 1 -- 172.21.15.90:0/1412478312 reap_dead start 2024-04-25T01:17:01.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.196+0000 7f3d6be00640 1 --2- 172.21.15.90:0/1412478312 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d58004800 0x7f3d58004c80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c00c210 tx=0x7f3d58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c92a36d3404530d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:01.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.196+0000 7f3d72000640 1 -- 172.21.15.90:0/1412478312 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d5c007d30 con 0x7f3d58004800 2024-04-25T01:17:01.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.196+0000 7f3d72000640 1 -- 172.21.15.90:0/1412478312 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d5c005ad0 con 0x7f3d58004800 2024-04-25T01:17:01.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.196+0000 7f3d72000640 1 -- 172.21.15.90:0/1412478312 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d5c007070 con 0x7f3d58004800 2024-04-25T01:17:01.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.196+0000 7f3d73e00640 1 -- 172.21.15.90:0/1412478312 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d58004800 msgr2=0x7f3d58004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:01.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.196+0000 7f3d73e00640 1 --2- 172.21.15.90:0/1412478312 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d58004800 0x7f3d58004c80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c00c210 tx=0x7f3d58005830 comp rx=0 tx=0).stop 2024-04-25T01:17:01.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.196+0000 7f3d73e00640 1 -- 172.21.15.90:0/1412478312 shutdown_connections 2024-04-25T01:17:01.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.196+0000 7f3d73e00640 1 --2- 172.21.15.90:0/1412478312 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d58004800 0x7f3d58004c80 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:01.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.196+0000 7f3d73e00640 1 -- 172.21.15.90:0/1412478312 >> 172.21.15.90:0/1412478312 conn(0x7f3d740b1660 msgr2=0x7f3d740b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:01.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.197+0000 7f3d73e00640 1 -- 172.21.15.90:0/1412478312 shutdown_connections 2024-04-25T01:17:01.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.197+0000 7f3d73e00640 1 -- 172.21.15.90:0/1412478312 wait complete. 2024-04-25T01:17:01.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.198+0000 7f3d73e00640 1 Processor -- start 2024-04-25T01:17:01.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.198+0000 7f3d73e00640 1 -- start start 2024-04-25T01:17:01.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.199+0000 7f3d73e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d58004800 0x7f3d7414d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:01.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.199+0000 7f3d73e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d741532f0 0x7f3d74140ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:01.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.199+0000 7f3d73e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3d74154e50 0x7f3d741552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:01.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.199+0000 7f3d73e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3d740af240 con 0x7f3d58004800 2024-04-25T01:17:01.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.199+0000 7f3d73e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3d7415c580 con 0x7f3d741532f0 2024-04-25T01:17:01.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.199+0000 7f3d73e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3d74153080 con 0x7f3d74154e50 2024-04-25T01:17:01.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.200+0000 7f3d72a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d58004800 0x7f3d7414d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:01.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.200+0000 7f3d72a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d58004800 0x7f3d7414d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56452/0 (socket says 172.21.15.90:56452) 2024-04-25T01:17:01.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.200+0000 7f3d72a00640 1 -- 172.21.15.90:0/3646806997 learned_addr learned my addr 172.21.15.90:0/3646806997 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:01.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.200+0000 7f3d72a00640 1 -- 172.21.15.90:0/3646806997 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3d74154e50 msgr2=0x7f3d741552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:01.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.200+0000 7f3d72a00640 1 --2- 172.21.15.90:0/3646806997 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3d74154e50 0x7f3d741552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:01.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.200+0000 7f3d72a00640 1 -- 172.21.15.90:0/3646806997 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d741532f0 msgr2=0x7f3d74140ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:01.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.200+0000 7f3d72a00640 1 --2- 172.21.15.90:0/3646806997 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d741532f0 0x7f3d74140ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:01.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.200+0000 7f3d72a00640 1 -- 172.21.15.90:0/3646806997 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d7414dcf0 con 0x7f3d58004800 2024-04-25T01:17:01.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.200+0000 7f3d72a00640 1 --2- 172.21.15.90:0/3646806997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d58004800 0x7f3d7414d7b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3d6400e860 tx=0x7f3d6400ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:01.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.200+0000 7f3d6aa00640 1 -- 172.21.15.90:0/3646806997 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d64017070 con 0x7f3d58004800 2024-04-25T01:17:01.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.201+0000 7f3d6aa00640 1 -- 172.21.15.90:0/3646806997 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d6400c8e0 con 0x7f3d58004800 2024-04-25T01:17:01.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.201+0000 7f3d6aa00640 1 -- 172.21.15.90:0/3646806997 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d6400ccb0 con 0x7f3d58004800 2024-04-25T01:17:01.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.201+0000 7f3d73e00640 1 -- 172.21.15.90:0/3646806997 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d7414df90 con 0x7f3d58004800 2024-04-25T01:17:01.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.201+0000 7f3d73e00640 1 -- 172.21.15.90:0/3646806997 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d7414e550 con 0x7f3d58004800 2024-04-25T01:17:01.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.202+0000 7f3d73e00640 1 -- 172.21.15.90:0/3646806997 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d74075350 con 0x7f3d58004800 2024-04-25T01:17:01.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.203+0000 7f3d6aa00640 1 -- 172.21.15.90:0/3646806997 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3d6401a070 con 0x7f3d58004800 2024-04-25T01:17:01.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.204+0000 7f3d6aa00640 1 --2- 172.21.15.90:0/3646806997 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f3d5006be60 0x7f3d5006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:01.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.204+0000 7f3d6aa00640 1 -- 172.21.15.90:0/3646806997 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f3d6408af00 con 0x7f3d58004800 2024-04-25T01:17:01.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.204+0000 7f3d6be00640 1 --2- 172.21.15.90:0/3646806997 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f3d5006be60 0x7f3d5006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:01.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.205+0000 7f3d6be00640 1 --2- 172.21.15.90:0/3646806997 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f3d5006be60 0x7f3d5006e320 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c033000 tx=0x7f3d5c009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:01.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.208+0000 7f3d6aa00640 1 -- 172.21.15.90:0/3646806997 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f3d64059600 con 0x7f3d58004800 2024-04-25T01:17:01.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.444+0000 7f3d73e00640 1 -- 172.21.15.90:0/3646806997 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d74159c20 con 0x7f3d58004800 2024-04-25T01:17:01.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.445+0000 7f3d6aa00640 1 -- 172.21.15.90:0/3646806997 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5180 (secure 0 0 0) 0x7f3d64058f00 con 0x7f3d58004800 2024-04-25T01:17:01.429 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:17:01.429 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6834/2605945289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":2605945289},{"type":"v1","addr":"172.21.15.103:6834","nonce":2605945289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6836/704639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":704639426},{"type":"v1","addr":"172.21.15.90:6836","nonce":704639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4245,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3477867755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6835","nonce":3477867755},{"type":"v1","addr":"172.21.15.90:6837","nonce":3477867755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:16:22.060018+0000","modified":"2024-04-25T01:16:50.405431+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6835/4291578198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6833","nonce":4291578198},{"type":"v1","addr":"172.21.15.103:6835","nonce":4291578198}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1}]} 2024-04-25T01:17:01.429 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 9 2024-04-25T01:17:01.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.447+0000 7f3d73e00640 1 -- 172.21.15.90:0/3646806997 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f3d5006be60 msgr2=0x7f3d5006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:01.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.447+0000 7f3d73e00640 1 --2- 172.21.15.90:0/3646806997 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f3d5006be60 0x7f3d5006e320 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c033000 tx=0x7f3d5c009030 comp rx=0 tx=0).stop 2024-04-25T01:17:01.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.447+0000 7f3d73e00640 1 -- 172.21.15.90:0/3646806997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d58004800 msgr2=0x7f3d7414d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:01.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.447+0000 7f3d73e00640 1 --2- 172.21.15.90:0/3646806997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d58004800 0x7f3d7414d7b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3d6400e860 tx=0x7f3d6400ed20 comp rx=0 tx=0).stop 2024-04-25T01:17:01.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.447+0000 7f3d73e00640 1 -- 172.21.15.90:0/3646806997 shutdown_connections 2024-04-25T01:17:01.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.447+0000 7f3d73e00640 1 --2- 172.21.15.90:0/3646806997 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f3d5006be60 0x7f3d5006e320 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:01.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.447+0000 7f3d73e00640 1 --2- 172.21.15.90:0/3646806997 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3d74154e50 0x7f3d741552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:01.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.447+0000 7f3d73e00640 1 --2- 172.21.15.90:0/3646806997 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d741532f0 0x7f3d74140ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:01.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.447+0000 7f3d73e00640 1 --2- 172.21.15.90:0/3646806997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d58004800 0x7f3d7414d7b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:01.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.447+0000 7f3d73e00640 1 -- 172.21.15.90:0/3646806997 >> 172.21.15.90:0/3646806997 conn(0x7f3d740b1660 msgr2=0x7f3d7406d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:01.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.448+0000 7f3d73e00640 1 -- 172.21.15.90:0/3646806997 shutdown_connections 2024-04-25T01:17:01.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.448+0000 7f3d73e00640 1 -- 172.21.15.90:0/3646806997 wait complete. 2024-04-25T01:17:01.449 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:17:01.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.661+0000 7f8e41c00640 1 Processor -- start 2024-04-25T01:17:01.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.661+0000 7f8e41c00640 1 -- start start 2024-04-25T01:17:01.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.662+0000 7f8e41c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e3c15ad40 0x7f8e3c06b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:01.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.662+0000 7f8e41c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8e3c0cf520 0x7f8e3c0ae730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:01.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.663+0000 7f8e41c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e3c06ba90 con 0x7f8e3c06d910 2024-04-25T01:17:01.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.663+0000 7f8e41c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f8e3c06cb50 con 0x7f8e3c0cf520 2024-04-25T01:17:01.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.663+0000 7f8e41c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8e3c06ce30 con 0x7f8e3c15ad40 2024-04-25T01:17:01.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.663+0000 7f8e33400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8e3c0cf520 0x7f8e3c0ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:01.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.663+0000 7f8e33400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8e3c0cf520 0x7f8e3c0ae730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56464/0 (socket says 172.21.15.90:56464) 2024-04-25T01:17:01.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.663+0000 7f8e33400640 1 -- 172.21.15.90:0/3145240121 learned_addr learned my addr 172.21.15.90:0/3145240121 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:01.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.663+0000 7f8e3be00640 1 --2- 172.21.15.90:0/3145240121 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e3c15ad40 0x7f8e3c06b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:01.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.663+0000 7f8e33400640 1 -- 172.21.15.90:0/3145240121 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e3c15ad40 msgr2=0x7f8e3c06b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:01.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.663+0000 7f8e33400640 1 --2- 172.21.15.90:0/3145240121 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e3c15ad40 0x7f8e3c06b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:01.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.663+0000 7f8e33400640 1 -- 172.21.15.90:0/3145240121 >> v1:172.21.15.103:6789/0 conn(0x7f8e3c06d910 legacy=0x7f8e3c15e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:01.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.663+0000 7f8e33400640 1 -- 172.21.15.90:0/3145240121 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e3c06d110 con 0x7f8e3c0cf520 2024-04-25T01:17:01.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.665+0000 7f8e33400640 1 --2- 172.21.15.90:0/3145240121 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e3c0cf520 0x7f8e3c0ae730 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8e2400bb10 tx=0x7f8e2402c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=134df624797e2981 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:01.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.665+0000 7f8e3b400640 1 -- 172.21.15.90:0/3145240121 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e24032790 con 0x7f8e3c0cf520 2024-04-25T01:17:01.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.665+0000 7f8e3b400640 1 -- 172.21.15.90:0/3145240121 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e24038070 con 0x7f8e3c0cf520 2024-04-25T01:17:01.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.665+0000 7f8e3b400640 1 -- 172.21.15.90:0/3145240121 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e24032b50 con 0x7f8e3c0cf520 2024-04-25T01:17:01.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.665+0000 7f8e41c00640 1 -- 172.21.15.90:0/3145240121 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e3c0cf520 msgr2=0x7f8e3c0ae730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:01.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.665+0000 7f8e41c00640 1 --2- 172.21.15.90:0/3145240121 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e3c0cf520 0x7f8e3c0ae730 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8e2400bb10 tx=0x7f8e2402c8d0 comp rx=0 tx=0).stop 2024-04-25T01:17:01.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.666+0000 7f8e41c00640 1 -- 172.21.15.90:0/3145240121 shutdown_connections 2024-04-25T01:17:01.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.666+0000 7f8e41c00640 1 --2- 172.21.15.90:0/3145240121 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e3c0cf520 0x7f8e3c0ae730 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:01.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.666+0000 7f8e41c00640 1 --2- 172.21.15.90:0/3145240121 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e3c15ad40 0x7f8e3c06b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:01.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.666+0000 7f8e41c00640 1 -- 172.21.15.90:0/3145240121 >> 172.21.15.90:0/3145240121 conn(0x7f8e3c0b1660 msgr2=0x7f8e3c0b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:01.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.666+0000 7f8e41c00640 1 -- 172.21.15.90:0/3145240121 shutdown_connections 2024-04-25T01:17:01.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.666+0000 7f8e41c00640 1 -- 172.21.15.90:0/3145240121 wait complete. 2024-04-25T01:17:01.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.667+0000 7f8e41c00640 1 Processor -- start 2024-04-25T01:17:01.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.668+0000 7f8e41c00640 1 -- start start 2024-04-25T01:17:01.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.668+0000 7f8e41c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e3c06d910 0x7f8e3c154db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:01.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.668+0000 7f8e41c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e3c0cf520 0x7f8e3c15d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:01.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.669+0000 7f8e41c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e3c15ad40 0x7f8e3c153c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:01.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.669+0000 7f8e41c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3c06b7e0 con 0x7f8e3c15ad40 2024-04-25T01:17:01.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.669+0000 7f8e41c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3c06b660 con 0x7f8e3c06d910 2024-04-25T01:17:01.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.669+0000 7f8e41c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8e3c0aec70 con 0x7f8e3c0cf520 2024-04-25T01:17:01.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.669+0000 7f8e41200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e3c15ad40 0x7f8e3c153c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:01.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.669+0000 7f8e41200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e3c15ad40 0x7f8e3c153c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56480/0 (socket says 172.21.15.90:56480) 2024-04-25T01:17:01.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.669+0000 7f8e41200640 1 -- 172.21.15.90:0/3644906278 learned_addr learned my addr 172.21.15.90:0/3644906278 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:01.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.669+0000 7f8e3be00640 1 --2- 172.21.15.90:0/3644906278 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e3c06d910 0x7f8e3c154db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:01.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.669+0000 7f8e33400640 1 --2- 172.21.15.90:0/3644906278 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e3c0cf520 0x7f8e3c15d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:01.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.669+0000 7f8e41200640 1 -- 172.21.15.90:0/3644906278 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e3c0cf520 msgr2=0x7f8e3c15d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:01.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.669+0000 7f8e41200640 1 --2- 172.21.15.90:0/3644906278 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e3c0cf520 0x7f8e3c15d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:01.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.669+0000 7f8e41200640 1 -- 172.21.15.90:0/3644906278 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e3c06d910 msgr2=0x7f8e3c154db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:01.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.669+0000 7f8e41200640 1 --2- 172.21.15.90:0/3644906278 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e3c06d910 0x7f8e3c154db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:01.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.669+0000 7f8e41200640 1 -- 172.21.15.90:0/3644906278 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e3c15c730 con 0x7f8e3c15ad40 2024-04-25T01:17:01.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.670+0000 7f8e41200640 1 --2- 172.21.15.90:0/3644906278 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e3c15ad40 0x7f8e3c153c60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8e3400c7b0 tx=0x7f8e3400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:01.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.670+0000 7f8e38c00640 1 -- 172.21.15.90:0/3644906278 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e34016020 con 0x7f8e3c15ad40 2024-04-25T01:17:01.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.670+0000 7f8e38c00640 1 -- 172.21.15.90:0/3644906278 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e3400aa50 con 0x7f8e3c15ad40 2024-04-25T01:17:01.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.670+0000 7f8e38c00640 1 -- 172.21.15.90:0/3644906278 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3400ce60 con 0x7f8e3c15ad40 2024-04-25T01:17:01.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.670+0000 7f8e41c00640 1 -- 172.21.15.90:0/3644906278 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e3c15ca10 con 0x7f8e3c15ad40 2024-04-25T01:17:01.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.670+0000 7f8e41c00640 1 -- 172.21.15.90:0/3644906278 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e3c15d3a0 con 0x7f8e3c15ad40 2024-04-25T01:17:01.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.671+0000 7f8e41c00640 1 -- 172.21.15.90:0/3644906278 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e3c075350 con 0x7f8e3c15ad40 2024-04-25T01:17:01.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.672+0000 7f8e38c00640 1 -- 172.21.15.90:0/3644906278 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8e34019020 con 0x7f8e3c15ad40 2024-04-25T01:17:01.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.673+0000 7f8e38c00640 1 --2- 172.21.15.90:0/3644906278 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f8e1806be60 0x7f8e1806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:01.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.673+0000 7f8e38c00640 1 -- 172.21.15.90:0/3644906278 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f8e3408b0a0 con 0x7f8e3c15ad40 2024-04-25T01:17:01.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.674+0000 7f8e3be00640 1 --2- 172.21.15.90:0/3644906278 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f8e1806be60 0x7f8e1806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:01.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.674+0000 7f8e3be00640 1 --2- 172.21.15.90:0/3644906278 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f8e1806be60 0x7f8e1806e320 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c006fd0 tx=0x7f8e2c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:01.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.678+0000 7f8e38c00640 1 -- 172.21.15.90:0/3644906278 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8e34020030 con 0x7f8e3c15ad40 2024-04-25T01:17:01.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.878+0000 7f8e41c00640 1 -- 172.21.15.90:0/3644906278 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8e3c06dff0 con 0x7f8e3c15ad40 2024-04-25T01:17:01.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.880+0000 7f8e38c00640 1 -- 172.21.15.90:0/3644906278 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18482 (secure 0 0 0) 0x7f8e340212b0 con 0x7f8e3c15ad40 2024-04-25T01:17:01.864 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:17:01.864 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":26,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:16:27.092855+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:16:20.094512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:16:21.078268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:16:24.670642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:17:01.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.883+0000 7f8e41c00640 1 -- 172.21.15.90:0/3644906278 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f8e1806be60 msgr2=0x7f8e1806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:01.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.883+0000 7f8e41c00640 1 --2- 172.21.15.90:0/3644906278 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f8e1806be60 0x7f8e1806e320 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c006fd0 tx=0x7f8e2c008040 comp rx=0 tx=0).stop 2024-04-25T01:17:01.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.883+0000 7f8e41c00640 1 -- 172.21.15.90:0/3644906278 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e3c15ad40 msgr2=0x7f8e3c153c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:01.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.883+0000 7f8e41c00640 1 --2- 172.21.15.90:0/3644906278 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e3c15ad40 0x7f8e3c153c60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8e3400c7b0 tx=0x7f8e3400cc70 comp rx=0 tx=0).stop 2024-04-25T01:17:01.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.883+0000 7f8e41c00640 1 -- 172.21.15.90:0/3644906278 shutdown_connections 2024-04-25T01:17:01.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.883+0000 7f8e41c00640 1 --2- 172.21.15.90:0/3644906278 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f8e1806be60 0x7f8e1806e320 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:01.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.883+0000 7f8e41c00640 1 --2- 172.21.15.90:0/3644906278 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e3c15ad40 0x7f8e3c153c60 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:01.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.883+0000 7f8e41c00640 1 --2- 172.21.15.90:0/3644906278 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e3c0cf520 0x7f8e3c15d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:01.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.883+0000 7f8e41c00640 1 --2- 172.21.15.90:0/3644906278 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e3c06d910 0x7f8e3c154db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:01.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.883+0000 7f8e41c00640 1 -- 172.21.15.90:0/3644906278 >> 172.21.15.90:0/3644906278 conn(0x7f8e3c0b1660 msgr2=0x7f8e3c0b2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:01.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.883+0000 7f8e41c00640 1 -- 172.21.15.90:0/3644906278 shutdown_connections 2024-04-25T01:17:01.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:01.883+0000 7f8e41c00640 1 -- 172.21.15.90:0/3644906278 wait complete. 2024-04-25T01:17:01.887 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:16:15.894126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:16:20.094512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:16:21.078268+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:16:24.670642+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:17:01.887 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:17:02.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.107+0000 7ff9a6800640 1 Processor -- start 2024-04-25T01:17:02.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.107+0000 7ff9a6800640 1 -- start start 2024-04-25T01:17:02.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.108+0000 7ff9a6800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff9a00cf4d0 0x7ff9a01600a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:02.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.108+0000 7ff9a6800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9a006d110 con 0x7ff9a00722e0 2024-04-25T01:17:02.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.109+0000 7ff9a6800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9a006d3f0 con 0x7ff9a015fc40 2024-04-25T01:17:02.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.109+0000 7ff9a6800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7ff9a0160aa0 con 0x7ff9a00cf4d0 2024-04-25T01:17:02.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.109+0000 7ff99f400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff9a00722e0 0x7ff9a006ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:35736/0 (socket says 172.21.15.90:35736) 2024-04-25T01:17:02.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.109+0000 7ff99f400640 1 -- 172.21.15.90:0/2092477867 learned_addr learned my addr 172.21.15.90:0/2092477867 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:02.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.109+0000 7ff99fe00640 1 --2- 172.21.15.90:0/2092477867 >> v2:172.21.15.90:3300/0 conn(0x7ff9a00cf4d0 0x7ff9a01600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:02.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.109+0000 7ff99ea00640 1 -- 172.21.15.90:0/2092477867 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1478835461 0 0) 0x7ff9a006d110 con 0x7ff9a00722e0 2024-04-25T01:17:02.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.109+0000 7ff99ea00640 1 -- 172.21.15.90:0/2092477867 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff984003520 con 0x7ff9a00722e0 2024-04-25T01:17:02.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.109+0000 7ff99fe00640 1 -- 172.21.15.90:0/2092477867 >> v1:172.21.15.103:6789/0 conn(0x7ff9a015fc40 legacy=0x7ff9a00cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:02.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.109+0000 7ff99fe00640 1 -- 172.21.15.90:0/2092477867 >> v1:172.21.15.90:6789/0 conn(0x7ff9a00722e0 legacy=0x7ff9a006ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:02.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.109+0000 7ff99fe00640 1 -- 172.21.15.90:0/2092477867 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a006d6d0 con 0x7ff9a00cf4d0 2024-04-25T01:17:02.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.111+0000 7ff99fe00640 1 --2- 172.21.15.90:0/2092477867 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9a00cf4d0 0x7ff9a01600a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff998009d90 tx=0x7ff99802a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb15905d9d1fa8dc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:02.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.111+0000 7ff99ea00640 1 -- 172.21.15.90:0/2092477867 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff998031600 con 0x7ff9a00cf4d0 2024-04-25T01:17:02.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.111+0000 7ff99ea00640 1 -- 172.21.15.90:0/2092477867 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9980317a0 con 0x7ff9a00cf4d0 2024-04-25T01:17:02.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.111+0000 7ff99ea00640 1 -- 172.21.15.90:0/2092477867 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff998031b70 con 0x7ff9a00cf4d0 2024-04-25T01:17:02.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.111+0000 7ff9a6800640 1 -- 172.21.15.90:0/2092477867 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9a00cf4d0 msgr2=0x7ff9a01600a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:02.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.111+0000 7ff9a6800640 1 --2- 172.21.15.90:0/2092477867 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9a00cf4d0 0x7ff9a01600a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff998009d90 tx=0x7ff99802a610 comp rx=0 tx=0).stop 2024-04-25T01:17:02.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.112+0000 7ff9a6800640 1 -- 172.21.15.90:0/2092477867 shutdown_connections 2024-04-25T01:17:02.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.112+0000 7ff9a6800640 1 --2- 172.21.15.90:0/2092477867 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9a00cf4d0 0x7ff9a01600a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:02.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.112+0000 7ff9a6800640 1 -- 172.21.15.90:0/2092477867 >> 172.21.15.90:0/2092477867 conn(0x7ff9a006b720 msgr2=0x7ff9a00b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:02.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.112+0000 7ff9a6800640 1 -- 172.21.15.90:0/2092477867 shutdown_connections 2024-04-25T01:17:02.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.112+0000 7ff9a6800640 1 -- 172.21.15.90:0/2092477867 wait complete. 2024-04-25T01:17:02.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.113+0000 7ff9a6800640 1 Processor -- start 2024-04-25T01:17:02.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.113+0000 7ff9a6800640 1 -- start start 2024-04-25T01:17:02.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.114+0000 7ff9a6800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9a00722e0 0x7ff9a00b1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:02.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.114+0000 7ff9a6800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9a00cf4d0 0x7ff9a013f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:02.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.114+0000 7ff9a6800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9a015fc40 0x7ff9a013ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:02.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.114+0000 7ff9a6800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a0070f60 con 0x7ff9a00722e0 2024-04-25T01:17:02.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.114+0000 7ff9a6800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a0070de0 con 0x7ff9a015fc40 2024-04-25T01:17:02.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.114+0000 7ff9a6800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff9a00710e0 con 0x7ff9a00cf4d0 2024-04-25T01:17:02.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.115+0000 7ff99f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9a00722e0 0x7ff9a00b1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:02.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.115+0000 7ff99f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9a00722e0 0x7ff9a00b1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56510/0 (socket says 172.21.15.90:56510) 2024-04-25T01:17:02.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.115+0000 7ff99f400640 1 -- 172.21.15.90:0/1269750461 learned_addr learned my addr 172.21.15.90:0/1269750461 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:02.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.115+0000 7ff99fe00640 1 --2- 172.21.15.90:0/1269750461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9a015fc40 0x7ff9a013ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:02.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.115+0000 7ff99f400640 1 -- 172.21.15.90:0/1269750461 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9a00cf4d0 msgr2=0x7ff9a013f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:02.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.115+0000 7ff99f400640 1 --2- 172.21.15.90:0/1269750461 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9a00cf4d0 0x7ff9a013f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:02.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.115+0000 7ff99f400640 1 -- 172.21.15.90:0/1269750461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9a015fc40 msgr2=0x7ff9a013ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:02.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.115+0000 7ff99f400640 1 --2- 172.21.15.90:0/1269750461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9a015fc40 0x7ff9a013ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:02.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.115+0000 7ff99f400640 1 -- 172.21.15.90:0/1269750461 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a013cfc0 con 0x7ff9a00722e0 2024-04-25T01:17:02.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.115+0000 7ff99f400640 1 --2- 172.21.15.90:0/1269750461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9a00722e0 0x7ff9a00b1270 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff99000c430 tx=0x7ff99000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:02.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.116+0000 7ff997400640 1 -- 172.21.15.90:0/1269750461 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff990017070 con 0x7ff9a00722e0 2024-04-25T01:17:02.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.116+0000 7ff997400640 1 -- 172.21.15.90:0/1269750461 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff990005ce0 con 0x7ff9a00722e0 2024-04-25T01:17:02.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.116+0000 7ff997400640 1 -- 172.21.15.90:0/1269750461 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9900072f0 con 0x7ff9a00722e0 2024-04-25T01:17:02.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.116+0000 7ff9a6800640 1 -- 172.21.15.90:0/1269750461 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9a01593f0 con 0x7ff9a00722e0 2024-04-25T01:17:02.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.116+0000 7ff9a6800640 1 -- 172.21.15.90:0/1269750461 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9a01599b0 con 0x7ff9a00722e0 2024-04-25T01:17:02.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.117+0000 7ff9a6800640 1 -- 172.21.15.90:0/1269750461 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9a0068fb0 con 0x7ff9a00722e0 2024-04-25T01:17:02.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.118+0000 7ff997400640 1 -- 172.21.15.90:0/1269750461 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff99001a070 con 0x7ff9a00722e0 2024-04-25T01:17:02.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.119+0000 7ff997400640 1 --2- 172.21.15.90:0/1269750461 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff97c06be30 0x7ff97c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:02.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.119+0000 7ff997400640 1 -- 172.21.15.90:0/1269750461 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7ff99008a660 con 0x7ff9a00722e0 2024-04-25T01:17:02.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.119+0000 7ff997e00640 1 --2- 172.21.15.90:0/1269750461 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff97c06be30 0x7ff97c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:02.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.120+0000 7ff997e00640 1 --2- 172.21.15.90:0/1269750461 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff97c06be30 0x7ff97c06e2f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff988006fd0 tx=0x7ff988006bc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:02.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.123+0000 7ff997400640 1 -- 172.21.15.90:0/1269750461 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff9a013cfc0 con 0x7ff9a00722e0 2024-04-25T01:17:02.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.322+0000 7ff9a6800640 1 -- 172.21.15.90:0/1269750461 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff9a00b1c30 con 0x7ff9a00722e0 2024-04-25T01:17:02.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.323+0000 7ff997400640 1 -- 172.21.15.90:0/1269750461 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7ff990058cf0 con 0x7ff9a00722e0 2024-04-25T01:17:02.307 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T01:17:02.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.326+0000 7ff9a6800640 1 -- 172.21.15.90:0/1269750461 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff97c06be30 msgr2=0x7ff97c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:02.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.326+0000 7ff9a6800640 1 --2- 172.21.15.90:0/1269750461 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff97c06be30 0x7ff97c06e2f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff988006fd0 tx=0x7ff988006bc0 comp rx=0 tx=0).stop 2024-04-25T01:17:02.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.326+0000 7ff9a6800640 1 -- 172.21.15.90:0/1269750461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9a00722e0 msgr2=0x7ff9a00b1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:02.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.326+0000 7ff9a6800640 1 --2- 172.21.15.90:0/1269750461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9a00722e0 0x7ff9a00b1270 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff99000c430 tx=0x7ff99000c8f0 comp rx=0 tx=0).stop 2024-04-25T01:17:02.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.326+0000 7ff9a6800640 1 -- 172.21.15.90:0/1269750461 shutdown_connections 2024-04-25T01:17:02.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.326+0000 7ff9a6800640 1 --2- 172.21.15.90:0/1269750461 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff97c06be30 0x7ff97c06e2f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:02.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.326+0000 7ff9a6800640 1 --2- 172.21.15.90:0/1269750461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9a015fc40 0x7ff9a013ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:02.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.326+0000 7ff9a6800640 1 --2- 172.21.15.90:0/1269750461 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9a00cf4d0 0x7ff9a013f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:02.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.326+0000 7ff9a6800640 1 --2- 172.21.15.90:0/1269750461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9a00722e0 0x7ff9a00b1270 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:02.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.326+0000 7ff9a6800640 1 -- 172.21.15.90:0/1269750461 >> 172.21.15.90:0/1269750461 conn(0x7ff9a006b720 msgr2=0x7ff9a01600a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:02.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.326+0000 7ff9a6800640 1 -- 172.21.15.90:0/1269750461 shutdown_connections 2024-04-25T01:17:02.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.326+0000 7ff9a6800640 1 -- 172.21.15.90:0/1269750461 wait complete. 2024-04-25T01:17:02.329 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:17:02.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.547+0000 7f3773000640 1 Processor -- start 2024-04-25T01:17:02.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.547+0000 7f3773000640 1 -- start start 2024-04-25T01:17:02.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.549+0000 7f3773000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f376c140d30 0x7f376c15f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:02.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.549+0000 7f3773000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f376c162360 0x7f376c0c9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:02.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.549+0000 7f3773000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f376c0ca790 con 0x7f376c06b120 2024-04-25T01:17:02.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.549+0000 7f3773000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f376c0caa70 con 0x7f376c140d30 2024-04-25T01:17:02.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.549+0000 7f3773000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f376c0cad50 con 0x7f376c162360 2024-04-25T01:17:02.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.549+0000 7f3771000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f376c06b120 0x7f376c143610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:35744/0 (socket says 172.21.15.90:35744) 2024-04-25T01:17:02.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.549+0000 7f3771000640 1 -- 172.21.15.90:0/1617047279 learned_addr learned my addr 172.21.15.90:0/1617047279 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:02.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.550+0000 7f3763400640 1 --2- 172.21.15.90:0/1617047279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f376c162360 0x7f376c0c9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:02.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.550+0000 7f376be00640 1 --2- 172.21.15.90:0/1617047279 >> v2:172.21.15.103:3300/0 conn(0x7f376c140d30 0x7f376c15f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:02.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.550+0000 7f376b400640 1 -- 172.21.15.90:0/1617047279 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3034011763 0 0) 0x7f376c0ca790 con 0x7f376c06b120 2024-04-25T01:17:02.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.550+0000 7f376b400640 1 -- 172.21.15.90:0/1617047279 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3750003520 con 0x7f376c06b120 2024-04-25T01:17:02.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.550+0000 7f376b400640 1 -- 172.21.15.90:0/1617047279 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3129532734 0 0) 0x7f3750003520 con 0x7f376c06b120 2024-04-25T01:17:02.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.550+0000 7f376b400640 1 -- 172.21.15.90:0/1617047279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f376c162360 msgr2=0x7f376c0c9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:02.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.550+0000 7f376b400640 1 --2- 172.21.15.90:0/1617047279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f376c162360 0x7f376c0c9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:02.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.550+0000 7f376b400640 1 -- 172.21.15.90:0/1617047279 >> v2:172.21.15.103:3300/0 conn(0x7f376c140d30 msgr2=0x7f376c15f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:02.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.550+0000 7f376b400640 1 --2- 172.21.15.90:0/1617047279 >> v2:172.21.15.103:3300/0 conn(0x7f376c140d30 0x7f376c15f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:02.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.550+0000 7f376b400640 1 -- 172.21.15.90:0/1617047279 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f376c161b60 con 0x7f376c06b120 2024-04-25T01:17:02.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.550+0000 7f376b400640 1 -- 172.21.15.90:0/1617047279 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f37640019f0 con 0x7f376c06b120 2024-04-25T01:17:02.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.550+0000 7f376b400640 1 -- 172.21.15.90:0/1617047279 >> v1:172.21.15.90:6789/0 conn(0x7f376c06b120 legacy=0x7f376c143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:02.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.550+0000 7f376b400640 1 --2- 172.21.15.90:0/1617047279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f376c162360 0x7f37500039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:02.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.551+0000 7f376b400640 1 --2- 172.21.15.90:0/1617047279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3750003f30 0x7f3750004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:02.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.551+0000 7f376b400640 1 --2- 172.21.15.90:0/1617047279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3750004d40 0x7f37500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:02.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.551+0000 7f376b400640 1 -- 172.21.15.90:0/1617047279 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f376c0caa70 con 0x7f3750004d40 2024-04-25T01:17:02.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.551+0000 7f376b400640 1 -- 172.21.15.90:0/1617047279 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f376c0cad50 con 0x7f3750003f30 2024-04-25T01:17:02.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.551+0000 7f376b400640 1 -- 172.21.15.90:0/1617047279 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f375000b9c0 con 0x7f376c162360 2024-04-25T01:17:02.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.551+0000 7f3771000640 1 --2- 172.21.15.90:0/1617047279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3750004d40 0x7f37500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:02.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.551+0000 7f3763400640 1 --2- 172.21.15.90:0/1617047279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3750003f30 0x7f3750004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:02.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.551+0000 7f376be00640 1 --2- 172.21.15.90:0/1617047279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f376c162360 0x7f37500039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:02.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.551+0000 7f3771000640 1 -- 172.21.15.90:0/1617047279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f376c162360 msgr2=0x7f37500039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:02.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.551+0000 7f3771000640 1 --2- 172.21.15.90:0/1617047279 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f376c162360 0x7f37500039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:02.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.551+0000 7f3771000640 1 -- 172.21.15.90:0/1617047279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3750003f30 msgr2=0x7f3750004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:02.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.551+0000 7f3771000640 1 --2- 172.21.15.90:0/1617047279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3750003f30 0x7f3750004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:02.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.551+0000 7f3771000640 1 -- 172.21.15.90:0/1617047279 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3750003520 con 0x7f3750004d40 2024-04-25T01:17:02.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.552+0000 7f3771000640 1 --2- 172.21.15.90:0/1617047279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3750004d40 0x7f37500051c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f376400a850 tx=0x7f3750005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a708e434d55b8d33 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:02.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.553+0000 7f376b400640 1 -- 172.21.15.90:0/1617047279 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f376400a4a0 con 0x7f3750004d40 2024-04-25T01:17:02.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.553+0000 7f376b400640 1 -- 172.21.15.90:0/1617047279 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f376402f070 con 0x7f3750004d40 2024-04-25T01:17:02.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.553+0000 7f376b400640 1 -- 172.21.15.90:0/1617047279 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f376402bd10 con 0x7f3750004d40 2024-04-25T01:17:02.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.553+0000 7f3773000640 1 -- 172.21.15.90:0/1617047279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3750004d40 msgr2=0x7f37500051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:02.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.553+0000 7f3773000640 1 --2- 172.21.15.90:0/1617047279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3750004d40 0x7f37500051c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f376400a850 tx=0x7f3750005830 comp rx=0 tx=0).stop 2024-04-25T01:17:02.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.553+0000 7f3771000640 1 -- 172.21.15.90:0/1617047279 reap_dead start 2024-04-25T01:17:02.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.553+0000 7f3773000640 1 -- 172.21.15.90:0/1617047279 shutdown_connections 2024-04-25T01:17:02.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.553+0000 7f3773000640 1 -- 172.21.15.90:0/1617047279 >> 172.21.15.90:0/1617047279 conn(0x7f376c0d0bf0 msgr2=0x7f376c0d1000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:02.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.554+0000 7f3773000640 1 -- 172.21.15.90:0/1617047279 shutdown_connections 2024-04-25T01:17:02.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.554+0000 7f3773000640 1 -- 172.21.15.90:0/1617047279 wait complete. 2024-04-25T01:17:02.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.555+0000 7f3773000640 1 Processor -- start 2024-04-25T01:17:02.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.555+0000 7f3773000640 1 -- start start 2024-04-25T01:17:02.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.556+0000 7f3773000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3750004d40 0x7f376c141480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:02.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.556+0000 7f3773000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f376c13dcc0 0x7f376c143ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:02.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.556+0000 7f3773000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f376c0cc240 0x7f376c0cc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:02.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.556+0000 7f3773000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f376c162940 con 0x7f376c13dcc0 2024-04-25T01:17:02.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.556+0000 7f3773000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f376c1627c0 con 0x7f376c0cc240 2024-04-25T01:17:02.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.556+0000 7f3773000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f376c141110 con 0x7f3750004d40 2024-04-25T01:17:02.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.557+0000 7f3763400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f376c13dcc0 0x7f376c143ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:02.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.557+0000 7f3763400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f376c13dcc0 0x7f376c143ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56528/0 (socket says 172.21.15.90:56528) 2024-04-25T01:17:02.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.557+0000 7f3763400640 1 -- 172.21.15.90:0/1497533909 learned_addr learned my addr 172.21.15.90:0/1497533909 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:02.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.557+0000 7f3771000640 1 --2- 172.21.15.90:0/1497533909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f376c0cc240 0x7f376c0cc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:02.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.557+0000 7f376be00640 1 --2- 172.21.15.90:0/1497533909 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3750004d40 0x7f376c141480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:02.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.557+0000 7f3763400640 1 -- 172.21.15.90:0/1497533909 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3750004d40 msgr2=0x7f376c141480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:02.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.557+0000 7f3763400640 1 --2- 172.21.15.90:0/1497533909 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3750004d40 0x7f376c141480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:02.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.557+0000 7f3763400640 1 -- 172.21.15.90:0/1497533909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f376c0cc240 msgr2=0x7f376c0cc6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:02.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.557+0000 7f3763400640 1 --2- 172.21.15.90:0/1497533909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f376c0cc240 0x7f376c0cc6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:02.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.557+0000 7f3763400640 1 -- 172.21.15.90:0/1497533909 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f376c13e120 con 0x7f376c13dcc0 2024-04-25T01:17:02.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.557+0000 7f376be00640 1 --2- 172.21.15.90:0/1497533909 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3750004d40 0x7f376c141480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:02.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.557+0000 7f3763400640 1 --2- 172.21.15.90:0/1497533909 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f376c13dcc0 0x7f376c143ba0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3754011570 tx=0x7f3754011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:02.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.558+0000 7f3768c00640 1 -- 172.21.15.90:0/1497533909 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3754015930 con 0x7f376c13dcc0 2024-04-25T01:17:02.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.558+0000 7f3768c00640 1 -- 172.21.15.90:0/1497533909 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3754015ad0 con 0x7f376c13dcc0 2024-04-25T01:17:02.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.558+0000 7f3773000640 1 -- 172.21.15.90:0/1497533909 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f376c144aa0 con 0x7f376c13dcc0 2024-04-25T01:17:02.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.558+0000 7f3768c00640 1 -- 172.21.15.90:0/1497533909 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f375400f460 con 0x7f376c13dcc0 2024-04-25T01:17:02.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.558+0000 7f3773000640 1 -- 172.21.15.90:0/1497533909 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f376c15fe30 con 0x7f376c13dcc0 2024-04-25T01:17:02.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.559+0000 7f3773000640 1 -- 172.21.15.90:0/1497533909 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f376c068fb0 con 0x7f376c13dcc0 2024-04-25T01:17:02.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.560+0000 7f3768c00640 1 -- 172.21.15.90:0/1497533909 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f375400f600 con 0x7f376c13dcc0 2024-04-25T01:17:02.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.561+0000 7f3768c00640 1 --2- 172.21.15.90:0/1497533909 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f374806be30 0x7f374806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:02.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.561+0000 7f3768c00640 1 -- 172.21.15.90:0/1497533909 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f375408ba70 con 0x7f376c13dcc0 2024-04-25T01:17:02.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.561+0000 7f376be00640 1 --2- 172.21.15.90:0/1497533909 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f374806be30 0x7f374806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:02.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.562+0000 7f376be00640 1 --2- 172.21.15.90:0/1497533909 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f374806be30 0x7f374806e2f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f375c009a80 tx=0x7f375c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:02.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.565+0000 7f3768c00640 1 -- 172.21.15.90:0/1497533909 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f3754019030 con 0x7f376c13dcc0 2024-04-25T01:17:02.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.770+0000 7f3773000640 1 -- 172.21.15.90:0/1497533909 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f376c073f10 con 0x7f376c13dcc0 2024-04-25T01:17:02.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.771+0000 7f3768c00640 1 -- 172.21.15.90:0/1497533909 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f375405a0d0 con 0x7f376c13dcc0 2024-04-25T01:17:02.755 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:17:02.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.774+0000 7f3773000640 1 -- 172.21.15.90:0/1497533909 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f374806be30 msgr2=0x7f374806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:02.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.774+0000 7f3773000640 1 --2- 172.21.15.90:0/1497533909 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f374806be30 0x7f374806e2f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f375c009a80 tx=0x7f375c008040 comp rx=0 tx=0).stop 2024-04-25T01:17:02.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.774+0000 7f3773000640 1 -- 172.21.15.90:0/1497533909 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f376c13dcc0 msgr2=0x7f376c143ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:02.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.774+0000 7f3773000640 1 --2- 172.21.15.90:0/1497533909 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f376c13dcc0 0x7f376c143ba0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3754011570 tx=0x7f3754011a30 comp rx=0 tx=0).stop 2024-04-25T01:17:02.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.774+0000 7f3773000640 1 -- 172.21.15.90:0/1497533909 shutdown_connections 2024-04-25T01:17:02.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.774+0000 7f3773000640 1 --2- 172.21.15.90:0/1497533909 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f374806be30 0x7f374806e2f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:02.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.774+0000 7f3773000640 1 --2- 172.21.15.90:0/1497533909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f376c0cc240 0x7f376c0cc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:02.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.774+0000 7f3773000640 1 --2- 172.21.15.90:0/1497533909 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f376c13dcc0 0x7f376c143ba0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:02.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.774+0000 7f3773000640 1 --2- 172.21.15.90:0/1497533909 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3750004d40 0x7f376c141480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:02.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.774+0000 7f3773000640 1 -- 172.21.15.90:0/1497533909 >> 172.21.15.90:0/1497533909 conn(0x7f376c0d0bf0 msgr2=0x7f376c0d00a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:02.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.774+0000 7f3773000640 1 -- 172.21.15.90:0/1497533909 shutdown_connections 2024-04-25T01:17:02.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.774+0000 7f3773000640 1 -- 172.21.15.90:0/1497533909 wait complete. 2024-04-25T01:17:02.777 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:17:02.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.995+0000 7f59bfc00640 1 Processor -- start 2024-04-25T01:17:02.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.995+0000 7f59bfc00640 1 -- start start 2024-04-25T01:17:02.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.997+0000 7f59bfc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f59b8140d30 0x7f59b815f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:02.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.997+0000 7f59bfc00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f59b806b120 0x7f59b806b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:02.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.997+0000 7f59bfc00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59b80ca790 con 0x7f59b8162360 2024-04-25T01:17:02.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.997+0000 7f59bfc00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f59b80caa70 con 0x7f59b8140d30 2024-04-25T01:17:02.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.997+0000 7f59bfc00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f59b80cad50 con 0x7f59b806b120 2024-04-25T01:17:02.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.997+0000 7f59bd200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f59b8140d30 0x7f59b815f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:02.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.997+0000 7f59bd200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f59b8140d30 0x7f59b815f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56540/0 (socket says 172.21.15.90:56540) 2024-04-25T01:17:02.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.997+0000 7f59b7e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f59b8162360 0x7f59b80c9f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:35756/0 (socket says 172.21.15.90:35756) 2024-04-25T01:17:02.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.997+0000 7f59bd200640 1 -- 172.21.15.90:0/658541092 learned_addr learned my addr 172.21.15.90:0/658541092 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:02.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.998+0000 7f59bdc00640 1 --2- 172.21.15.90:0/658541092 >> v2:172.21.15.103:3300/0 conn(0x7f59b806b120 0x7f59b806b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:02.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.998+0000 7f59bd200640 1 -- 172.21.15.90:0/658541092 >> v2:172.21.15.103:3300/0 conn(0x7f59b806b120 msgr2=0x7f59b806b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:02.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.998+0000 7f59bd200640 1 --2- 172.21.15.90:0/658541092 >> v2:172.21.15.103:3300/0 conn(0x7f59b806b120 0x7f59b806b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:02.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.998+0000 7f59bd200640 1 -- 172.21.15.90:0/658541092 >> v1:172.21.15.90:6789/0 conn(0x7f59b8162360 legacy=0x7f59b80c9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:02.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.998+0000 7f59bd200640 1 -- 172.21.15.90:0/658541092 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59b8161b60 con 0x7f59b8140d30 2024-04-25T01:17:02.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.998+0000 7f59bdc00640 1 --2- 172.21.15.90:0/658541092 >> v2:172.21.15.103:3300/0 conn(0x7f59b806b120 0x7f59b806b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:02.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.999+0000 7f59bd200640 1 --2- 172.21.15.90:0/658541092 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b8140d30 0x7f59b815f050 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f59b815c010 tx=0x7f59a802a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2d676f130df1d6b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:02.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.999+0000 7f59b7400640 1 -- 172.21.15.90:0/658541092 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a8036070 con 0x7f59b8140d30 2024-04-25T01:17:02.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.999+0000 7f59b7400640 1 -- 172.21.15.90:0/658541092 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59a8031600 con 0x7f59b8140d30 2024-04-25T01:17:02.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:02.999+0000 7f59b7400640 1 -- 172.21.15.90:0/658541092 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a80319d0 con 0x7f59b8140d30 2024-04-25T01:17:02.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.000+0000 7f59bfc00640 1 -- 172.21.15.90:0/658541092 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b8140d30 msgr2=0x7f59b815f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:02.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.000+0000 7f59bfc00640 1 --2- 172.21.15.90:0/658541092 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b8140d30 0x7f59b815f050 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f59b815c010 tx=0x7f59a802a730 comp rx=0 tx=0).stop 2024-04-25T01:17:02.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.000+0000 7f59bfc00640 1 -- 172.21.15.90:0/658541092 shutdown_connections 2024-04-25T01:17:02.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.000+0000 7f59bfc00640 1 --2- 172.21.15.90:0/658541092 >> v2:172.21.15.103:3300/0 conn(0x7f59b806b120 0x7f59b806b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:02.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.000+0000 7f59bfc00640 1 --2- 172.21.15.90:0/658541092 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b8140d30 0x7f59b815f050 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:02.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.000+0000 7f59bfc00640 1 -- 172.21.15.90:0/658541092 >> 172.21.15.90:0/658541092 conn(0x7f59b80d0bf0 msgr2=0x7f59b80d1000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:02.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.000+0000 7f59bfc00640 1 -- 172.21.15.90:0/658541092 shutdown_connections 2024-04-25T01:17:02.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.000+0000 7f59bfc00640 1 -- 172.21.15.90:0/658541092 wait complete. 2024-04-25T01:17:02.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.002+0000 7f59bfc00640 1 Processor -- start 2024-04-25T01:17:02.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.002+0000 7f59bfc00640 1 -- start start 2024-04-25T01:17:02.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.004+0000 7f59bfc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59b806b120 0x7f59b813fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:02.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.004+0000 7f59bfc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b8140d30 0x7f59b81436a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:02.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.004+0000 7f59bfc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f59b8162360 0x7f59b8141670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:02.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.004+0000 7f59bfc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f59b8162940 con 0x7f59b8140d30 2024-04-25T01:17:02.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.004+0000 7f59bfc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f59b81627c0 con 0x7f59b806b120 2024-04-25T01:17:02.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.004+0000 7f59bfc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f59b8141110 con 0x7f59b8162360 2024-04-25T01:17:02.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.004+0000 7f59b7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b8140d30 0x7f59b81436a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:02.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.004+0000 7f59b7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b8140d30 0x7f59b81436a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56548/0 (socket says 172.21.15.90:56548) 2024-04-25T01:17:02.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.004+0000 7f59b7e00640 1 -- 172.21.15.90:0/1428611753 learned_addr learned my addr 172.21.15.90:0/1428611753 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:02.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.004+0000 7f59bd200640 1 --2- 172.21.15.90:0/1428611753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59b806b120 0x7f59b813fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:02.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.004+0000 7f59b7e00640 1 -- 172.21.15.90:0/1428611753 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f59b8162360 msgr2=0x7f59b8141670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:02.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.004+0000 7f59bdc00640 1 --2- 172.21.15.90:0/1428611753 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f59b8162360 0x7f59b8141670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:02.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.005+0000 7f59b7e00640 1 --2- 172.21.15.90:0/1428611753 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f59b8162360 0x7f59b8141670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:02.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.005+0000 7f59b7e00640 1 -- 172.21.15.90:0/1428611753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59b806b120 msgr2=0x7f59b813fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:02.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.005+0000 7f59b7e00640 1 --2- 172.21.15.90:0/1428611753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59b806b120 0x7f59b813fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:02.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.005+0000 7f59b7e00640 1 -- 172.21.15.90:0/1428611753 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59b8144110 con 0x7f59b8140d30 2024-04-25T01:17:02.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.005+0000 7f59bd200640 1 --2- 172.21.15.90:0/1428611753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59b806b120 0x7f59b813fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:17:02.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.005+0000 7f59bdc00640 1 --2- 172.21.15.90:0/1428611753 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f59b8162360 0x7f59b8141670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:02.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.005+0000 7f59b7e00640 1 --2- 172.21.15.90:0/1428611753 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b8140d30 0x7f59b81436a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f59a000c3f0 tx=0x7f59a000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:02.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.005+0000 7f59b4c00640 1 -- 172.21.15.90:0/1428611753 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a00074e0 con 0x7f59b8140d30 2024-04-25T01:17:02.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.005+0000 7f59b4c00640 1 -- 172.21.15.90:0/1428611753 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59a0007680 con 0x7f59b8140d30 2024-04-25T01:17:02.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.005+0000 7f59b4c00640 1 -- 172.21.15.90:0/1428611753 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a0004a00 con 0x7f59b8140d30 2024-04-25T01:17:02.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.005+0000 7f59bfc00640 1 -- 172.21.15.90:0/1428611753 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59b81443f0 con 0x7f59b8140d30 2024-04-25T01:17:02.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.005+0000 7f59bfc00640 1 -- 172.21.15.90:0/1428611753 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59b8145400 con 0x7f59b8140d30 2024-04-25T01:17:02.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.006+0000 7f59bfc00640 1 -- 172.21.15.90:0/1428611753 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59b8068fb0 con 0x7f59b8140d30 2024-04-25T01:17:02.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.008+0000 7f59b4c00640 1 -- 172.21.15.90:0/1428611753 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f59a001a020 con 0x7f59b8140d30 2024-04-25T01:17:02.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.009+0000 7f59b4c00640 1 --2- 172.21.15.90:0/1428611753 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f599406be60 0x7f599406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:02.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.009+0000 7f59b4c00640 1 -- 172.21.15.90:0/1428611753 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f59a008a9a0 con 0x7f59b8140d30 2024-04-25T01:17:02.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.010+0000 7f59bd200640 1 --2- 172.21.15.90:0/1428611753 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f599406be60 0x7f599406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:02.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.010+0000 7f59bd200640 1 --2- 172.21.15.90:0/1428611753 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f599406be60 0x7f599406e320 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f59a8033000 tx=0x7f59a8009000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:02.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.014+0000 7f59b4c00640 1 -- 172.21.15.90:0/1428611753 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f59a0010300 con 0x7f59b8140d30 2024-04-25T01:17:03.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.214+0000 7f59bfc00640 1 -- 172.21.15.90:0/1428611753 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f59b8161060 con 0x7f59b8140d30 2024-04-25T01:17:03.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.215+0000 7f59b4c00640 1 -- 172.21.15.90:0/1428611753 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f59a0059000 con 0x7f59b8140d30 2024-04-25T01:17:03.199 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-25T01:17:03.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.218+0000 7f59bfc00640 1 -- 172.21.15.90:0/1428611753 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f599406be60 msgr2=0x7f599406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:03.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.218+0000 7f59bfc00640 1 --2- 172.21.15.90:0/1428611753 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f599406be60 0x7f599406e320 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f59a8033000 tx=0x7f59a8009000 comp rx=0 tx=0).stop 2024-04-25T01:17:03.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.218+0000 7f59bfc00640 1 -- 172.21.15.90:0/1428611753 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b8140d30 msgr2=0x7f59b81436a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:03.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.218+0000 7f59bfc00640 1 --2- 172.21.15.90:0/1428611753 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b8140d30 0x7f59b81436a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f59a000c3f0 tx=0x7f59a000c8b0 comp rx=0 tx=0).stop 2024-04-25T01:17:03.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.218+0000 7f59bfc00640 1 -- 172.21.15.90:0/1428611753 shutdown_connections 2024-04-25T01:17:03.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.218+0000 7f59bfc00640 1 --2- 172.21.15.90:0/1428611753 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f599406be60 0x7f599406e320 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:03.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.218+0000 7f59bfc00640 1 --2- 172.21.15.90:0/1428611753 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f59b8162360 0x7f59b8141670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:03.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.218+0000 7f59bfc00640 1 --2- 172.21.15.90:0/1428611753 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b8140d30 0x7f59b81436a0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:03.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.218+0000 7f59bfc00640 1 --2- 172.21.15.90:0/1428611753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59b806b120 0x7f59b813fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:03.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.218+0000 7f59bfc00640 1 -- 172.21.15.90:0/1428611753 >> 172.21.15.90:0/1428611753 conn(0x7f59b80d0bf0 msgr2=0x7f59b80b0f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:03.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.218+0000 7f59bfc00640 1 -- 172.21.15.90:0/1428611753 shutdown_connections 2024-04-25T01:17:03.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.218+0000 7f59bfc00640 1 -- 172.21.15.90:0/1428611753 wait complete. 2024-04-25T01:17:03.220 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:17:03.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.437+0000 7f60b9400640 1 Processor -- start 2024-04-25T01:17:03.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.437+0000 7f60b9400640 1 -- start start 2024-04-25T01:17:03.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.439+0000 7f60b9400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f60b415fa80 0x7f60b415fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:03.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.439+0000 7f60b9400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60b40cf7e0 0x7f60b40ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:03.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.439+0000 7f60b9400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60b41603a0 con 0x7f60b415c150 2024-04-25T01:17:03.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.439+0000 7f60b9400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f60b40cfc70 con 0x7f60b415fa80 2024-04-25T01:17:03.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.439+0000 7f60b9400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f60b40cff50 con 0x7f60b40cf7e0 2024-04-25T01:17:03.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.439+0000 7f60b8a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f60b415c150 0x7f60b415c5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:35770/0 (socket says 172.21.15.90:35770) 2024-04-25T01:17:03.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.439+0000 7f60b8a00640 1 -- 172.21.15.90:0/3836163797 learned_addr learned my addr 172.21.15.90:0/3836163797 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:03.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.439+0000 7f60b3e00640 1 --2- 172.21.15.90:0/3836163797 >> v2:172.21.15.103:3300/0 conn(0x7f60b415fa80 0x7f60b415fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:03.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.440+0000 7f60b3400640 1 --2- 172.21.15.90:0/3836163797 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60b40cf7e0 0x7f60b40ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:03.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.440+0000 7f60b2a00640 1 -- 172.21.15.90:0/3836163797 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1354086505 0 0) 0x7f60b41603a0 con 0x7f60b415c150 2024-04-25T01:17:03.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.440+0000 7f60b2a00640 1 -- 172.21.15.90:0/3836163797 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6098003520 con 0x7f60b415c150 2024-04-25T01:17:03.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.440+0000 7f60b2a00640 1 -- 172.21.15.90:0/3836163797 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4189271423 0 0) 0x7f6098003520 con 0x7f60b415c150 2024-04-25T01:17:03.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.440+0000 7f60b2a00640 1 -- 172.21.15.90:0/3836163797 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60b40cf7e0 msgr2=0x7f60b40ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:03.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.440+0000 7f60b2a00640 1 --2- 172.21.15.90:0/3836163797 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60b40cf7e0 0x7f60b40ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:03.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.440+0000 7f60b2a00640 1 -- 172.21.15.90:0/3836163797 >> v2:172.21.15.103:3300/0 conn(0x7f60b415fa80 msgr2=0x7f60b415fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:03.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.440+0000 7f60b2a00640 1 --2- 172.21.15.90:0/3836163797 >> v2:172.21.15.103:3300/0 conn(0x7f60b415fa80 0x7f60b415fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:03.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.440+0000 7f60b2a00640 1 -- 172.21.15.90:0/3836163797 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60b40d0230 con 0x7f60b415c150 2024-04-25T01:17:03.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.440+0000 7f60b2a00640 1 -- 172.21.15.90:0/3836163797 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f60ac0019f0 con 0x7f60b415c150 2024-04-25T01:17:03.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.440+0000 7f60b2a00640 1 -- 172.21.15.90:0/3836163797 >> v1:172.21.15.90:6789/0 conn(0x7f60b415c150 legacy=0x7f60b415c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:03.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.440+0000 7f60b2a00640 1 --2- 172.21.15.90:0/3836163797 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60980039b0 0x7f6098003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:03.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.440+0000 7f60b2a00640 1 --2- 172.21.15.90:0/3836163797 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6098004800 0x7f6098004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:03.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.440+0000 7f60b2a00640 1 --2- 172.21.15.90:0/3836163797 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6098005180 0x7f609800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:03.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.441+0000 7f60b2a00640 1 -- 172.21.15.90:0/3836163797 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f60b40cfc70 con 0x7f60980039b0 2024-04-25T01:17:03.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.441+0000 7f60b2a00640 1 -- 172.21.15.90:0/3836163797 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f60b40cff50 con 0x7f6098005180 2024-04-25T01:17:03.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.441+0000 7f60b2a00640 1 -- 172.21.15.90:0/3836163797 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f609800bf00 con 0x7f6098004800 2024-04-25T01:17:03.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.441+0000 7f60b3e00640 1 --2- 172.21.15.90:0/3836163797 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60980039b0 0x7f6098003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:03.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.441+0000 7f60b3e00640 1 -- 172.21.15.90:0/3836163797 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6098004800 msgr2=0x7f6098004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:03.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.441+0000 7f60b3e00640 1 --2- 172.21.15.90:0/3836163797 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6098004800 0x7f6098004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:03.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.441+0000 7f60b3e00640 1 -- 172.21.15.90:0/3836163797 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6098005180 msgr2=0x7f609800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:03.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.441+0000 7f60b3e00640 1 --2- 172.21.15.90:0/3836163797 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6098005180 0x7f609800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:03.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.441+0000 7f60b3e00640 1 -- 172.21.15.90:0/3836163797 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6098003520 con 0x7f60980039b0 2024-04-25T01:17:03.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.442+0000 7f60b3e00640 1 --2- 172.21.15.90:0/3836163797 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60980039b0 0x7f6098003dd0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f60a4009ac0 tx=0x7f6098005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc2f473d1722e518 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:03.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.442+0000 7f60b2a00640 1 -- 172.21.15.90:0/3836163797 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a4036070 con 0x7f60980039b0 2024-04-25T01:17:03.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.442+0000 7f60b2a00640 1 -- 172.21.15.90:0/3836163797 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60a4031400 con 0x7f60980039b0 2024-04-25T01:17:03.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.442+0000 7f60b2a00640 1 -- 172.21.15.90:0/3836163797 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a4009690 con 0x7f60980039b0 2024-04-25T01:17:03.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.443+0000 7f60b9400640 1 -- 172.21.15.90:0/3836163797 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60980039b0 msgr2=0x7f6098003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:03.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.443+0000 7f60b9400640 1 --2- 172.21.15.90:0/3836163797 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60980039b0 0x7f6098003dd0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f60a4009ac0 tx=0x7f6098005830 comp rx=0 tx=0).stop 2024-04-25T01:17:03.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.443+0000 7f60b8a00640 1 -- 172.21.15.90:0/3836163797 reap_dead start 2024-04-25T01:17:03.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.443+0000 7f60b9400640 1 -- 172.21.15.90:0/3836163797 shutdown_connections 2024-04-25T01:17:03.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.443+0000 7f60b9400640 1 -- 172.21.15.90:0/3836163797 >> 172.21.15.90:0/3836163797 conn(0x7f60b40d1c10 msgr2=0x7f60b40d2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:03.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.443+0000 7f60b9400640 1 -- 172.21.15.90:0/3836163797 shutdown_connections 2024-04-25T01:17:03.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.443+0000 7f60b9400640 1 -- 172.21.15.90:0/3836163797 wait complete. 2024-04-25T01:17:03.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.445+0000 7f60b9400640 1 Processor -- start 2024-04-25T01:17:03.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.445+0000 7f60b9400640 1 -- start start 2024-04-25T01:17:03.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.446+0000 7f60b9400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60980039b0 0x7f60b415d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:03.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.446+0000 7f60b9400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60b40b62e0 0x7f60b40b4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:03.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.446+0000 7f60b9400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60b415e100 0x7f60b4155d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:03.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.446+0000 7f60b9400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f60b40cd890 con 0x7f60b415e100 2024-04-25T01:17:03.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.446+0000 7f60b9400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f60b40d07f0 con 0x7f60980039b0 2024-04-25T01:17:03.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.446+0000 7f60b9400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f60b40cda10 con 0x7f60b40b62e0 2024-04-25T01:17:03.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.446+0000 7f60b8a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60b415e100 0x7f60b4155d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:03.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.446+0000 7f60b8a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60b415e100 0x7f60b4155d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56570/0 (socket says 172.21.15.90:56570) 2024-04-25T01:17:03.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.446+0000 7f60b8a00640 1 -- 172.21.15.90:0/3726790129 learned_addr learned my addr 172.21.15.90:0/3726790129 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:03.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.447+0000 7f60b8a00640 1 -- 172.21.15.90:0/3726790129 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60b40b62e0 msgr2=0x7f60b40b4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:03.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.447+0000 7f60b8a00640 1 --2- 172.21.15.90:0/3726790129 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60b40b62e0 0x7f60b40b4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:03.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.447+0000 7f60b8a00640 1 -- 172.21.15.90:0/3726790129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60980039b0 msgr2=0x7f60b415d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:03.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.447+0000 7f60b8a00640 1 --2- 172.21.15.90:0/3726790129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60980039b0 0x7f60b415d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:03.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.447+0000 7f60b8a00640 1 -- 172.21.15.90:0/3726790129 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60b40b4fa0 con 0x7f60b415e100 2024-04-25T01:17:03.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.447+0000 7f60b8a00640 1 --2- 172.21.15.90:0/3726790129 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60b415e100 0x7f60b4155d50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f60b40ce290 tx=0x7f60ac00ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:03.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.447+0000 7f60abe00640 1 -- 172.21.15.90:0/3726790129 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60ac00a5b0 con 0x7f60b415e100 2024-04-25T01:17:03.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.447+0000 7f60abe00640 1 -- 172.21.15.90:0/3726790129 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60ac00a750 con 0x7f60b415e100 2024-04-25T01:17:03.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.447+0000 7f60abe00640 1 -- 172.21.15.90:0/3726790129 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60ac00ab40 con 0x7f60b415e100 2024-04-25T01:17:03.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.447+0000 7f60b9400640 1 -- 172.21.15.90:0/3726790129 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60b40b5240 con 0x7f60b415e100 2024-04-25T01:17:03.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.447+0000 7f60b9400640 1 -- 172.21.15.90:0/3726790129 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60b40b5800 con 0x7f60b415e100 2024-04-25T01:17:03.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.448+0000 7f60b9400640 1 -- 172.21.15.90:0/3726790129 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60b4075350 con 0x7f60b415e100 2024-04-25T01:17:03.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.449+0000 7f60abe00640 1 -- 172.21.15.90:0/3726790129 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f60ac00ace0 con 0x7f60b415e100 2024-04-25T01:17:03.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.451+0000 7f60abe00640 1 --2- 172.21.15.90:0/3726790129 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f609406be60 0x7f609406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:03.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.451+0000 7f60abe00640 1 -- 172.21.15.90:0/3726790129 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f60ac08a9f0 con 0x7f60b415e100 2024-04-25T01:17:03.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.451+0000 7f60b3e00640 1 --2- 172.21.15.90:0/3726790129 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f609406be60 0x7f609406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:03.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.452+0000 7f60b3e00640 1 --2- 172.21.15.90:0/3726790129 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f609406be60 0x7f609406e320 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f60a4033000 tx=0x7f60a402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:03.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.455+0000 7f60abe00640 1 -- 172.21.15.90:0/3726790129 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f60ac01a020 con 0x7f60b415e100 2024-04-25T01:17:03.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.654+0000 7f60b9400640 1 -- 172.21.15.90:0/3726790129 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f60b4160060 con 0x7f60b415e100 2024-04-25T01:17:03.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.654+0000 7f60abe00640 1 -- 172.21.15.90:0/3726790129 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f60ac058fc0 con 0x7f60b415e100 2024-04-25T01:17:03.638 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:17:03.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.657+0000 7f60b9400640 1 -- 172.21.15.90:0/3726790129 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f609406be60 msgr2=0x7f609406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:03.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.657+0000 7f60b9400640 1 --2- 172.21.15.90:0/3726790129 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f609406be60 0x7f609406e320 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f60a4033000 tx=0x7f60a402e000 comp rx=0 tx=0).stop 2024-04-25T01:17:03.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.657+0000 7f60b9400640 1 -- 172.21.15.90:0/3726790129 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60b415e100 msgr2=0x7f60b4155d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:03.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.657+0000 7f60b9400640 1 --2- 172.21.15.90:0/3726790129 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60b415e100 0x7f60b4155d50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f60b40ce290 tx=0x7f60ac00ccd0 comp rx=0 tx=0).stop 2024-04-25T01:17:03.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.657+0000 7f60b9400640 1 -- 172.21.15.90:0/3726790129 shutdown_connections 2024-04-25T01:17:03.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.657+0000 7f60b9400640 1 --2- 172.21.15.90:0/3726790129 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f609406be60 0x7f609406e320 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:03.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.657+0000 7f60b9400640 1 --2- 172.21.15.90:0/3726790129 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60b415e100 0x7f60b4155d50 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:03.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.657+0000 7f60b9400640 1 --2- 172.21.15.90:0/3726790129 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60b40b62e0 0x7f60b40b4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:03.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.657+0000 7f60b9400640 1 --2- 172.21.15.90:0/3726790129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60980039b0 0x7f60b415d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:03.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.658+0000 7f60b9400640 1 -- 172.21.15.90:0/3726790129 >> 172.21.15.90:0/3726790129 conn(0x7f60b40d1c10 msgr2=0x7f60b40b1ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:03.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.658+0000 7f60b9400640 1 -- 172.21.15.90:0/3726790129 shutdown_connections 2024-04-25T01:17:03.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.658+0000 7f60b9400640 1 -- 172.21.15.90:0/3726790129 wait complete. 2024-04-25T01:17:03.661 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:17:03.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.878+0000 7fbadb400640 1 Processor -- start 2024-04-25T01:17:03.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.878+0000 7fbadb400640 1 -- start start 2024-04-25T01:17:03.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.880+0000 7fbadb400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbad4145590 0x7fbad40b4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:03.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.880+0000 7fbadb400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbad415f9b0 0x7fbad415fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:03.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.880+0000 7fbadb400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbad40ce110 0x7fbad40ce5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:03.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.880+0000 7fbadb400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fbad4160830 con 0x7fbad4145590 2024-04-25T01:17:03.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.880+0000 7fbadb400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fbad4160350 con 0x7fbad415f9b0 2024-04-25T01:17:03.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.880+0000 7fbadb400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbad406cde0 con 0x7fbad40ce110 2024-04-25T01:17:03.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.880+0000 7fbad8a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbad4145590 0x7fbad40b4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:03.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.880+0000 7fbad8a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbad4145590 0x7fbad40b4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56576/0 (socket says 172.21.15.90:56576) 2024-04-25T01:17:03.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.880+0000 7fbad8a00640 1 -- 172.21.15.90:0/4238325982 learned_addr learned my addr 172.21.15.90:0/4238325982 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:03.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.881+0000 7fbad8a00640 1 -- 172.21.15.90:0/4238325982 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbad40ce110 msgr2=0x7fbad40ce5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:03.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.881+0000 7fbad8a00640 1 --2- 172.21.15.90:0/4238325982 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbad40ce110 0x7fbad40ce5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:03.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.881+0000 7fbad8a00640 1 -- 172.21.15.90:0/4238325982 >> v2:172.21.15.103:3300/0 conn(0x7fbad415f9b0 msgr2=0x7fbad415fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:03.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.881+0000 7fbad8a00640 1 --2- 172.21.15.90:0/4238325982 >> v2:172.21.15.103:3300/0 conn(0x7fbad415f9b0 0x7fbad415fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:03.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.881+0000 7fbad8a00640 1 -- 172.21.15.90:0/4238325982 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbad40b4240 con 0x7fbad4145590 2024-04-25T01:17:03.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.882+0000 7fbad8a00640 1 --2- 172.21.15.90:0/4238325982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbad4145590 0x7fbad40b4a20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbad40b46c0 tx=0x7fbac402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb62fe89d2e205e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:03.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.882+0000 7fbad3e00640 1 -- 172.21.15.90:0/4238325982 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac4037070 con 0x7fbad4145590 2024-04-25T01:17:03.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.882+0000 7fbad3e00640 1 -- 172.21.15.90:0/4238325982 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbac4032440 con 0x7fbad4145590 2024-04-25T01:17:03.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.882+0000 7fbad3e00640 1 -- 172.21.15.90:0/4238325982 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac4032810 con 0x7fbad4145590 2024-04-25T01:17:03.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.883+0000 7fbadb400640 1 -- 172.21.15.90:0/4238325982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbad4145590 msgr2=0x7fbad40b4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:03.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.883+0000 7fbadb400640 1 --2- 172.21.15.90:0/4238325982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbad4145590 0x7fbad40b4a20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbad40b46c0 tx=0x7fbac402a1e0 comp rx=0 tx=0).stop 2024-04-25T01:17:03.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.883+0000 7fbadb400640 1 -- 172.21.15.90:0/4238325982 shutdown_connections 2024-04-25T01:17:03.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.883+0000 7fbadb400640 1 --2- 172.21.15.90:0/4238325982 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbad40ce110 0x7fbad40ce5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:03.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.883+0000 7fbadb400640 1 --2- 172.21.15.90:0/4238325982 >> v2:172.21.15.103:3300/0 conn(0x7fbad415f9b0 0x7fbad415fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:03.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.883+0000 7fbadb400640 1 --2- 172.21.15.90:0/4238325982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbad4145590 0x7fbad40b4a20 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:03.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.883+0000 7fbadb400640 1 -- 172.21.15.90:0/4238325982 >> 172.21.15.90:0/4238325982 conn(0x7fbad40b1820 msgr2=0x7fbad40b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:03.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.883+0000 7fbadb400640 1 -- 172.21.15.90:0/4238325982 shutdown_connections 2024-04-25T01:17:03.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.883+0000 7fbadb400640 1 -- 172.21.15.90:0/4238325982 wait complete. 2024-04-25T01:17:03.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.885+0000 7fbadb400640 1 Processor -- start 2024-04-25T01:17:03.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.885+0000 7fbadb400640 1 -- start start 2024-04-25T01:17:03.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.886+0000 7fbadb400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbad40ce110 0x7fbad4154830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:03.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.886+0000 7fbadb400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbad4145590 0x7fbad40b26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:03.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.886+0000 7fbadb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbad415f9b0 0x7fbad40b2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:03.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.886+0000 7fbadb400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbad40b3c80 con 0x7fbad415f9b0 2024-04-25T01:17:03.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.886+0000 7fbadb400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbad40b39a0 con 0x7fbad40ce110 2024-04-25T01:17:03.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.886+0000 7fbadb400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbad40b3f60 con 0x7fbad4145590 2024-04-25T01:17:03.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.886+0000 7fbad9400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbad415f9b0 0x7fbad40b2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:03.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.887+0000 7fbad9400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbad415f9b0 0x7fbad40b2c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56582/0 (socket says 172.21.15.90:56582) 2024-04-25T01:17:03.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.887+0000 7fbad9400640 1 -- 172.21.15.90:0/1996890575 learned_addr learned my addr 172.21.15.90:0/1996890575 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:03.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.887+0000 7fbad9400640 1 -- 172.21.15.90:0/1996890575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbad4145590 msgr2=0x7fbad40b26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:03.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.887+0000 7fbad9400640 1 --2- 172.21.15.90:0/1996890575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbad4145590 0x7fbad40b26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:03.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.887+0000 7fbad8a00640 1 --2- 172.21.15.90:0/1996890575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbad40ce110 0x7fbad4154830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:03.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.887+0000 7fbad9400640 1 -- 172.21.15.90:0/1996890575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbad40ce110 msgr2=0x7fbad4154830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:03.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.887+0000 7fbad9400640 1 --2- 172.21.15.90:0/1996890575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbad40ce110 0x7fbad4154830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:03.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.887+0000 7fbad9400640 1 -- 172.21.15.90:0/1996890575 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbad4071700 con 0x7fbad415f9b0 2024-04-25T01:17:03.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.887+0000 7fbad8a00640 1 --2- 172.21.15.90:0/1996890575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbad40ce110 0x7fbad4154830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:03.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.887+0000 7fbad9400640 1 --2- 172.21.15.90:0/1996890575 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbad415f9b0 0x7fbad40b2c10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fbacc00b970 tx=0x7fbacc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:03.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.887+0000 7fbad1600640 1 -- 172.21.15.90:0/1996890575 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbacc019070 con 0x7fbad415f9b0 2024-04-25T01:17:03.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.888+0000 7fbad1600640 1 -- 172.21.15.90:0/1996890575 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbacc00cb60 con 0x7fbad415f9b0 2024-04-25T01:17:03.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.888+0000 7fbad1600640 1 -- 172.21.15.90:0/1996890575 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbacc014440 con 0x7fbad415f9b0 2024-04-25T01:17:03.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.888+0000 7fbadb400640 1 -- 172.21.15.90:0/1996890575 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbad40719e0 con 0x7fbad415f9b0 2024-04-25T01:17:03.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.888+0000 7fbadb400640 1 -- 172.21.15.90:0/1996890575 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbad4071fa0 con 0x7fbad415f9b0 2024-04-25T01:17:03.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.889+0000 7fbadb400640 1 -- 172.21.15.90:0/1996890575 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbad4068fb0 con 0x7fbad415f9b0 2024-04-25T01:17:03.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.890+0000 7fbad1600640 1 -- 172.21.15.90:0/1996890575 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbacc01c070 con 0x7fbad415f9b0 2024-04-25T01:17:03.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.891+0000 7fbad1600640 1 --2- 172.21.15.90:0/1996890575 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbab006be30 0x7fbab006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:03.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.891+0000 7fbad1600640 1 -- 172.21.15.90:0/1996890575 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fbacc08af60 con 0x7fbad415f9b0 2024-04-25T01:17:03.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.891+0000 7fbad8a00640 1 --2- 172.21.15.90:0/1996890575 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbab006be30 0x7fbab006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:03.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.892+0000 7fbad8a00640 1 --2- 172.21.15.90:0/1996890575 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbab006be30 0x7fbab006e2f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbac4034000 tx=0x7fbac402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:03.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:03.895+0000 7fbad1600640 1 -- 172.21.15.90:0/1996890575 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fbacc059530 con 0x7fbad415f9b0 2024-04-25T01:17:04.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.134+0000 7fbadb400640 1 -- 172.21.15.90:0/1996890575 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbad406ea50 con 0x7fbad415f9b0 2024-04-25T01:17:04.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.136+0000 7fbad1600640 1 -- 172.21.15.90:0/1996890575 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5180 (secure 0 0 0) 0x7fbacc058e30 con 0x7fbad415f9b0 2024-04-25T01:17:04.120 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:17:04.120 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6834/2605945289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":2605945289},{"type":"v1","addr":"172.21.15.103:6834","nonce":2605945289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6836/704639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":704639426},{"type":"v1","addr":"172.21.15.90:6836","nonce":704639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4245,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3477867755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6835","nonce":3477867755},{"type":"v1","addr":"172.21.15.90:6837","nonce":3477867755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:16:22.060018+0000","modified":"2024-04-25T01:16:50.405431+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6835/4291578198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6833","nonce":4291578198},{"type":"v1","addr":"172.21.15.103:6835","nonce":4291578198}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1}]} 2024-04-25T01:17:04.120 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 9 2024-04-25T01:17:04.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.139+0000 7fbadb400640 1 -- 172.21.15.90:0/1996890575 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbab006be30 msgr2=0x7fbab006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:04.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.139+0000 7fbadb400640 1 --2- 172.21.15.90:0/1996890575 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbab006be30 0x7fbab006e2f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbac4034000 tx=0x7fbac402e000 comp rx=0 tx=0).stop 2024-04-25T01:17:04.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.139+0000 7fbadb400640 1 -- 172.21.15.90:0/1996890575 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbad415f9b0 msgr2=0x7fbad40b2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:04.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.139+0000 7fbadb400640 1 --2- 172.21.15.90:0/1996890575 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbad415f9b0 0x7fbad40b2c10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fbacc00b970 tx=0x7fbacc00be30 comp rx=0 tx=0).stop 2024-04-25T01:17:04.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.139+0000 7fbadb400640 1 -- 172.21.15.90:0/1996890575 shutdown_connections 2024-04-25T01:17:04.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.139+0000 7fbadb400640 1 --2- 172.21.15.90:0/1996890575 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbab006be30 0x7fbab006e2f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:04.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.139+0000 7fbadb400640 1 --2- 172.21.15.90:0/1996890575 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbad415f9b0 0x7fbad40b2c10 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:04.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.139+0000 7fbadb400640 1 --2- 172.21.15.90:0/1996890575 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbad4145590 0x7fbad40b26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:04.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.139+0000 7fbadb400640 1 --2- 172.21.15.90:0/1996890575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbad40ce110 0x7fbad4154830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:04.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.139+0000 7fbadb400640 1 -- 172.21.15.90:0/1996890575 >> 172.21.15.90:0/1996890575 conn(0x7fbad40b1820 msgr2=0x7fbad40cf370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:04.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.139+0000 7fbadb400640 1 -- 172.21.15.90:0/1996890575 shutdown_connections 2024-04-25T01:17:04.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.139+0000 7fbadb400640 1 -- 172.21.15.90:0/1996890575 wait complete. 2024-04-25T01:17:04.142 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:17:04.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.358+0000 7fd682200640 1 Processor -- start 2024-04-25T01:17:04.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.358+0000 7fd682200640 1 -- start start 2024-04-25T01:17:04.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.359+0000 7fd682200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd67c143d20 0x7fd67c15ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:04.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.359+0000 7fd682200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd67c06ed60 0x7fd67c15f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:04.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.360+0000 7fd682200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd67c15f770 con 0x7fd67c143940 2024-04-25T01:17:04.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.360+0000 7fd682200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fd67c070e60 con 0x7fd67c06ed60 2024-04-25T01:17:04.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.360+0000 7fd682200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd67c071140 con 0x7fd67c143d20 2024-04-25T01:17:04.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.360+0000 7fd67be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd67c06ed60 0x7fd67c15f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:04.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.360+0000 7fd67be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd67c06ed60 0x7fd67c15f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56594/0 (socket says 172.21.15.90:56594) 2024-04-25T01:17:04.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.360+0000 7fd67be00640 1 -- 172.21.15.90:0/3792446842 learned_addr learned my addr 172.21.15.90:0/3792446842 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:04.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.360+0000 7fd67aa00640 1 --2- 172.21.15.90:0/3792446842 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd67c143d20 0x7fd67c15ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:04.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.360+0000 7fd67be00640 1 -- 172.21.15.90:0/3792446842 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd67c143d20 msgr2=0x7fd67c15ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:04.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.360+0000 7fd67be00640 1 --2- 172.21.15.90:0/3792446842 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd67c143d20 0x7fd67c15ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:04.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.360+0000 7fd67be00640 1 -- 172.21.15.90:0/3792446842 >> v1:172.21.15.103:6789/0 conn(0x7fd67c143940 legacy=0x7fd67c06e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:04.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.360+0000 7fd67be00640 1 -- 172.21.15.90:0/3792446842 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd67c071420 con 0x7fd67c06ed60 2024-04-25T01:17:04.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.360+0000 7fd67aa00640 1 --2- 172.21.15.90:0/3792446842 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd67c143d20 0x7fd67c15ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:04.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.361+0000 7fd67be00640 1 --2- 172.21.15.90:0/3792446842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd67c06ed60 0x7fd67c15f230 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd674009e80 tx=0x7fd67402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0ea880b276c3a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:04.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.362+0000 7fd67a000640 1 -- 172.21.15.90:0/3792446842 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd674036070 con 0x7fd67c06ed60 2024-04-25T01:17:04.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.362+0000 7fd67a000640 1 -- 172.21.15.90:0/3792446842 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd674031600 con 0x7fd67c06ed60 2024-04-25T01:17:04.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.362+0000 7fd67a000640 1 -- 172.21.15.90:0/3792446842 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6740319d0 con 0x7fd67c06ed60 2024-04-25T01:17:04.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.362+0000 7fd682200640 1 -- 172.21.15.90:0/3792446842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd67c06ed60 msgr2=0x7fd67c15f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:04.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.362+0000 7fd682200640 1 --2- 172.21.15.90:0/3792446842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd67c06ed60 0x7fd67c15f230 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd674009e80 tx=0x7fd67402a6f0 comp rx=0 tx=0).stop 2024-04-25T01:17:04.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.363+0000 7fd682200640 1 -- 172.21.15.90:0/3792446842 shutdown_connections 2024-04-25T01:17:04.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.363+0000 7fd682200640 1 --2- 172.21.15.90:0/3792446842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd67c06ed60 0x7fd67c15f230 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:04.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.363+0000 7fd682200640 1 --2- 172.21.15.90:0/3792446842 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd67c143d20 0x7fd67c15ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:04.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.363+0000 7fd682200640 1 -- 172.21.15.90:0/3792446842 >> 172.21.15.90:0/3792446842 conn(0x7fd67c0b1820 msgr2=0x7fd67c0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:04.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.363+0000 7fd682200640 1 -- 172.21.15.90:0/3792446842 shutdown_connections 2024-04-25T01:17:04.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.363+0000 7fd682200640 1 -- 172.21.15.90:0/3792446842 wait complete. 2024-04-25T01:17:04.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.365+0000 7fd682200640 1 Processor -- start 2024-04-25T01:17:04.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.365+0000 7fd682200640 1 -- start start 2024-04-25T01:17:04.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.366+0000 7fd682200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd67c06ed60 0x7fd67c13f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:04.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.366+0000 7fd682200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd67c143940 0x7fd67c0ceeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:04.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.366+0000 7fd682200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd67c143d20 0x7fd67c14de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:04.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.366+0000 7fd682200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c071da0 con 0x7fd67c143940 2024-04-25T01:17:04.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.366+0000 7fd682200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c071c20 con 0x7fd67c06ed60 2024-04-25T01:17:04.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.366+0000 7fd682200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd67c071f20 con 0x7fd67c143d20 2024-04-25T01:17:04.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.367+0000 7fd67aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd67c143940 0x7fd67c0ceeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:04.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.367+0000 7fd67aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd67c143940 0x7fd67c0ceeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56608/0 (socket says 172.21.15.90:56608) 2024-04-25T01:17:04.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.367+0000 7fd67aa00640 1 -- 172.21.15.90:0/2543492093 learned_addr learned my addr 172.21.15.90:0/2543492093 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:04.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.367+0000 7fd67b400640 1 --2- 172.21.15.90:0/2543492093 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd67c06ed60 0x7fd67c13f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:04.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.367+0000 7fd67be00640 1 --2- 172.21.15.90:0/2543492093 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd67c143d20 0x7fd67c14de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:04.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.367+0000 7fd67aa00640 1 -- 172.21.15.90:0/2543492093 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd67c143d20 msgr2=0x7fd67c14de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:04.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.367+0000 7fd67aa00640 1 --2- 172.21.15.90:0/2543492093 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd67c143d20 0x7fd67c14de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:04.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.367+0000 7fd67aa00640 1 -- 172.21.15.90:0/2543492093 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd67c06ed60 msgr2=0x7fd67c13f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:04.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.367+0000 7fd67aa00640 1 --2- 172.21.15.90:0/2543492093 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd67c06ed60 0x7fd67c13f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:04.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.367+0000 7fd67aa00640 1 -- 172.21.15.90:0/2543492093 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd67c150c70 con 0x7fd67c143940 2024-04-25T01:17:04.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.367+0000 7fd67b400640 1 --2- 172.21.15.90:0/2543492093 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd67c06ed60 0x7fd67c13f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:17:04.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.367+0000 7fd67aa00640 1 --2- 172.21.15.90:0/2543492093 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd67c143940 0x7fd67c0ceeb0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd66800c7f0 tx=0x7fd66800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:04.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.368+0000 7fd673400640 1 -- 172.21.15.90:0/2543492093 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd668015070 con 0x7fd67c143940 2024-04-25T01:17:04.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.368+0000 7fd673400640 1 -- 172.21.15.90:0/2543492093 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd668007820 con 0x7fd67c143940 2024-04-25T01:17:04.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.368+0000 7fd682200640 1 -- 172.21.15.90:0/2543492093 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd67c1e4a60 con 0x7fd67c143940 2024-04-25T01:17:04.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.368+0000 7fd673400640 1 -- 172.21.15.90:0/2543492093 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd668007bd0 con 0x7fd67c143940 2024-04-25T01:17:04.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.368+0000 7fd682200640 1 -- 172.21.15.90:0/2543492093 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd67c1e5020 con 0x7fd67c143940 2024-04-25T01:17:04.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.369+0000 7fd682200640 1 -- 172.21.15.90:0/2543492093 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd67c068fb0 con 0x7fd67c143940 2024-04-25T01:17:04.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.370+0000 7fd673400640 1 -- 172.21.15.90:0/2543492093 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd668018070 con 0x7fd67c143940 2024-04-25T01:17:04.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.371+0000 7fd673400640 1 --2- 172.21.15.90:0/2543492093 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd65c06be60 0x7fd65c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:04.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.372+0000 7fd673400640 1 -- 172.21.15.90:0/2543492093 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fd66808b340 con 0x7fd67c143940 2024-04-25T01:17:04.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.372+0000 7fd67b400640 1 --2- 172.21.15.90:0/2543492093 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd65c06be60 0x7fd65c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:04.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.372+0000 7fd67b400640 1 --2- 172.21.15.90:0/2543492093 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd65c06be60 0x7fd65c06e320 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd66c003eb0 tx=0x7fd66c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:04.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.376+0000 7fd673400640 1 -- 172.21.15.90:0/2543492093 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd668059910 con 0x7fd67c143940 2024-04-25T01:17:04.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.567+0000 7fd682200640 1 -- 172.21.15.90:0/2543492093 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd67c15da40 con 0x7fd67c143940 2024-04-25T01:17:04.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.570+0000 7fd673400640 1 -- 172.21.15.90:0/2543492093 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18482 (secure 0 0 0) 0x7fd668059210 con 0x7fd67c143940 2024-04-25T01:17:04.554 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:17:04.555 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":26,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:16:27.092855+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:16:20.094512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:16:21.078268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:16:24.670642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:17:04.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.572+0000 7fd682200640 1 -- 172.21.15.90:0/2543492093 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd65c06be60 msgr2=0x7fd65c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:04.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.572+0000 7fd682200640 1 --2- 172.21.15.90:0/2543492093 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd65c06be60 0x7fd65c06e320 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd66c003eb0 tx=0x7fd66c009040 comp rx=0 tx=0).stop 2024-04-25T01:17:04.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.573+0000 7fd682200640 1 -- 172.21.15.90:0/2543492093 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd67c143940 msgr2=0x7fd67c0ceeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:04.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.573+0000 7fd682200640 1 --2- 172.21.15.90:0/2543492093 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd67c143940 0x7fd67c0ceeb0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd66800c7f0 tx=0x7fd66800ccb0 comp rx=0 tx=0).stop 2024-04-25T01:17:04.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.573+0000 7fd682200640 1 -- 172.21.15.90:0/2543492093 shutdown_connections 2024-04-25T01:17:04.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.573+0000 7fd682200640 1 --2- 172.21.15.90:0/2543492093 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd65c06be60 0x7fd65c06e320 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:04.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.573+0000 7fd682200640 1 --2- 172.21.15.90:0/2543492093 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd67c143d20 0x7fd67c14de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:04.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.573+0000 7fd682200640 1 --2- 172.21.15.90:0/2543492093 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd67c143940 0x7fd67c0ceeb0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:04.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.573+0000 7fd682200640 1 --2- 172.21.15.90:0/2543492093 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd67c06ed60 0x7fd67c13f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:04.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.573+0000 7fd682200640 1 -- 172.21.15.90:0/2543492093 >> 172.21.15.90:0/2543492093 conn(0x7fd67c0b1820 msgr2=0x7fd67c0b3cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:04.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.573+0000 7fd682200640 1 -- 172.21.15.90:0/2543492093 shutdown_connections 2024-04-25T01:17:04.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.573+0000 7fd682200640 1 -- 172.21.15.90:0/2543492093 wait complete. 2024-04-25T01:17:04.575 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:17:04.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.783+0000 7f607b200640 1 Processor -- start 2024-04-25T01:17:04.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.783+0000 7f607b200640 1 -- start start 2024-04-25T01:17:04.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.785+0000 7f607b200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f607c06b120 0x7f607c06b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:04.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.785+0000 7f607b200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f607c15ca10 0x7f607c15ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:04.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.785+0000 7f607b200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f607c06ba40 con 0x7f607c0ae730 2024-04-25T01:17:04.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.785+0000 7f607b200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f607c0cef40 con 0x7f607c15ca10 2024-04-25T01:17:04.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.785+0000 7f607b200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f607c15c580 con 0x7f607c06b120 2024-04-25T01:17:04.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.785+0000 7f607a800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f607c15ca10 0x7f607c15ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:04.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.785+0000 7f607a800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f607c15ca10 0x7f607c15ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56618/0 (socket says 172.21.15.90:56618) 2024-04-25T01:17:04.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.785+0000 7f607a800640 1 -- 172.21.15.90:0/2300846376 learned_addr learned my addr 172.21.15.90:0/2300846376 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:04.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.785+0000 7f6079e00640 1 --2- 172.21.15.90:0/2300846376 >> v2:172.21.15.103:3300/0 conn(0x7f607c06b120 0x7f607c06b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:04.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.785+0000 7f607a800640 1 -- 172.21.15.90:0/2300846376 >> v2:172.21.15.103:3300/0 conn(0x7f607c06b120 msgr2=0x7f607c06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:04.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.785+0000 7f607a800640 1 --2- 172.21.15.90:0/2300846376 >> v2:172.21.15.103:3300/0 conn(0x7f607c06b120 0x7f607c06b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:04.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.785+0000 7f607a800640 1 -- 172.21.15.90:0/2300846376 >> v1:172.21.15.90:6789/0 conn(0x7f607c0ae730 legacy=0x7f607c0cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:04.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.786+0000 7f607a800640 1 -- 172.21.15.90:0/2300846376 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f607c15c220 con 0x7f607c15ca10 2024-04-25T01:17:04.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.786+0000 7f6079e00640 1 --2- 172.21.15.90:0/2300846376 >> v2:172.21.15.103:3300/0 conn(0x7f607c06b120 0x7f607c06b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:04.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.786+0000 7f607a800640 1 --2- 172.21.15.90:0/2300846376 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f607c15ca10 0x7f607c15ce80 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f607c0b2560 tx=0x7f607402a740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef3c9cca2bc477ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:04.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.787+0000 7f6078a00640 1 -- 172.21.15.90:0/2300846376 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6074036070 con 0x7f607c15ca10 2024-04-25T01:17:04.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.787+0000 7f6078a00640 1 -- 172.21.15.90:0/2300846376 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6074031600 con 0x7f607c15ca10 2024-04-25T01:17:04.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.787+0000 7f6078a00640 1 -- 172.21.15.90:0/2300846376 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60740319d0 con 0x7f607c15ca10 2024-04-25T01:17:04.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.787+0000 7f607b200640 1 -- 172.21.15.90:0/2300846376 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f607c15ca10 msgr2=0x7f607c15ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:04.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.787+0000 7f607b200640 1 --2- 172.21.15.90:0/2300846376 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f607c15ca10 0x7f607c15ce80 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f607c0b2560 tx=0x7f607402a740 comp rx=0 tx=0).stop 2024-04-25T01:17:04.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.787+0000 7f607b200640 1 -- 172.21.15.90:0/2300846376 shutdown_connections 2024-04-25T01:17:04.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.787+0000 7f607b200640 1 --2- 172.21.15.90:0/2300846376 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f607c15ca10 0x7f607c15ce80 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:04.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.787+0000 7f607b200640 1 --2- 172.21.15.90:0/2300846376 >> v2:172.21.15.103:3300/0 conn(0x7f607c06b120 0x7f607c06b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:04.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.787+0000 7f607b200640 1 -- 172.21.15.90:0/2300846376 >> 172.21.15.90:0/2300846376 conn(0x7f607c0b1660 msgr2=0x7f607c0b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:04.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.788+0000 7f607b200640 1 -- 172.21.15.90:0/2300846376 shutdown_connections 2024-04-25T01:17:04.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.788+0000 7f607b200640 1 -- 172.21.15.90:0/2300846376 wait complete. 2024-04-25T01:17:04.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.789+0000 7f607b200640 1 Processor -- start 2024-04-25T01:17:04.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.789+0000 7f607b200640 1 -- start start 2024-04-25T01:17:04.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.790+0000 7f607b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f607c06b120 0x7f607c0a94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:04.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.790+0000 7f607b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f607c0ae730 0x7f607c1f5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:04.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.790+0000 7f607b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f607c15ca10 0x7f607c1f5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:04.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.790+0000 7f607b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f607c0af240 con 0x7f607c0ae730 2024-04-25T01:17:04.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.790+0000 7f607b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f607c0aef60 con 0x7f607c06b120 2024-04-25T01:17:04.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.790+0000 7f607b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f607c153080 con 0x7f607c15ca10 2024-04-25T01:17:04.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.791+0000 7f6079400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f607c0ae730 0x7f607c1f5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:04.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.791+0000 7f6079400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f607c0ae730 0x7f607c1f5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44640/0 (socket says 172.21.15.90:44640) 2024-04-25T01:17:04.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.791+0000 7f6079400640 1 -- 172.21.15.90:0/2558706803 learned_addr learned my addr 172.21.15.90:0/2558706803 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:04.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.791+0000 7f6079e00640 1 --2- 172.21.15.90:0/2558706803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f607c06b120 0x7f607c0a94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:04.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.791+0000 7f607a800640 1 --2- 172.21.15.90:0/2558706803 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f607c15ca10 0x7f607c1f5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:04.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.791+0000 7f6079400640 1 -- 172.21.15.90:0/2558706803 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f607c15ca10 msgr2=0x7f607c1f5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:04.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.791+0000 7f6079400640 1 --2- 172.21.15.90:0/2558706803 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f607c15ca10 0x7f607c1f5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:04.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.791+0000 7f6079400640 1 -- 172.21.15.90:0/2558706803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f607c06b120 msgr2=0x7f607c0a94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:04.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.791+0000 7f6079400640 1 --2- 172.21.15.90:0/2558706803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f607c06b120 0x7f607c0a94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:04.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.791+0000 7f6079400640 1 -- 172.21.15.90:0/2558706803 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f607c0a9e40 con 0x7f607c0ae730 2024-04-25T01:17:04.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.791+0000 7f6079e00640 1 --2- 172.21.15.90:0/2558706803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f607c06b120 0x7f607c0a94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:04.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.791+0000 7f6079400640 1 --2- 172.21.15.90:0/2558706803 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f607c0ae730 0x7f607c1f5a30 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f606800c400 tx=0x7f606800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:04.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.792+0000 7f6072000640 1 -- 172.21.15.90:0/2558706803 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60680052d0 con 0x7f607c0ae730 2024-04-25T01:17:04.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.792+0000 7f6072000640 1 -- 172.21.15.90:0/2558706803 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6068005490 con 0x7f607c0ae730 2024-04-25T01:17:04.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.792+0000 7f6072000640 1 -- 172.21.15.90:0/2558706803 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6068007260 con 0x7f607c0ae730 2024-04-25T01:17:04.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.792+0000 7f607b200640 1 -- 172.21.15.90:0/2558706803 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f607c0aa120 con 0x7f607c0ae730 2024-04-25T01:17:04.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.792+0000 7f607b200640 1 -- 172.21.15.90:0/2558706803 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f607c0aa6e0 con 0x7f607c0ae730 2024-04-25T01:17:04.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.793+0000 7f607b200640 1 -- 172.21.15.90:0/2558706803 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f607c075350 con 0x7f607c0ae730 2024-04-25T01:17:04.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.794+0000 7f6072000640 1 -- 172.21.15.90:0/2558706803 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6068007400 con 0x7f607c0ae730 2024-04-25T01:17:04.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.795+0000 7f6072000640 1 --2- 172.21.15.90:0/2558706803 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f605c06be30 0x7f605c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:04.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.795+0000 7f6072000640 1 -- 172.21.15.90:0/2558706803 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f606808a700 con 0x7f607c0ae730 2024-04-25T01:17:04.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.795+0000 7f6079e00640 1 --2- 172.21.15.90:0/2558706803 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f605c06be30 0x7f605c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:04.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.795+0000 7f6079e00640 1 --2- 172.21.15.90:0/2558706803 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f605c06be30 0x7f605c06e2f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f606c006fd0 tx=0x7f606c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:04.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:04.799+0000 7f6072000640 1 -- 172.21.15.90:0/2558706803 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6068058d60 con 0x7f607c0ae730 2024-04-25T01:17:05.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.021+0000 7f607b200640 1 -- 172.21.15.90:0/2558706803 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f607c0cb940 con 0x7f607c0ae730 2024-04-25T01:17:05.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.022+0000 7f6072000640 1 -- 172.21.15.90:0/2558706803 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5180 (secure 0 0 0) 0x7f6068058660 con 0x7f607c0ae730 2024-04-25T01:17:05.006 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:17:05.006 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6834/2605945289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":2605945289},{"type":"v1","addr":"172.21.15.103:6834","nonce":2605945289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6836/704639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":704639426},{"type":"v1","addr":"172.21.15.90:6836","nonce":704639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4245,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3477867755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6835","nonce":3477867755},{"type":"v1","addr":"172.21.15.90:6837","nonce":3477867755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:16:22.060018+0000","modified":"2024-04-25T01:16:50.405431+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6835/4291578198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6833","nonce":4291578198},{"type":"v1","addr":"172.21.15.103:6835","nonce":4291578198}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1}]} 2024-04-25T01:17:05.007 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 9 2024-04-25T01:17:05.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.025+0000 7f607b200640 1 -- 172.21.15.90:0/2558706803 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f605c06be30 msgr2=0x7f605c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:05.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.025+0000 7f607b200640 1 --2- 172.21.15.90:0/2558706803 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f605c06be30 0x7f605c06e2f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f606c006fd0 tx=0x7f606c008040 comp rx=0 tx=0).stop 2024-04-25T01:17:05.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.025+0000 7f607b200640 1 -- 172.21.15.90:0/2558706803 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f607c0ae730 msgr2=0x7f607c1f5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:05.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.025+0000 7f607b200640 1 --2- 172.21.15.90:0/2558706803 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f607c0ae730 0x7f607c1f5a30 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f606800c400 tx=0x7f606800c8c0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.025+0000 7f607b200640 1 -- 172.21.15.90:0/2558706803 shutdown_connections 2024-04-25T01:17:05.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.025+0000 7f607b200640 1 --2- 172.21.15.90:0/2558706803 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f605c06be30 0x7f605c06e2f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.025+0000 7f607b200640 1 --2- 172.21.15.90:0/2558706803 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f607c15ca10 0x7f607c1f5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.025+0000 7f607b200640 1 --2- 172.21.15.90:0/2558706803 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f607c0ae730 0x7f607c1f5a30 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.025+0000 7f607b200640 1 --2- 172.21.15.90:0/2558706803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f607c06b120 0x7f607c0a94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.025+0000 7f607b200640 1 -- 172.21.15.90:0/2558706803 >> 172.21.15.90:0/2558706803 conn(0x7f607c0b1660 msgr2=0x7f607c06e330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:05.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.025+0000 7f607b200640 1 -- 172.21.15.90:0/2558706803 shutdown_connections 2024-04-25T01:17:05.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.025+0000 7f607b200640 1 -- 172.21.15.90:0/2558706803 wait complete. 2024-04-25T01:17:05.427 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T01:16:22.060018+0000', 'modified': '2024-04-25T01:16:50.405431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4254}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.103:6835/4291578198', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6833', 'nonce': 4291578198}, {'type': 'v1', 'addr': '172.21.15.103:6835', 'nonce': 4291578198}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4254, 'qdb_cluster': [4254]} 2024-04-25T01:17:05.428 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T01:17:05.428 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-25T01:17:05.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.643+0000 7f0ddd800640 1 Processor -- start 2024-04-25T01:17:05.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.644+0000 7f0ddd800640 1 -- start start 2024-04-25T01:17:05.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.645+0000 7f0ddd800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0dd8161760 0x7f0dd80d2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:05.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.645+0000 7f0ddd800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dd80cebb0 con 0x7f0dd80b5340 2024-04-25T01:17:05.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.645+0000 7f0ddd800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dd80d46d0 con 0x7f0dd80d0f60 2024-04-25T01:17:05.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.645+0000 7f0ddd800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f0dd80d49b0 con 0x7f0dd8161760 2024-04-25T01:17:05.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.645+0000 7f0dd7e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0dd8161760 0x7f0dd80d2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:05.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.645+0000 7f0dd7e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0dd8161760 0x7f0dd80d2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44650/0 (socket says 172.21.15.90:44650) 2024-04-25T01:17:05.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.645+0000 7f0dd7400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f0dd80b5340 0x7f0dd80d0850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:49200/0 (socket says 172.21.15.90:49200) 2024-04-25T01:17:05.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.645+0000 7f0dd7e00640 1 -- 172.21.15.90:0/122347104 learned_addr learned my addr 172.21.15.90:0/122347104 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:05.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.646+0000 7f0dd7e00640 1 -- 172.21.15.90:0/122347104 >> v1:172.21.15.103:6789/0 conn(0x7f0dd80d0f60 legacy=0x7f0dd80cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:05.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.646+0000 7f0dd7e00640 1 -- 172.21.15.90:0/122347104 >> v1:172.21.15.90:6789/0 conn(0x7f0dd80b5340 legacy=0x7f0dd80d0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:05.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.646+0000 7f0dd7e00640 1 -- 172.21.15.90:0/122347104 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dd80cee50 con 0x7f0dd8161760 2024-04-25T01:17:05.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.647+0000 7f0dd7e00640 1 --2- 172.21.15.90:0/122347104 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dd8161760 0x7f0dd80d2b10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0dc8009870 tx=0x7f0dc802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bebc825a1934ce80 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:05.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.647+0000 7f0dd6a00640 1 -- 172.21.15.90:0/122347104 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc8037070 con 0x7f0dd8161760 2024-04-25T01:17:05.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.647+0000 7f0dd6a00640 1 -- 172.21.15.90:0/122347104 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dc8032440 con 0x7f0dd8161760 2024-04-25T01:17:05.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.647+0000 7f0dd6a00640 1 -- 172.21.15.90:0/122347104 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc8032810 con 0x7f0dd8161760 2024-04-25T01:17:05.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.648+0000 7f0ddd800640 1 -- 172.21.15.90:0/122347104 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dd8161760 msgr2=0x7f0dd80d2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:05.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.648+0000 7f0ddd800640 1 --2- 172.21.15.90:0/122347104 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dd8161760 0x7f0dd80d2b10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0dc8009870 tx=0x7f0dc802a210 comp rx=0 tx=0).stop 2024-04-25T01:17:05.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.648+0000 7f0ddd800640 1 -- 172.21.15.90:0/122347104 shutdown_connections 2024-04-25T01:17:05.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.648+0000 7f0ddd800640 1 --2- 172.21.15.90:0/122347104 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dd8161760 0x7f0dd80d2b10 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.648+0000 7f0ddd800640 1 -- 172.21.15.90:0/122347104 >> 172.21.15.90:0/122347104 conn(0x7f0dd809d170 msgr2=0x7f0dd80d3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:05.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.648+0000 7f0ddd800640 1 -- 172.21.15.90:0/122347104 shutdown_connections 2024-04-25T01:17:05.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.648+0000 7f0ddd800640 1 -- 172.21.15.90:0/122347104 wait complete. 2024-04-25T01:17:05.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.649+0000 7f0ddd800640 1 Processor -- start 2024-04-25T01:17:05.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.650+0000 7f0ddd800640 1 -- start start 2024-04-25T01:17:05.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.651+0000 7f0ddd800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0dd80b5340 0x7f0dd80c8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:05.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.651+0000 7f0ddd800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0dd80d0f60 0x7f0dd80ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:05.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.651+0000 7f0ddd800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dd8161760 0x7f0dd815a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:05.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.651+0000 7f0ddd800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd8161b40 con 0x7f0dd8161760 2024-04-25T01:17:05.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.651+0000 7f0ddd800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd80b3740 con 0x7f0dd80d0f60 2024-04-25T01:17:05.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.651+0000 7f0ddd800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0dd815f1c0 con 0x7f0dd80b5340 2024-04-25T01:17:05.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.651+0000 7f0ddce00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dd8161760 0x7f0dd815a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:05.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.651+0000 7f0ddce00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dd8161760 0x7f0dd815a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44654/0 (socket says 172.21.15.90:44654) 2024-04-25T01:17:05.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.651+0000 7f0ddce00640 1 -- 172.21.15.90:0/246273537 learned_addr learned my addr 172.21.15.90:0/246273537 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:05.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.651+0000 7f0dd7400640 1 --2- 172.21.15.90:0/246273537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0dd80d0f60 0x7f0dd80ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:05.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.651+0000 7f0dd7e00640 1 --2- 172.21.15.90:0/246273537 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0dd80b5340 0x7f0dd80c8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:05.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.651+0000 7f0ddce00640 1 -- 172.21.15.90:0/246273537 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0dd80b5340 msgr2=0x7f0dd80c8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:05.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.651+0000 7f0ddce00640 1 --2- 172.21.15.90:0/246273537 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0dd80b5340 0x7f0dd80c8140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.651+0000 7f0ddce00640 1 -- 172.21.15.90:0/246273537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0dd80d0f60 msgr2=0x7f0dd80ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:05.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.651+0000 7f0ddce00640 1 --2- 172.21.15.90:0/246273537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0dd80d0f60 0x7f0dd80ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.651+0000 7f0ddce00640 1 -- 172.21.15.90:0/246273537 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dd80c8720 con 0x7f0dd8161760 2024-04-25T01:17:05.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.651+0000 7f0dd7400640 1 --2- 172.21.15.90:0/246273537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0dd80d0f60 0x7f0dd80ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:05.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.652+0000 7f0ddce00640 1 --2- 172.21.15.90:0/246273537 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dd8161760 0x7f0dd815a1a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0dd000c3e0 tx=0x7f0dd000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:05.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.652+0000 7f0dcfe00640 1 -- 172.21.15.90:0/246273537 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd0011790 con 0x7f0dd8161760 2024-04-25T01:17:05.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.652+0000 7f0dcfe00640 1 -- 172.21.15.90:0/246273537 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dd0017070 con 0x7f0dd8161760 2024-04-25T01:17:05.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.652+0000 7f0dcfe00640 1 -- 172.21.15.90:0/246273537 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd0011bb0 con 0x7f0dd8161760 2024-04-25T01:17:05.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.652+0000 7f0ddd800640 1 -- 172.21.15.90:0/246273537 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dd80c8a00 con 0x7f0dd8161760 2024-04-25T01:17:05.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.652+0000 7f0ddd800640 1 -- 172.21.15.90:0/246273537 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dd815ee50 con 0x7f0dd8161760 2024-04-25T01:17:05.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.654+0000 7f0dcfe00640 1 -- 172.21.15.90:0/246273537 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0dd001c460 con 0x7f0dd8161760 2024-04-25T01:17:05.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.655+0000 7f0dcfe00640 1 --2- 172.21.15.90:0/246273537 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0db406be60 0x7f0db406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:05.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.655+0000 7f0dcfe00640 1 -- 172.21.15.90:0/246273537 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f0dd008af80 con 0x7f0dd8161760 2024-04-25T01:17:05.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.656+0000 7f0dd7e00640 1 --2- 172.21.15.90:0/246273537 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0db406be60 0x7f0db406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:05.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.656+0000 7f0de1dd6740 1 Processor -- start 2024-04-25T01:17:05.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.656+0000 7f0de1dd6740 1 -- start start 2024-04-25T01:17:05.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.656+0000 7f0dd7e00640 1 --2- 172.21.15.90:0/246273537 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0db406be60 0x7f0db406e320 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0dc8034000 tx=0x7f0dc802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:05.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.657+0000 7f0de1dd6740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563b9b238720 0x563b9b238b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:05.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.657+0000 7f0de1dd6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b9b255fc0 0x563b9b256510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:05.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.657+0000 7f0de1dd6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b9b20e400 0x563b9b254680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:05.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.657+0000 7f0de1dd6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563b9b10ded0 con 0x563b9b255fc0 2024-04-25T01:17:05.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.657+0000 7f0de1dd6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563b9b170df0 con 0x563b9b20e400 2024-04-25T01:17:05.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.657+0000 7f0de1dd6740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x563b9b1198d0 con 0x563b9b238720 2024-04-25T01:17:05.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.657+0000 7f0dd7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b9b255fc0 0x563b9b256510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:05.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.657+0000 7f0dd7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b9b255fc0 0x563b9b256510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44668/0 (socket says 172.21.15.90:44668) 2024-04-25T01:17:05.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.657+0000 7f0dd7400640 1 -- 172.21.15.90:0/3708372262 learned_addr learned my addr 172.21.15.90:0/3708372262 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:05.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.657+0000 7f0dd7400640 1 -- 172.21.15.90:0/3708372262 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563b9b238720 msgr2=0x563b9b238b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:05.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.657+0000 7f0dd7400640 1 --2- 172.21.15.90:0/3708372262 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563b9b238720 0x563b9b238b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.657+0000 7f0ddce00640 1 --2- 172.21.15.90:0/3708372262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b9b20e400 0x563b9b254680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:05.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.657+0000 7f0dd7400640 1 -- 172.21.15.90:0/3708372262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b9b20e400 msgr2=0x563b9b254680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:05.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.658+0000 7f0dd7400640 1 --2- 172.21.15.90:0/3708372262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b9b20e400 0x563b9b254680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.658+0000 7f0dd7400640 1 -- 172.21.15.90:0/3708372262 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b9ae852d0 con 0x563b9b255fc0 2024-04-25T01:17:05.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.658+0000 7f0dd7400640 1 --2- 172.21.15.90:0/3708372262 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b9b255fc0 0x563b9b256510 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0001e40 tx=0x7f0dc000d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=158dfb7262b69e5c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:05.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.658+0000 7f0dccc00640 1 -- 172.21.15.90:0/3708372262 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc0018070 con 0x563b9b255fc0 2024-04-25T01:17:05.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.658+0000 7f0dccc00640 1 -- 172.21.15.90:0/3708372262 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dc000de30 con 0x563b9b255fc0 2024-04-25T01:17:05.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.658+0000 7f0dccc00640 1 -- 172.21.15.90:0/3708372262 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc0005db0 con 0x563b9b255fc0 2024-04-25T01:17:05.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.659+0000 7f0de1dd6740 1 -- 172.21.15.90:0/3708372262 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b9b255fc0 msgr2=0x563b9b256510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:05.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.659+0000 7f0de1dd6740 1 --2- 172.21.15.90:0/3708372262 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b9b255fc0 0x563b9b256510 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0001e40 tx=0x7f0dc000d520 comp rx=0 tx=0).stop 2024-04-25T01:17:05.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.659+0000 7f0de1dd6740 1 -- 172.21.15.90:0/3708372262 shutdown_connections 2024-04-25T01:17:05.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.659+0000 7f0de1dd6740 1 --2- 172.21.15.90:0/3708372262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b9b20e400 0x563b9b254680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.659+0000 7f0de1dd6740 1 --2- 172.21.15.90:0/3708372262 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b9b255fc0 0x563b9b256510 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.659+0000 7f0de1dd6740 1 --2- 172.21.15.90:0/3708372262 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563b9b238720 0x563b9b238b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.659+0000 7f0de1dd6740 1 -- 172.21.15.90:0/3708372262 >> 172.21.15.90:0/3708372262 conn(0x563b9b20ad30 msgr2=0x563b9b253530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:05.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.659+0000 7f0de1dd6740 1 -- 172.21.15.90:0/3708372262 shutdown_connections 2024-04-25T01:17:05.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.659+0000 7f0de1dd6740 1 -- 172.21.15.90:0/3708372262 wait complete. 2024-04-25T01:17:05.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.659+0000 7f0de1dd6740 1 -- start start 2024-04-25T01:17:05.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.660+0000 7f0de1dd6740 1 Processor -- start 2024-04-25T01:17:05.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.661+0000 7f0de1dd6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b9b238720 0x563b9b2350a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:05.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.661+0000 7f0de1dd6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b9b255fc0 0x563b9b23c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:05.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.661+0000 7f0de1dd6740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563b9b24a5e0 0x563b9b24aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:05.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.661+0000 7f0de1dd6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563b9b12cd20 con 0x563b9b238720 2024-04-25T01:17:05.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.661+0000 7f0de1dd6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563b9b16cdb0 con 0x563b9b255fc0 2024-04-25T01:17:05.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.661+0000 7f0de1dd6740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x563b9b20f1a0 con 0x563b9b24a5e0 2024-04-25T01:17:05.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.661+0000 7f0de1dd6740 10 client.0 _ll_register_callbacks cb 0x563b9b167f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:17:05.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.661+0000 7f0dd7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b9b238720 0x563b9b2350a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:05.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.661+0000 7f0dd7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b9b238720 0x563b9b2350a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44670/0 (socket says 172.21.15.90:44670) 2024-04-25T01:17:05.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.661+0000 7f0dd7e00640 1 -- 172.21.15.90:0/3919614831 learned_addr learned my addr 172.21.15.90:0/3919614831 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:05.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.661+0000 7f0dd7e00640 1 -- 172.21.15.90:0/3919614831 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563b9b24a5e0 msgr2=0x563b9b24aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:05.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.661+0000 7f0dd7e00640 1 --2- 172.21.15.90:0/3919614831 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563b9b24a5e0 0x563b9b24aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.661+0000 7f0dd7e00640 1 -- 172.21.15.90:0/3919614831 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b9b255fc0 msgr2=0x563b9b23c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:05.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.661+0000 7f0dd7400640 1 --2- 172.21.15.90:0/3919614831 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b9b255fc0 0x563b9b23c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:05.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.661+0000 7f0dd7e00640 1 --2- 172.21.15.90:0/3919614831 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b9b255fc0 0x563b9b23c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.661+0000 7f0dd7e00640 1 -- 172.21.15.90:0/3919614831 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b9af4f6b0 con 0x563b9b238720 2024-04-25T01:17:05.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.662+0000 7f0dd7e00640 1 --2- 172.21.15.90:0/3919614831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b9b238720 0x563b9b2350a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0dc80090b0 tx=0x7f0dc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5844c62f354cf95f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:05.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.662+0000 7f0dc5600640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T01:17:05.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.662+0000 7f0dc5600640 1 -- 172.21.15.90:0/3919614831 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc804e060 con 0x563b9b238720 2024-04-25T01:17:05.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.662+0000 7f0dc5600640 1 -- 172.21.15.90:0/3919614831 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f0dc8032d00 con 0x563b9b238720 2024-04-25T01:17:05.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.662+0000 7f0de1dd6740 1 -- 172.21.15.90:0/3919614831 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0dc000de30 con 0x563b9b238720 2024-04-25T01:17:05.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.662+0000 7f0dc5600640 1 -- 172.21.15.90:0/3919614831 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) ==== 24+0+0 (secure 0 0 0) 0x7f0dc8037040 con 0x563b9b238720 2024-04-25T01:17:05.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.662+0000 7f0de1dd6740 10 client.4851 fetch_fsmap learned FSMap version 9 2024-04-25T01:17:05.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.662+0000 7f0de1dd6740 1 -- 172.21.15.90:0/3919614831 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) -- 0x563b9b24c230 con 0x563b9b238720 2024-04-25T01:17:05.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.662+0000 7f0dc5600640 1 -- 172.21.15.90:0/3919614831 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dc000de30 con 0x563b9b238720 2024-04-25T01:17:05.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.662+0000 7f0dc5600640 1 -- 172.21.15.90:0/3919614831 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 9) ==== 3502+0+0 (secure 0 0 0) 0x7f0dc8057020 con 0x563b9b238720 2024-04-25T01:17:05.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.663+0000 7f0de1dd6740 10 client.4851 fetch_fsmap finished waiting for FSMap version 9 2024-04-25T01:17:05.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.663+0000 7f0de1dd6740 10 client.4851 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-25T01:17:05.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.663+0000 7f0de1dd6740 20 client.4851 populate_metadata read hostname 'smithi090' 2024-04-25T01:17:05.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.663+0000 7f0de1dd6740 1 --2- 172.21.15.90:0/3919614831 >> [v2:172.21.15.103:6833/4291578198,v1:172.21.15.103:6835/4291578198] conn(0x563b9b267780 0x563b9b2724f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:05.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.663+0000 7f0de1dd6740 4 client.4851 mds_command: new command op to 4254 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:17:05.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.663+0000 7f0de1dd6740 1 -- 172.21.15.90:0/3919614831 --> [v2:172.21.15.103:6833/4291578198,v1:172.21.15.103:6835/4291578198] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b9b17e330 con 0x563b9b267780 2024-04-25T01:17:05.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.663+0000 7f0dd7400640 1 --2- 172.21.15.90:0/3919614831 >> [v2:172.21.15.103:6833/4291578198,v1:172.21.15.103:6835/4291578198] conn(0x563b9b267780 0x563b9b2724f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:05.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.664+0000 7f0dd7400640 1 --2- 172.21.15.90:0/3919614831 >> [v2:172.21.15.103:6833/4291578198,v1:172.21.15.103:6835/4291578198] conn(0x563b9b267780 0x563b9b2724f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99c9d3253b65a2a2 server_cookie=82589bb7854e3aca in_seq=0 out_seq=0 2024-04-25T01:17:05.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.664+0000 7f0dc5600640 10 client.4851 ms_handle_connect on v2:172.21.15.103:6833/4291578198 2024-04-25T01:17:05.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.666+0000 7f0dc5600640 1 -- 172.21.15.90:0/3919614831 <== mds.0 v2:172.21.15.103:6833/4291578198 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563b9b17e330 con 0x563b9b267780 2024-04-25T01:17:05.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.666+0000 7f0dc5600640 10 client.4851 handle_command_reply: tid=0 2024-04-25T01:17:05.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.666+0000 7f0de1dd6740 1 client.4851 shutdown 2024-04-25T01:17:05.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.666+0000 7f0de1dd6740 1 -- 172.21.15.90:0/3919614831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b9b238720 msgr2=0x563b9b2350a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:05.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.666+0000 7f0de1dd6740 1 --2- 172.21.15.90:0/3919614831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b9b238720 0x563b9b2350a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0dc80090b0 tx=0x7f0dc80046c0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.666+0000 7f0de1dd6740 1 -- 172.21.15.90:0/3919614831 shutdown_connections 2024-04-25T01:17:05.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.666+0000 7f0de1dd6740 1 --2- 172.21.15.90:0/3919614831 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563b9b24a5e0 0x563b9b24aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.666+0000 7f0de1dd6740 1 --2- 172.21.15.90:0/3919614831 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b9b255fc0 0x563b9b23c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.666+0000 7f0de1dd6740 1 --2- 172.21.15.90:0/3919614831 >> [v2:172.21.15.103:6833/4291578198,v1:172.21.15.103:6835/4291578198] conn(0x563b9b267780 0x563b9b2724f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.666+0000 7f0de1dd6740 1 --2- 172.21.15.90:0/3919614831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b9b238720 0x563b9b2350a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.666+0000 7f0de1dd6740 1 -- 172.21.15.90:0/3919614831 >> 172.21.15.90:0/3919614831 conn(0x563b9b20e400 msgr2=0x563b9b252d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:05.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.666+0000 7f0dc5600640 0 client.4851 ms_handle_reset on v2:172.21.15.103:6833/4291578198 2024-04-25T01:17:05.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.667+0000 7f0de1dd6740 1 -- 172.21.15.90:0/3919614831 shutdown_connections 2024-04-25T01:17:05.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.667+0000 7f0de1dd6740 1 -- 172.21.15.90:0/3919614831 wait complete. 2024-04-25T01:17:05.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.667+0000 7f0de1dd6740 20 client.4851 trim_cache size 0 max 16384 2024-04-25T01:17:05.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.686+0000 7f0de1dd6740 1 Processor -- start 2024-04-25T01:17:05.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.686+0000 7f0de1dd6740 1 -- start start 2024-04-25T01:17:05.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.687+0000 7f0de1dd6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b9b238720 0x563b9b277b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:05.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.687+0000 7f0de1dd6740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563b9b24a5e0 0x563b9b253810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:05.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.687+0000 7f0de1dd6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b9b255fc0 0x563b9b20e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:05.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.687+0000 7f0de1dd6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd8201a10 con 0x563b9b238720 2024-04-25T01:17:05.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.687+0000 7f0de1dd6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd8200a90 con 0x563b9b255fc0 2024-04-25T01:17:05.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.687+0000 7f0de1dd6740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0dd81ffe20 con 0x563b9b24a5e0 2024-04-25T01:17:05.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.687+0000 7f0dd7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b9b238720 0x563b9b277b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:05.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.687+0000 7f0dd7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b9b238720 0x563b9b277b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44686/0 (socket says 172.21.15.90:44686) 2024-04-25T01:17:05.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.687+0000 7f0dd7e00640 1 -- 172.21.15.90:0/458150906 learned_addr learned my addr 172.21.15.90:0/458150906 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:05.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.687+0000 7f0dd7e00640 1 -- 172.21.15.90:0/458150906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563b9b24a5e0 msgr2=0x563b9b253810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:05.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.687+0000 7f0dd7e00640 1 --2- 172.21.15.90:0/458150906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563b9b24a5e0 0x563b9b253810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.687+0000 7f0dd7e00640 1 -- 172.21.15.90:0/458150906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b9b255fc0 msgr2=0x563b9b20e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:05.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.687+0000 7f0dd7e00640 1 --2- 172.21.15.90:0/458150906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b9b255fc0 0x563b9b20e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.687+0000 7f0dd7e00640 1 -- 172.21.15.90:0/458150906 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b9b24add0 con 0x563b9b238720 2024-04-25T01:17:05.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.687+0000 7f0dd7e00640 1 --2- 172.21.15.90:0/458150906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b9b238720 0x563b9b277b90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0dc8000c00 tx=0x7f0dc8043000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e1c8ad74f4ed685 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:05.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.688+0000 7f0dc7e00640 1 -- 172.21.15.90:0/458150906 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc8032440 con 0x563b9b238720 2024-04-25T01:17:05.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.688+0000 7f0dc7e00640 1 -- 172.21.15.90:0/458150906 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dc804f250 con 0x563b9b238720 2024-04-25T01:17:05.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.688+0000 7f0dc7e00640 1 -- 172.21.15.90:0/458150906 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc804f640 con 0x563b9b238720 2024-04-25T01:17:05.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.688+0000 7f0de1dd6740 1 -- 172.21.15.90:0/458150906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b9b238720 msgr2=0x563b9b277b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:05.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.688+0000 7f0de1dd6740 1 --2- 172.21.15.90:0/458150906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b9b238720 0x563b9b277b90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0dc8000c00 tx=0x7f0dc8043000 comp rx=0 tx=0).stop 2024-04-25T01:17:05.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.688+0000 7f0de1dd6740 1 -- 172.21.15.90:0/458150906 shutdown_connections 2024-04-25T01:17:05.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.688+0000 7f0de1dd6740 1 --2- 172.21.15.90:0/458150906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b9b255fc0 0x563b9b20e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.688+0000 7f0de1dd6740 1 --2- 172.21.15.90:0/458150906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563b9b24a5e0 0x563b9b253810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.688+0000 7f0de1dd6740 1 --2- 172.21.15.90:0/458150906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b9b238720 0x563b9b277b90 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.688+0000 7f0de1dd6740 1 -- 172.21.15.90:0/458150906 >> 172.21.15.90:0/458150906 conn(0x563b9b20e400 msgr2=0x563b9b2640c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:05.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.688+0000 7f0de1dd6740 1 -- 172.21.15.90:0/458150906 shutdown_connections 2024-04-25T01:17:05.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.689+0000 7f0de1dd6740 1 -- 172.21.15.90:0/458150906 wait complete. 2024-04-25T01:17:05.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.689+0000 7f0de1dd6740 1 -- start start 2024-04-25T01:17:05.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.689+0000 7f0de1dd6740 1 Processor -- start 2024-04-25T01:17:05.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.690+0000 7f0de1dd6740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563b9b238720 0x563b9b2cd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:05.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.690+0000 7f0de1dd6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b9b24a5e0 0x563b9b20e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:05.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.690+0000 7f0de1dd6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b9b255fc0 0x563b9b20ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:05.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.690+0000 7f0de1dd6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563b9b24c510 con 0x563b9b255fc0 2024-04-25T01:17:05.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.690+0000 7f0de1dd6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd8200440 con 0x563b9b24a5e0 2024-04-25T01:17:05.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.690+0000 7f0de1dd6740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x563b9b23a520 con 0x563b9b238720 2024-04-25T01:17:05.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.690+0000 7f0de1dd6740 10 client.0 _ll_register_callbacks cb 0x7f0dc8035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:17:05.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.691+0000 7f0ddce00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b9b255fc0 0x563b9b20ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:05.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.691+0000 7f0ddce00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b9b255fc0 0x563b9b20ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44702/0 (socket says 172.21.15.90:44702) 2024-04-25T01:17:05.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.691+0000 7f0ddce00640 1 -- 172.21.15.90:0/2252197322 learned_addr learned my addr 172.21.15.90:0/2252197322 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:05.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.691+0000 7f0ddce00640 1 -- 172.21.15.90:0/2252197322 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563b9b238720 msgr2=0x563b9b2cd8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:05.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.691+0000 7f0ddce00640 1 --2- 172.21.15.90:0/2252197322 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563b9b238720 0x563b9b2cd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.691+0000 7f0dd7400640 1 --2- 172.21.15.90:0/2252197322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b9b24a5e0 0x563b9b20e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:05.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.691+0000 7f0ddce00640 1 -- 172.21.15.90:0/2252197322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b9b24a5e0 msgr2=0x563b9b20e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:05.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.691+0000 7f0ddce00640 1 --2- 172.21.15.90:0/2252197322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b9b24a5e0 0x563b9b20e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.691+0000 7f0ddce00640 1 -- 172.21.15.90:0/2252197322 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b9b24b650 con 0x563b9b255fc0 2024-04-25T01:17:05.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.691+0000 7f0ddce00640 1 --2- 172.21.15.90:0/2252197322 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b9b255fc0 0x563b9b20ed20 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0058e70 tx=0x7f0dd0058ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c3ebd60450b7e05 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:05.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.691+0000 7f0dc5600640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T01:17:05.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.691+0000 7f0dc5600640 1 -- 172.21.15.90:0/2252197322 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd0087060 con 0x563b9b255fc0 2024-04-25T01:17:05.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.691+0000 7f0dc5600640 1 -- 172.21.15.90:0/2252197322 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f0dd0055550 con 0x563b9b255fc0 2024-04-25T01:17:05.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.691+0000 7f0de1dd6740 1 -- 172.21.15.90:0/2252197322 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0dc804f250 con 0x563b9b255fc0 2024-04-25T01:17:05.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.692+0000 7f0dc5600640 1 -- 172.21.15.90:0/2252197322 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) ==== 24+0+0 (secure 0 0 0) 0x7f0dd0061ac0 con 0x563b9b255fc0 2024-04-25T01:17:05.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.692+0000 7f0de1dd6740 10 client.4857 fetch_fsmap learned FSMap version 9 2024-04-25T01:17:05.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.692+0000 7f0de1dd6740 1 -- 172.21.15.90:0/2252197322 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) -- 0x563b9b255220 con 0x563b9b255fc0 2024-04-25T01:17:05.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.692+0000 7f0dc5600640 1 -- 172.21.15.90:0/2252197322 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dc804f250 con 0x563b9b255fc0 2024-04-25T01:17:05.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.692+0000 7f0dc5600640 1 -- 172.21.15.90:0/2252197322 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 9) ==== 3502+0+0 (secure 0 0 0) 0x7f0dd0058330 con 0x563b9b255fc0 2024-04-25T01:17:05.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.692+0000 7f0de1dd6740 10 client.4857 fetch_fsmap finished waiting for FSMap version 9 2024-04-25T01:17:05.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.692+0000 7f0de1dd6740 10 client.4857 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-25T01:17:05.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.692+0000 7f0de1dd6740 20 client.4857 populate_metadata read hostname 'smithi090' 2024-04-25T01:17:05.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.692+0000 7f0de1dd6740 1 --2- 172.21.15.90:0/2252197322 >> [v2:172.21.15.103:6833/4291578198,v1:172.21.15.103:6835/4291578198] conn(0x563b9b276510 0x563b9b2768f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:05.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.692+0000 7f0de1dd6740 4 client.4857 mds_command: new command op to 4254 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:17:05.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.692+0000 7f0de1dd6740 1 -- 172.21.15.90:0/2252197322 --> [v2:172.21.15.103:6833/4291578198,v1:172.21.15.103:6835/4291578198] -- command(tid 0: {"prefix": "status"}) -- 0x563b9b178680 con 0x563b9b276510 2024-04-25T01:17:05.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.693+0000 7f0dd7e00640 1 --2- 172.21.15.90:0/2252197322 >> [v2:172.21.15.103:6833/4291578198,v1:172.21.15.103:6835/4291578198] conn(0x563b9b276510 0x563b9b2768f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:05.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.693+0000 7f0dd7e00640 1 --2- 172.21.15.90:0/2252197322 >> [v2:172.21.15.103:6833/4291578198,v1:172.21.15.103:6835/4291578198] conn(0x563b9b276510 0x563b9b2768f0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd8222127f10b423 server_cookie=13edf0008497f7bd in_seq=0 out_seq=0 2024-04-25T01:17:05.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.693+0000 7f0dc5600640 10 client.4857 ms_handle_connect on v2:172.21.15.103:6833/4291578198 2024-04-25T01:17:05.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.694+0000 7f0dc5600640 1 -- 172.21.15.90:0/2252197322 <== mds.0 v2:172.21.15.103:6833/4291578198 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x563b9b178680 con 0x563b9b276510 2024-04-25T01:17:05.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.694+0000 7f0dc5600640 10 client.4857 handle_command_reply: tid=0 2024-04-25T01:17:05.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.694+0000 7f0de1dd6740 1 client.4857 shutdown 2024-04-25T01:17:05.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.694+0000 7f0de1dd6740 1 -- 172.21.15.90:0/2252197322 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b9b255fc0 msgr2=0x563b9b20ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:05.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.694+0000 7f0de1dd6740 1 --2- 172.21.15.90:0/2252197322 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b9b255fc0 0x563b9b20ed20 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0058e70 tx=0x7f0dd0058ee0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.694+0000 7f0de1dd6740 1 -- 172.21.15.90:0/2252197322 shutdown_connections 2024-04-25T01:17:05.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.694+0000 7f0de1dd6740 1 --2- 172.21.15.90:0/2252197322 >> [v2:172.21.15.103:6833/4291578198,v1:172.21.15.103:6835/4291578198] conn(0x563b9b276510 0x563b9b2768f0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.694+0000 7f0de1dd6740 1 --2- 172.21.15.90:0/2252197322 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b9b255fc0 0x563b9b20ed20 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.694+0000 7f0de1dd6740 1 --2- 172.21.15.90:0/2252197322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b9b24a5e0 0x563b9b20e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.694+0000 7f0de1dd6740 1 --2- 172.21.15.90:0/2252197322 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563b9b238720 0x563b9b2cd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.694+0000 7f0dc5600640 0 client.4857 ms_handle_reset on v2:172.21.15.103:6833/4291578198 2024-04-25T01:17:05.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.694+0000 7f0de1dd6740 1 -- 172.21.15.90:0/2252197322 >> 172.21.15.90:0/2252197322 conn(0x563b9b20e400 msgr2=0x563b9b2527e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:05.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.694+0000 7f0de1dd6740 1 -- 172.21.15.90:0/2252197322 shutdown_connections 2024-04-25T01:17:05.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.694+0000 7f0de1dd6740 1 -- 172.21.15.90:0/2252197322 wait complete. 2024-04-25T01:17:05.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.695+0000 7f0de1dd6740 20 client.4857 trim_cache size 0 max 16384 2024-04-25T01:17:05.683 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T01:17:05.683 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21", 2024-04-25T01:17:05.683 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 0, 2024-04-25T01:17:05.683 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 4254, 2024-04-25T01:17:05.683 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-04-25T01:17:05.683 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-04-25T01:17:05.683 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-04-25T01:17:05.683 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 43.092316072000003, 2024-04-25T01:17:05.683 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 9, 2024-04-25T01:17:05.683 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 26, 2024-04-25T01:17:05.683 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 26, 2024-04-25T01:17:05.683 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 50.648717165000001 2024-04-25T01:17:05.684 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T01:17:05.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.695+0000 7f0ddd800640 1 -- 172.21.15.90:0/246273537 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0db406be60 msgr2=0x7f0db406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:05.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.695+0000 7f0ddd800640 1 --2- 172.21.15.90:0/246273537 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0db406be60 0x7f0db406e320 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0dc8034000 tx=0x7f0dc802e000 comp rx=0 tx=0).stop 2024-04-25T01:17:05.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.695+0000 7f0ddd800640 1 -- 172.21.15.90:0/246273537 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dd8161760 msgr2=0x7f0dd815a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:05.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.695+0000 7f0ddd800640 1 --2- 172.21.15.90:0/246273537 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dd8161760 0x7f0dd815a1a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0dd000c3e0 tx=0x7f0dd000c8a0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.695+0000 7f0ddd800640 1 -- 172.21.15.90:0/246273537 shutdown_connections 2024-04-25T01:17:05.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.695+0000 7f0ddd800640 1 --2- 172.21.15.90:0/246273537 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0db406be60 0x7f0db406e320 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.695+0000 7f0ddd800640 1 --2- 172.21.15.90:0/246273537 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dd8161760 0x7f0dd815a1a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.695+0000 7f0ddd800640 1 --2- 172.21.15.90:0/246273537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0dd80d0f60 0x7f0dd80ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.695+0000 7f0ddd800640 1 --2- 172.21.15.90:0/246273537 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0dd80b5340 0x7f0dd80c8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:05.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.695+0000 7f0ddd800640 1 -- 172.21.15.90:0/246273537 >> 172.21.15.90:0/246273537 conn(0x7f0dd809d170 msgr2=0x7f0dd80b57d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:05.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.695+0000 7f0ddd800640 1 -- 172.21.15.90:0/246273537 shutdown_connections 2024-04-25T01:17:05.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:05.695+0000 7f0ddd800640 1 -- 172.21.15.90:0/246273537 wait complete. 2024-04-25T01:17:05.696 INFO:teuthology.run_tasks:Running task sequential... 2024-04-25T01:17:05.705 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-25T01:17:05.705 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-25T01:17:05.706 INFO:teuthology.task.print:Enabling mgr modules 2024-04-25T01:17:05.706 INFO:teuthology.run_tasks:Running task kclient... 2024-04-25T01:17:05.734 INFO:tasks.kclient:Mounting kernel clients... 2024-04-25T01:17:05.734 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-04-25T01:17:05.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:05.734 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-04-25T01:17:05.784 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-25T01:17:05.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:05.785 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T01:17:05.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:05.866 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link delete ceph-brx 2024-04-25T01:17:05.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:05.967 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-04-25T01:17:06.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:06.008 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link delete ceph-brx 2024-04-25T01:17:06.123 INFO:teuthology.orchestra.run.smithi090.stderr:Cannot find device "ceph-brx" 2024-04-25T01:17:06.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:17:06.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:06.126 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-04-25T01:17:06.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:06.187 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link delete ceph-brx 2024-04-25T01:17:06.281 INFO:teuthology.orchestra.run.smithi103.stderr:Cannot find device "ceph-brx" 2024-04-25T01:17:06.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:17:06.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:06.286 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-04-25T01:17:06.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:06.345 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link delete ceph-brx 2024-04-25T01:17:06.433 INFO:teuthology.orchestra.run.smithi103.stderr:Cannot find device "ceph-brx" 2024-04-25T01:17:06.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:17:06.437 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-04-25T01:17:06.437 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T01:17:06.437 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T01:17:06.438 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T01:17:06.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.666+0000 7fcdef200640 1 Processor -- start 2024-04-25T01:17:06.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.666+0000 7fcdef200640 1 -- start start 2024-04-25T01:17:06.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.667+0000 7fcdef200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdf0161760 0x7fcdf00d2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:06.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.667+0000 7fcdef200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdf00cebb0 con 0x7fcdf00b5340 2024-04-25T01:17:06.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.667+0000 7fcdef200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdf00d46d0 con 0x7fcdf00d0f60 2024-04-25T01:17:06.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.667+0000 7fcdef200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcdf00b57a0 con 0x7fcdf0161760 2024-04-25T01:17:06.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.668+0000 7fcded400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fcdf00b5340 0x7fcdf00d0850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:49214/0 (socket says 172.21.15.90:49214) 2024-04-25T01:17:06.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.668+0000 7fcded400640 1 -- 172.21.15.90:0/2739189204 learned_addr learned my addr 172.21.15.90:0/2739189204 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:06.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.668+0000 7fcdede00640 1 --2- 172.21.15.90:0/2739189204 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdf0161760 0x7fcdf00d2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:06.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.669+0000 7fcdeca00640 1 -- 172.21.15.90:0/2739189204 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1158174693 0 0) 0x7fcdf00cebb0 con 0x7fcdf00b5340 2024-04-25T01:17:06.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.669+0000 7fcdeca00640 1 -- 172.21.15.90:0/2739189204 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcdd4003520 con 0x7fcdf00b5340 2024-04-25T01:17:06.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.669+0000 7fcdeca00640 1 -- 172.21.15.90:0/2739189204 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1921157004 0 0) 0x7fcdd4003520 con 0x7fcdf00b5340 2024-04-25T01:17:06.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.669+0000 7fcdeca00640 1 -- 172.21.15.90:0/2739189204 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdf0161760 msgr2=0x7fcdf00d2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:06.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.669+0000 7fcdeca00640 1 --2- 172.21.15.90:0/2739189204 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdf0161760 0x7fcdf00d2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:06.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.669+0000 7fcdeca00640 1 -- 172.21.15.90:0/2739189204 >> v1:172.21.15.103:6789/0 conn(0x7fcdf00d0f60 legacy=0x7fcdf00cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:06.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.669+0000 7fcdeca00640 1 -- 172.21.15.90:0/2739189204 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdf00d49b0 con 0x7fcdf00b5340 2024-04-25T01:17:06.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.669+0000 7fcdeca00640 1 -- 172.21.15.90:0/2739189204 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7fcddc001690 con 0x7fcdf00b5340 2024-04-25T01:17:06.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.669+0000 7fcdeca00640 1 -- 172.21.15.90:0/2739189204 >> v1:172.21.15.90:6789/0 conn(0x7fcdf00b5340 legacy=0x7fcdf00d0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:06.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.669+0000 7fcdeca00640 1 --2- 172.21.15.90:0/2739189204 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdd40039e0 0x7fcdd4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:06.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.669+0000 7fcdeca00640 1 --2- 172.21.15.90:0/2739189204 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdf0161760 0x7fcdd4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:06.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.669+0000 7fcdeca00640 1 --2- 172.21.15.90:0/2739189204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdd4004d40 0x7fcdd40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:06.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.669+0000 7fcdeca00640 1 -- 172.21.15.90:0/2739189204 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd400bac0 con 0x7fcdd4004d40 2024-04-25T01:17:06.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.669+0000 7fcdeca00640 1 -- 172.21.15.90:0/2739189204 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd400bc70 con 0x7fcdd40039e0 2024-04-25T01:17:06.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.669+0000 7fcdeca00640 1 -- 172.21.15.90:0/2739189204 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcdd400be70 con 0x7fcdf0161760 2024-04-25T01:17:06.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.670+0000 7fcdee800640 1 --2- 172.21.15.90:0/2739189204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdd4004d40 0x7fcdd40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:06.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.670+0000 7fcded400640 1 --2- 172.21.15.90:0/2739189204 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdf0161760 0x7fcdd4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:06.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.670+0000 7fcdee800640 1 -- 172.21.15.90:0/2739189204 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdf0161760 msgr2=0x7fcdd4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:06.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.670+0000 7fcdee800640 1 --2- 172.21.15.90:0/2739189204 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdf0161760 0x7fcdd4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:06.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.670+0000 7fcdee800640 1 -- 172.21.15.90:0/2739189204 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdd40039e0 msgr2=0x7fcdd4003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:06.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.670+0000 7fcdee800640 1 --2- 172.21.15.90:0/2739189204 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdd40039e0 0x7fcdd4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:06.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.670+0000 7fcdee800640 1 -- 172.21.15.90:0/2739189204 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdd4003520 con 0x7fcdd4004d40 2024-04-25T01:17:06.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.670+0000 7fcded400640 1 --2- 172.21.15.90:0/2739189204 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdf0161760 0x7fcdd4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:06.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.671+0000 7fcdee800640 1 --2- 172.21.15.90:0/2739189204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdd4004d40 0x7fcdd40051c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcdf00d1df0 tx=0x7fcdd4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d52294264df513a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:06.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.672+0000 7fcdeca00640 1 -- 172.21.15.90:0/2739189204 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde8001a70 con 0x7fcdd4004d40 2024-04-25T01:17:06.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.672+0000 7fcdeca00640 1 -- 172.21.15.90:0/2739189204 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcde802ae10 con 0x7fcdd4004d40 2024-04-25T01:17:06.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.672+0000 7fcdeca00640 1 -- 172.21.15.90:0/2739189204 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde8003bb0 con 0x7fcdd4004d40 2024-04-25T01:17:06.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.672+0000 7fcdef200640 1 -- 172.21.15.90:0/2739189204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdd4004d40 msgr2=0x7fcdd40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:06.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.672+0000 7fcdef200640 1 --2- 172.21.15.90:0/2739189204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdd4004d40 0x7fcdd40051c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcdf00d1df0 tx=0x7fcdd4005830 comp rx=0 tx=0).stop 2024-04-25T01:17:06.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.672+0000 7fcdee800640 1 -- 172.21.15.90:0/2739189204 reap_dead start 2024-04-25T01:17:06.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.672+0000 7fcdef200640 1 -- 172.21.15.90:0/2739189204 shutdown_connections 2024-04-25T01:17:06.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.672+0000 7fcdef200640 1 -- 172.21.15.90:0/2739189204 >> 172.21.15.90:0/2739189204 conn(0x7fcdf009d170 msgr2=0x7fcdf00d3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:06.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.673+0000 7fcdef200640 1 -- 172.21.15.90:0/2739189204 shutdown_connections 2024-04-25T01:17:06.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.673+0000 7fcdef200640 1 -- 172.21.15.90:0/2739189204 wait complete. 2024-04-25T01:17:06.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.675+0000 7fcdef200640 1 Processor -- start 2024-04-25T01:17:06.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.675+0000 7fcdef200640 1 -- start start 2024-04-25T01:17:06.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.676+0000 7fcdef200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdd4004d40 0x7fcdf0158590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:06.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.676+0000 7fcdef200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdf015f070 0x7fcdf01f3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:06.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.676+0000 7fcdef200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdf01f4240 0x7fcdf00b0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:06.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.676+0000 7fcdef200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf0161b40 con 0x7fcdf01f4240 2024-04-25T01:17:06.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.676+0000 7fcdef200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf00d4c90 con 0x7fcdf015f070 2024-04-25T01:17:06.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.676+0000 7fcdef200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcdf0158ad0 con 0x7fcdd4004d40 2024-04-25T01:17:06.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.676+0000 7fcdee800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdf01f4240 0x7fcdf00b0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:06.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.676+0000 7fcdee800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdf01f4240 0x7fcdf00b0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44728/0 (socket says 172.21.15.90:44728) 2024-04-25T01:17:06.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.676+0000 7fcdee800640 1 -- 172.21.15.90:0/3655405715 learned_addr learned my addr 172.21.15.90:0/3655405715 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:06.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.676+0000 7fcded400640 1 --2- 172.21.15.90:0/3655405715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdf015f070 0x7fcdf01f3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:06.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.676+0000 7fcdede00640 1 --2- 172.21.15.90:0/3655405715 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdd4004d40 0x7fcdf0158590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:06.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.676+0000 7fcdee800640 1 -- 172.21.15.90:0/3655405715 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdd4004d40 msgr2=0x7fcdf0158590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:06.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.676+0000 7fcdee800640 1 --2- 172.21.15.90:0/3655405715 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdd4004d40 0x7fcdf0158590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:06.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.676+0000 7fcdee800640 1 -- 172.21.15.90:0/3655405715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdf015f070 msgr2=0x7fcdf01f3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:06.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.676+0000 7fcdee800640 1 --2- 172.21.15.90:0/3655405715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdf015f070 0x7fcdf01f3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:06.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.676+0000 7fcdee800640 1 -- 172.21.15.90:0/3655405715 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdf01f4750 con 0x7fcdf01f4240 2024-04-25T01:17:06.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.677+0000 7fcdee800640 1 --2- 172.21.15.90:0/3655405715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdf01f4240 0x7fcdf00b0c80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcde802a8c0 tx=0x7fcde80014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:06.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.677+0000 7fcde6000640 1 -- 172.21.15.90:0/3655405715 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde8037040 con 0x7fcdf01f4240 2024-04-25T01:17:06.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.677+0000 7fcde6000640 1 -- 172.21.15.90:0/3655405715 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcde802ac10 con 0x7fcdf01f4240 2024-04-25T01:17:06.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.677+0000 7fcde6000640 1 -- 172.21.15.90:0/3655405715 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde8001a70 con 0x7fcdf01f4240 2024-04-25T01:17:06.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.677+0000 7fcdef200640 1 -- 172.21.15.90:0/3655405715 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdf01f4a30 con 0x7fcdf01f4240 2024-04-25T01:17:06.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.677+0000 7fcdef200640 1 -- 172.21.15.90:0/3655405715 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdf00cc840 con 0x7fcdf01f4240 2024-04-25T01:17:06.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.679+0000 7fcdef200640 1 -- 172.21.15.90:0/3655405715 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdf0075350 con 0x7fcdf01f4240 2024-04-25T01:17:06.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.680+0000 7fcde6000640 1 -- 172.21.15.90:0/3655405715 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fcde8042020 con 0x7fcdf01f4240 2024-04-25T01:17:06.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.680+0000 7fcde6000640 1 --2- 172.21.15.90:0/3655405715 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fcdd006be30 0x7fcdd006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:06.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.681+0000 7fcde6000640 1 -- 172.21.15.90:0/3655405715 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fcde80aa6f0 con 0x7fcdf01f4240 2024-04-25T01:17:06.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.681+0000 7fcdede00640 1 --2- 172.21.15.90:0/3655405715 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fcdd006be30 0x7fcdd006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:06.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.682+0000 7fcdede00640 1 --2- 172.21.15.90:0/3655405715 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fcdd006be30 0x7fcdd006e2f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcde0004770 tx=0x7fcde0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:06.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.685+0000 7fcde6000640 1 -- 172.21.15.90:0/3655405715 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fcde8078d40 con 0x7fcdf01f4240 2024-04-25T01:17:06.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.924+0000 7fcdef200640 1 -- 172.21.15.90:0/3655405715 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fcdf00678f0 con 0x7fcdf01f4240 2024-04-25T01:17:06.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.925+0000 7fcde6000640 1 -- 172.21.15.90:0/3655405715 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7fcde8078640 con 0x7fcdf01f4240 2024-04-25T01:17:06.909 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T01:17:06.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.928+0000 7fcdef200640 1 -- 172.21.15.90:0/3655405715 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fcdd006be30 msgr2=0x7fcdd006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:06.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.928+0000 7fcdef200640 1 --2- 172.21.15.90:0/3655405715 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fcdd006be30 0x7fcdd006e2f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcde0004770 tx=0x7fcde0008040 comp rx=0 tx=0).stop 2024-04-25T01:17:06.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.928+0000 7fcdef200640 1 -- 172.21.15.90:0/3655405715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdf01f4240 msgr2=0x7fcdf00b0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:06.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.928+0000 7fcdef200640 1 --2- 172.21.15.90:0/3655405715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdf01f4240 0x7fcdf00b0c80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcde802a8c0 tx=0x7fcde80014a0 comp rx=0 tx=0).stop 2024-04-25T01:17:06.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.928+0000 7fcdef200640 1 -- 172.21.15.90:0/3655405715 shutdown_connections 2024-04-25T01:17:06.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.928+0000 7fcdef200640 1 --2- 172.21.15.90:0/3655405715 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fcdd006be30 0x7fcdd006e2f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:06.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.928+0000 7fcdef200640 1 --2- 172.21.15.90:0/3655405715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdf01f4240 0x7fcdf00b0c80 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:06.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.928+0000 7fcdef200640 1 --2- 172.21.15.90:0/3655405715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdf015f070 0x7fcdf01f3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:06.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.928+0000 7fcdef200640 1 --2- 172.21.15.90:0/3655405715 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcdd4004d40 0x7fcdf0158590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:06.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.928+0000 7fcdef200640 1 -- 172.21.15.90:0/3655405715 >> 172.21.15.90:0/3655405715 conn(0x7fcdf009d170 msgr2=0x7fcdf006bab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:06.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.928+0000 7fcdef200640 1 -- 172.21.15.90:0/3655405715 shutdown_connections 2024-04-25T01:17:06.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:06.928+0000 7fcdef200640 1 -- 172.21.15.90:0/3655405715 wait complete. 2024-04-25T01:17:06.931 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T01:17:06.932 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T01:17:06.932 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2024-04-25T01:17:06.932 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T01:17:06.932 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T01:17:06.932 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T01:17:06.932 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T01:17:06.932 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T01:17:06.932 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T01:17:06.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:06.932 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2024-04-25T01:17:06.985 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T01:17:06.985 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T01:17:06.985 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T01:17:06.985 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:06.985 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-04-25T01:17:06.986 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:06.986 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:17:06.986 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:17:06.986 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f0 2024-04-25T01:17:06.986 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:17:06.986 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:17:06.986 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f1 2024-04-25T01:17:06.986 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:17:06.986 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:17:06.986 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T01:17:06.986 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:17:06.986 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-25T01:17:06.986 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 42612sec preferred_lft 42612sec 2024-04-25T01:17:06.986 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::ec4:7aff:fe8f:d767/64 scope link 2024-04-25T01:17:06.987 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:06.988 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T01:17:06.988 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:17:06.988 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-04-25T01:17:06.988 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add name ceph-brx type bridge 2024-04-25T01:17:06.988 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip addr flush dev ceph-brx 2024-04-25T01:17:06.988 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set ceph-brx up 2024-04-25T01:17:06.988 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T01:17:06.988 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-04-25T01:17:07.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:17:07.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:07.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:07.232 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T01:17:07.293 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-04-25T01:17:07.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:07.296 DEBUG:teuthology.orchestra.run.smithi090:> route 2024-04-25T01:17:07.341 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2024-04-25T01:17:07.342 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T01:17:07.342 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T01:17:07.342 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T01:17:07.342 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T01:17:07.342 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:17:07.342 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-04-25T01:17:07.342 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T01:17:07.342 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T01:17:07.342 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T01:17:07.342 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-04-25T01:17:07.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:17:07.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:07.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:07.591 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-04-25T01:17:07.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:07.635 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list-id 2024-04-25T01:17:07.733 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:17:07.733 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-04-25T01:17:07.733 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T01:17:07.733 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-25T01:17:07.733 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-04-25T01:17:07.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:17:07.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:07.922 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-25T01:17:07.922 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:17:07.922 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-04-25T01:17:07.922 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-25T01:17:07.922 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T01:17:07.923 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-25T01:17:07.923 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-25T01:17:07.923 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-25T01:17:07.923 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-04-25T01:17:08.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:17:08.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:08.188 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:17:08.189 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-04-25T01:17:08.189 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.0 up 2024-04-25T01:17:08.189 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T01:17:08.189 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-04-25T01:17:08.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:17:08.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:08.395 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:17:08.395 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T01:17:08.437 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T01:17:08.437 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:17:08.437 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T01:17:08.532 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-25T01:17:08.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:08.532 DEBUG:teuthology.orchestra.run.smithi090:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-25T01:17:08.658 INFO:teuthology.orchestra.run.smithi090.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-25T01:17:08.658 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-25T01:17:08.658 INFO:teuthology.orchestra.run.smithi090.stdout:invalid new device string format 2024-04-25T01:17:09.427 INFO:teuthology.orchestra.run.smithi090.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-25T01:17:09.427 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-25T01:17:09.427 INFO:teuthology.orchestra.run.smithi090.stdout:invalid new device string format 2024-04-25T01:17:09.428 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: resolved to: "172.21.15.90:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-25T01:17:09.428 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-25T01:17:09.460 INFO:teuthology.orchestra.run.smithi090.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-25T01:17:09.460 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-25T01:17:09.460 INFO:teuthology.orchestra.run.smithi090.stdout:invalid new device string format 2024-04-25T01:17:09.461 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: resolved to: "172.21.15.90:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-25T01:17:09.461 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-25T01:17:09.461 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: resolved to: "172.21.15.90:6789,172.21.15.103:6789,172.21.15.103:6790" 2024-04-25T01:17:09.461 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.90:6789,172.21.15.103:6789,172.21.15.103:6790:/ 2024-04-25T01:17:09.461 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21" will pass to kernel 2024-04-25T01:17:09.463 INFO:teuthology.orchestra.run.smithi090.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-25T01:17:09.463 INFO:teuthology.orchestra.run.smithi090.stdout: You just mounted a file system that supports labels which does not 2024-04-25T01:17:09.463 INFO:teuthology.orchestra.run.smithi090.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T01:17:09.463 INFO:teuthology.orchestra.run.smithi090.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T01:17:09.463 INFO:teuthology.orchestra.run.smithi090.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T01:17:09.467 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T01:17:09.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:09.467 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T01:17:09.538 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:17:09.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:09 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:17:09.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:09.658 INFO:teuthology.orchestra.run.smithi090.stdout:client4866 2024-04-25T01:17:09.659 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:17:09.659 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:17:09.711 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:17:09.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:09 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:17:09.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:09.869 INFO:teuthology.orchestra.run.smithi090.stdout:client4866 2024-04-25T01:17:09.870 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:17:09.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:09 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:17:09.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:09.989 INFO:teuthology.orchestra.run.smithi090.stdout:client4866 2024-04-25T01:17:09.990 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21.client4866/status') 2024-04-25T01:17:10.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:17:10.122 INFO:teuthology.orchestra.run.smithi090.stdout:instance: client.4866 (3)192.168.0.1:0/2278985955 2024-04-25T01:17:10.122 INFO:teuthology.orchestra.run.smithi090.stdout:blocklisted: false 2024-04-25T01:17:10.122 INFO:teuthology.orchestra.run.smithi090.stderr:0+1 records in 2024-04-25T01:17:10.122 INFO:teuthology.orchestra.run.smithi090.stderr:0+1 records out 2024-04-25T01:17:10.122 INFO:teuthology.orchestra.run.smithi090.stderr:69 bytes copied, 8.0627e-05 s, 856 kB/s 2024-04-25T01:17:10.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:10.135 INFO:tasks.kclient:client.2 config is {'syntax': 'v1'} 2024-04-25T01:17:10.135 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T01:17:10.135 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-25T01:17:10.135 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T01:17:10.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.346+0000 7f86fea00640 1 Processor -- start 2024-04-25T01:17:10.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.347+0000 7f86fea00640 1 -- start start 2024-04-25T01:17:10.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.348+0000 7f86fea00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f86f806b120 0x7f86f806b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:10.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.348+0000 7f86fea00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86f815ca10 0x7f86f815cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:10.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.348+0000 7f86fea00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86f806ba40 con 0x7f86f80ae730 2024-04-25T01:17:10.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.348+0000 7f86fea00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f86f80cef40 con 0x7f86f806b120 2024-04-25T01:17:10.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.348+0000 7f86fea00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f86f815c580 con 0x7f86f815ca10 2024-04-25T01:17:10.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.348+0000 7f86fcc00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f86f80ae730 0x7f86f80cd900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:49218/0 (socket says 172.21.15.90:49218) 2024-04-25T01:17:10.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.349+0000 7f86fcc00640 1 -- 172.21.15.90:0/2164895576 learned_addr learned my addr 172.21.15.90:0/2164895576 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:10.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.349+0000 7f86fe000640 1 --2- 172.21.15.90:0/2164895576 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86f815ca10 0x7f86f815cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:10.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.349+0000 7f86fd600640 1 --2- 172.21.15.90:0/2164895576 >> v2:172.21.15.103:3300/0 conn(0x7f86f806b120 0x7f86f806b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:10.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.349+0000 7f86f7e00640 1 -- 172.21.15.90:0/2164895576 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 242661623 0 0) 0x7f86f806ba40 con 0x7f86f80ae730 2024-04-25T01:17:10.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.349+0000 7f86f7e00640 1 -- 172.21.15.90:0/2164895576 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86e0003520 con 0x7f86f80ae730 2024-04-25T01:17:10.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.349+0000 7f86f7e00640 1 -- 172.21.15.90:0/2164895576 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3455214171 0 0) 0x7f86e0003520 con 0x7f86f80ae730 2024-04-25T01:17:10.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.350+0000 7f86f7e00640 1 -- 172.21.15.90:0/2164895576 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86f815ca10 msgr2=0x7f86f815cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:10.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.350+0000 7f86f7e00640 1 --2- 172.21.15.90:0/2164895576 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86f815ca10 0x7f86f815cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:10.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.350+0000 7f86f7e00640 1 -- 172.21.15.90:0/2164895576 >> v2:172.21.15.103:3300/0 conn(0x7f86f806b120 msgr2=0x7f86f806b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:10.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.350+0000 7f86f7e00640 1 --2- 172.21.15.90:0/2164895576 >> v2:172.21.15.103:3300/0 conn(0x7f86f806b120 0x7f86f806b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:10.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.350+0000 7f86f7e00640 1 -- 172.21.15.90:0/2164895576 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86f815c220 con 0x7f86f80ae730 2024-04-25T01:17:10.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.350+0000 7f86f7e00640 1 -- 172.21.15.90:0/2164895576 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f86e4001670 con 0x7f86f80ae730 2024-04-25T01:17:10.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.350+0000 7f86f7e00640 1 -- 172.21.15.90:0/2164895576 >> v1:172.21.15.90:6789/0 conn(0x7f86f80ae730 legacy=0x7f86f80cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:10.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.350+0000 7f86f7e00640 1 --2- 172.21.15.90:0/2164895576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86e00039b0 0x7f86e0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:10.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.350+0000 7f86f7e00640 1 --2- 172.21.15.90:0/2164895576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86e0004800 0x7f86e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:10.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.350+0000 7f86f7e00640 1 --2- 172.21.15.90:0/2164895576 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86f815ca10 0x7f86e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:10.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.350+0000 7f86f7e00640 1 -- 172.21.15.90:0/2164895576 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f86f815c580 con 0x7f86e00039b0 2024-04-25T01:17:10.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.350+0000 7f86f7e00640 1 -- 172.21.15.90:0/2164895576 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f86e000b9c0 con 0x7f86e0004800 2024-04-25T01:17:10.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.350+0000 7f86f7e00640 1 -- 172.21.15.90:0/2164895576 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f86e000bb90 con 0x7f86f815ca10 2024-04-25T01:17:10.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.350+0000 7f86fd600640 1 --2- 172.21.15.90:0/2164895576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86e00039b0 0x7f86e0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:10.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.350+0000 7f86fe000640 1 --2- 172.21.15.90:0/2164895576 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86f815ca10 0x7f86e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:10.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.350+0000 7f86fcc00640 1 --2- 172.21.15.90:0/2164895576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86e0004800 0x7f86e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:10.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.351+0000 7f86fd600640 1 -- 172.21.15.90:0/2164895576 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86f815ca10 msgr2=0x7f86e00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:10.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.351+0000 7f86fd600640 1 --2- 172.21.15.90:0/2164895576 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86f815ca10 0x7f86e00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:10.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.351+0000 7f86fd600640 1 -- 172.21.15.90:0/2164895576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86e0004800 msgr2=0x7f86e0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:10.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.351+0000 7f86fd600640 1 --2- 172.21.15.90:0/2164895576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86e0004800 0x7f86e0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:10.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.351+0000 7f86fd600640 1 -- 172.21.15.90:0/2164895576 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86e0003520 con 0x7f86e00039b0 2024-04-25T01:17:10.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.352+0000 7f86fd600640 1 --2- 172.21.15.90:0/2164895576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86e00039b0 0x7f86e0003dd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f86ec00ba30 tx=0x7f86e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbb0e11ec457c445 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:10.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.353+0000 7f86f7e00640 1 -- 172.21.15.90:0/2164895576 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ec030070 con 0x7f86e00039b0 2024-04-25T01:17:10.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.353+0000 7f86f7e00640 1 -- 172.21.15.90:0/2164895576 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86ec02ae10 con 0x7f86e00039b0 2024-04-25T01:17:10.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.353+0000 7f86f7e00640 1 -- 172.21.15.90:0/2164895576 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ec00b600 con 0x7f86e00039b0 2024-04-25T01:17:10.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.353+0000 7f86fea00640 1 -- 172.21.15.90:0/2164895576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86e00039b0 msgr2=0x7f86e0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:10.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.353+0000 7f86fea00640 1 --2- 172.21.15.90:0/2164895576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86e00039b0 0x7f86e0003dd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f86ec00ba30 tx=0x7f86e0005830 comp rx=0 tx=0).stop 2024-04-25T01:17:10.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.353+0000 7f86fe000640 1 -- 172.21.15.90:0/2164895576 reap_dead start 2024-04-25T01:17:10.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.354+0000 7f86fea00640 1 -- 172.21.15.90:0/2164895576 shutdown_connections 2024-04-25T01:17:10.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.354+0000 7f86fea00640 1 -- 172.21.15.90:0/2164895576 >> 172.21.15.90:0/2164895576 conn(0x7f86f80b1660 msgr2=0x7f86f80b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:10.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.354+0000 7f86fea00640 1 -- 172.21.15.90:0/2164895576 shutdown_connections 2024-04-25T01:17:10.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.354+0000 7f86fea00640 1 -- 172.21.15.90:0/2164895576 wait complete. 2024-04-25T01:17:10.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.356+0000 7f86fea00640 1 Processor -- start 2024-04-25T01:17:10.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.356+0000 7f86fea00640 1 -- start start 2024-04-25T01:17:10.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.357+0000 7f86fea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86e00039b0 0x7f86f8141540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:10.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.357+0000 7f86fea00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86f8141e90 0x7f86f8142310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:10.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.357+0000 7f86fea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86f8153f30 0x7f86f81543b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:10.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.357+0000 7f86fea00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f86f80af240 con 0x7f86e00039b0 2024-04-25T01:17:10.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.357+0000 7f86fea00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f86f80aef60 con 0x7f86f8153f30 2024-04-25T01:17:10.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.357+0000 7f86fea00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f86f8153080 con 0x7f86f8141e90 2024-04-25T01:17:10.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.357+0000 7f86fd600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86e00039b0 0x7f86f8141540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:10.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.357+0000 7f86fd600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86e00039b0 0x7f86f8141540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44742/0 (socket says 172.21.15.90:44742) 2024-04-25T01:17:10.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.357+0000 7f86fd600640 1 -- 172.21.15.90:0/1531252754 learned_addr learned my addr 172.21.15.90:0/1531252754 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:10.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.358+0000 7f86fcc00640 1 --2- 172.21.15.90:0/1531252754 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86f8141e90 0x7f86f8142310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:10.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.358+0000 7f86fe000640 1 --2- 172.21.15.90:0/1531252754 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86f8153f30 0x7f86f81543b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:10.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.358+0000 7f86fd600640 1 -- 172.21.15.90:0/1531252754 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86f8141e90 msgr2=0x7f86f8142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:10.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.358+0000 7f86fd600640 1 --2- 172.21.15.90:0/1531252754 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86f8141e90 0x7f86f8142310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:10.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.358+0000 7f86fd600640 1 -- 172.21.15.90:0/1531252754 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86f8153f30 msgr2=0x7f86f81543b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:10.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.358+0000 7f86fd600640 1 --2- 172.21.15.90:0/1531252754 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86f8153f30 0x7f86f81543b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:10.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.358+0000 7f86fd600640 1 -- 172.21.15.90:0/1531252754 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86f81594a0 con 0x7f86e00039b0 2024-04-25T01:17:10.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.358+0000 7f86fd600640 1 --2- 172.21.15.90:0/1531252754 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86e00039b0 0x7f86f8141540 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f86ec00bb50 tx=0x7f86ec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:10.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.359+0000 7f86f5600640 1 -- 172.21.15.90:0/1531252754 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ec042030 con 0x7f86e00039b0 2024-04-25T01:17:10.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.359+0000 7f86f5600640 1 -- 172.21.15.90:0/1531252754 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86ec02ac10 con 0x7f86e00039b0 2024-04-25T01:17:10.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.359+0000 7f86f5600640 1 -- 172.21.15.90:0/1531252754 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ec00b600 con 0x7f86e00039b0 2024-04-25T01:17:10.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.359+0000 7f86fea00640 1 -- 172.21.15.90:0/1531252754 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86f8153450 con 0x7f86e00039b0 2024-04-25T01:17:10.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.359+0000 7f86fea00640 1 -- 172.21.15.90:0/1531252754 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86f8153a10 con 0x7f86e00039b0 2024-04-25T01:17:10.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.360+0000 7f86fea00640 1 -- 172.21.15.90:0/1531252754 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86f8074d20 con 0x7f86e00039b0 2024-04-25T01:17:10.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.361+0000 7f86f5600640 1 -- 172.21.15.90:0/1531252754 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f86ec045070 con 0x7f86e00039b0 2024-04-25T01:17:10.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.362+0000 7f86f5600640 1 --2- 172.21.15.90:0/1531252754 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f86d806be30 0x7f86d806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:10.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.362+0000 7f86f5600640 1 -- 172.21.15.90:0/1531252754 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f86ec0ab870 con 0x7f86e00039b0 2024-04-25T01:17:10.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.362+0000 7f86fcc00640 1 --2- 172.21.15.90:0/1531252754 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f86d806be30 0x7f86d806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:10.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.363+0000 7f86fcc00640 1 --2- 172.21.15.90:0/1531252754 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f86d806be30 0x7f86d806e2f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f86f8158850 tx=0x7f86e4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:10.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.367+0000 7f86f5600640 1 -- 172.21.15.90:0/1531252754 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f86f8074d20 con 0x7f86e00039b0 2024-04-25T01:17:10.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.602+0000 7f86fea00640 1 -- 172.21.15.90:0/1531252754 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f86f8075350 con 0x7f86e00039b0 2024-04-25T01:17:10.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.603+0000 7f86f5600640 1 -- 172.21.15.90:0/1531252754 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f86ec079f30 con 0x7f86e00039b0 2024-04-25T01:17:10.587 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T01:17:10.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.605+0000 7f86fea00640 1 -- 172.21.15.90:0/1531252754 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f86d806be30 msgr2=0x7f86d806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:10.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.605+0000 7f86fea00640 1 --2- 172.21.15.90:0/1531252754 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f86d806be30 0x7f86d806e2f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f86f8158850 tx=0x7f86e4009040 comp rx=0 tx=0).stop 2024-04-25T01:17:10.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.606+0000 7f86fea00640 1 -- 172.21.15.90:0/1531252754 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86e00039b0 msgr2=0x7f86f8141540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:10.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.606+0000 7f86fea00640 1 --2- 172.21.15.90:0/1531252754 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86e00039b0 0x7f86f8141540 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f86ec00bb50 tx=0x7f86ec004730 comp rx=0 tx=0).stop 2024-04-25T01:17:10.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.606+0000 7f86fea00640 1 -- 172.21.15.90:0/1531252754 shutdown_connections 2024-04-25T01:17:10.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.606+0000 7f86fea00640 1 --2- 172.21.15.90:0/1531252754 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f86d806be30 0x7f86d806e2f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:10.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.606+0000 7f86fea00640 1 --2- 172.21.15.90:0/1531252754 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86f8153f30 0x7f86f81543b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:10.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.606+0000 7f86fea00640 1 --2- 172.21.15.90:0/1531252754 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86f8141e90 0x7f86f8142310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:10.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.606+0000 7f86fea00640 1 --2- 172.21.15.90:0/1531252754 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86e00039b0 0x7f86f8141540 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:10.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.606+0000 7f86fea00640 1 -- 172.21.15.90:0/1531252754 >> 172.21.15.90:0/1531252754 conn(0x7f86f80b1660 msgr2=0x7f86f80cbcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:10.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.606+0000 7f86fea00640 1 -- 172.21.15.90:0/1531252754 shutdown_connections 2024-04-25T01:17:10.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10.606+0000 7f86fea00640 1 -- 172.21.15.90:0/1531252754 wait complete. 2024-04-25T01:17:10.609 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T01:17:10.609 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T01:17:10.610 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2024-04-25T01:17:10.610 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-04-25T01:17:10.610 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-25T01:17:10.610 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T01:17:10.610 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T01:17:10.610 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T01:17:10.610 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-25T01:17:10.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:10.610 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2024-04-25T01:17:10.661 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T01:17:10.661 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T01:17:10.661 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T01:17:10.661 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:10.661 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-04-25T01:17:10.661 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:10.661 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:17:10.661 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:17:10.661 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f0 2024-04-25T01:17:10.661 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:17:10.661 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:17:10.661 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f1 2024-04-25T01:17:10.662 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:17:10.662 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:17:10.662 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T01:17:10.662 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:17:10.662 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-25T01:17:10.662 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 42609sec preferred_lft 42609sec 2024-04-25T01:17:10.662 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::ec4:7aff:fe8f:d767/64 scope link 2024-04-25T01:17:10.662 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:10.662 INFO:teuthology.orchestra.run.smithi090.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T01:17:10.662 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 1e:6a:0d:c3:bc:1e brd ff:ff:ff:ff:ff:ff 2024-04-25T01:17:10.662 INFO:teuthology.orchestra.run.smithi090.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T01:17:10.662 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:10.662 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::405b:4dff:feb7:7d54/64 scope link 2024-04-25T01:17:10.663 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:10.663 INFO:teuthology.orchestra.run.smithi090.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T01:17:10.663 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 1e:6a:0d:c3:bc:1e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T01:17:10.663 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::1c6a:dff:fec3:bc1e/64 scope link 2024-04-25T01:17:10.663 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:10.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:10.663 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T01:17:10.737 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-04-25T01:17:10.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:10.741 DEBUG:teuthology.orchestra.run.smithi090:> route 2024-04-25T01:17:10.791 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2024-04-25T01:17:10.791 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T01:17:10.791 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T01:17:10.791 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T01:17:10.791 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T01:17:10.792 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:17:10.792 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-04-25T01:17:10.792 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T01:17:10.792 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T01:17:10.792 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T01:17:10.792 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-04-25T01:17:10.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:17:10.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:10.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:10.998 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-04-25T01:17:11.047 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T01:17:11.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:11.049 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list-id 2024-04-25T01:17:11.094 INFO:teuthology.orchestra.run.smithi090.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-04-25T01:17:11.095 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:17:11.096 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-04-25T01:17:11.096 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T01:17:11.096 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-04-25T01:17:11.096 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-04-25T01:17:11.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:17:11.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:11.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:11.273 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-25T01:17:11.351 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T01:17:11.351 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T01:17:11.351 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T01:17:11.351 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:11.351 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-04-25T01:17:11.351 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:11.351 INFO:teuthology.orchestra.run.smithi090.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T01:17:11.351 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 96:73:9e:26:ca:ad brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-25T01:17:11.351 INFO:teuthology.orchestra.run.smithi090.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-25T01:17:11.352 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:11.352 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::9473:9eff:fe26:caad/64 scope link 2024-04-25T01:17:11.352 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:11.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:11.355 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-25T01:17:11.438 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T01:17:11.438 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T01:17:11.438 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T01:17:11.438 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:11.438 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-04-25T01:17:11.438 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:11.438 INFO:teuthology.orchestra.run.smithi090.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T01:17:11.438 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 96:73:9e:26:ca:ad brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-25T01:17:11.438 INFO:teuthology.orchestra.run.smithi090.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-25T01:17:11.438 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:11.438 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::9473:9eff:fe26:caad/64 scope link 2024-04-25T01:17:11.439 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:11.442 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-04-25T01:17:11.442 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:17:11.442 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-04-25T01:17:11.442 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-04-25T01:17:11.442 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-25T01:17:11.442 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-04-25T01:17:11.442 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-04-25T01:17:11.442 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-04-25T01:17:11.442 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-04-25T01:17:11.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:17:11.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:11.721 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:17:11.721 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-04-25T01:17:11.721 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.1 up 2024-04-25T01:17:11.721 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set dev brx.1 master ceph-brx 2024-04-25T01:17:11.721 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-04-25T01:17:11.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:17:11.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:11.931 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:17:11.931 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-04-25T01:17:11.982 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-04-25T01:17:11.982 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:17:11.982 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-04-25T01:17:12.031 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-25T01:17:12.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:12.031 DEBUG:teuthology.orchestra.run.smithi090:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-04-25T01:17:12.196 INFO:teuthology.orchestra.run.smithi090.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-04-25T01:17:12.196 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-04-25T01:17:12.196 INFO:teuthology.orchestra.run.smithi090.stdout:invalid new device string format 2024-04-25T01:17:12.216 INFO:teuthology.orchestra.run.smithi090.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-04-25T01:17:12.216 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-04-25T01:17:12.217 INFO:teuthology.orchestra.run.smithi090.stdout:invalid new device string format 2024-04-25T01:17:12.217 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: resolved to: "172.21.15.90:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-25T01:17:12.217 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-25T01:17:12.244 INFO:teuthology.orchestra.run.smithi090.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-04-25T01:17:12.244 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-04-25T01:17:12.244 INFO:teuthology.orchestra.run.smithi090.stdout:invalid new device string format 2024-04-25T01:17:12.245 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: resolved to: "172.21.15.90:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-25T01:17:12.245 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-25T01:17:12.245 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: resolved to: "172.21.15.90:6789,172.21.15.103:6789,172.21.15.103:6790" 2024-04-25T01:17:12.245 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.90:6789,172.21.15.103:6789,172.21.15.103:6790:/ 2024-04-25T01:17:12.245 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs,key=2,fsid=e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21" will pass to kernel 2024-04-25T01:17:12.246 INFO:teuthology.orchestra.run.smithi090.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2024-04-25T01:17:12.246 INFO:teuthology.orchestra.run.smithi090.stdout: You just mounted a file system that supports labels which does not 2024-04-25T01:17:12.246 INFO:teuthology.orchestra.run.smithi090.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T01:17:12.247 INFO:teuthology.orchestra.run.smithi090.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T01:17:12.247 INFO:teuthology.orchestra.run.smithi090.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T01:17:12.252 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T01:17:12.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:12.252 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-04-25T01:17:12.331 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:17:12.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:12 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:17:12.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:12.451 INFO:teuthology.orchestra.run.smithi090.stdout:client4630 2024-04-25T01:17:12.452 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:17:12.452 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:17:12.500 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:17:12.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:12 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:17:12.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:12.666 INFO:teuthology.orchestra.run.smithi090.stdout:client4630 2024-04-25T01:17:12.667 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:17:12.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:12 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:17:12.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:12.787 INFO:teuthology.orchestra.run.smithi090.stdout:client4630 2024-04-25T01:17:12.787 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21.client4630/status') 2024-04-25T01:17:12.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:17:12.917 INFO:teuthology.orchestra.run.smithi090.stdout:instance: client.4630 (3)172.21.15.90:0/1091799751 2024-04-25T01:17:12.917 INFO:teuthology.orchestra.run.smithi090.stdout:blocklisted: false 2024-04-25T01:17:12.918 INFO:teuthology.orchestra.run.smithi090.stderr:0+1 records in 2024-04-25T01:17:12.918 INFO:teuthology.orchestra.run.smithi090.stderr:0+1 records out 2024-04-25T01:17:12.918 INFO:teuthology.orchestra.run.smithi090.stderr:70 bytes copied, 7.751e-05 s, 903 kB/s 2024-04-25T01:17:12.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:12.930 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-04-25T01:17:12.930 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T01:17:12.931 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-25T01:17:12.931 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T01:17:13.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.145+0000 7fbd2cc00640 1 Processor -- start 2024-04-25T01:17:13.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.145+0000 7fbd2cc00640 1 -- start start 2024-04-25T01:17:13.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.147+0000 7fbd2cc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd28143940 0x7fbd2806e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:13.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.147+0000 7fbd2cc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbd2815f400 0x7fbd2806eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:13.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.147+0000 7fbd2cc00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd28070e60 con 0x7fbd28143d20 2024-04-25T01:17:13.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.147+0000 7fbd2cc00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fbd28071140 con 0x7fbd2815f400 2024-04-25T01:17:13.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.147+0000 7fbd2cc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbd28071420 con 0x7fbd28143940 2024-04-25T01:17:13.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.147+0000 7fbd26800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbd2815f400 0x7fbd2806eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:13.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.147+0000 7fbd26800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbd2815f400 0x7fbd2806eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44750/0 (socket says 172.21.15.90:44750) 2024-04-25T01:17:13.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.147+0000 7fbd26800640 1 -- 172.21.15.90:0/592114900 learned_addr learned my addr 172.21.15.90:0/592114900 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:13.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.147+0000 7fbd25e00640 1 --2- 172.21.15.90:0/592114900 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd28143940 0x7fbd2806e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:13.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.147+0000 7fbd26800640 1 -- 172.21.15.90:0/592114900 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd28143940 msgr2=0x7fbd2806e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:13.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.148+0000 7fbd26800640 1 --2- 172.21.15.90:0/592114900 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd28143940 0x7fbd2806e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:13.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.148+0000 7fbd26800640 1 -- 172.21.15.90:0/592114900 >> v1:172.21.15.103:6789/0 conn(0x7fbd28143d20 legacy=0x7fbd2815ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:13.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.148+0000 7fbd26800640 1 -- 172.21.15.90:0/592114900 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd28071700 con 0x7fbd2815f400 2024-04-25T01:17:13.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.148+0000 7fbd25e00640 1 --2- 172.21.15.90:0/592114900 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd28143940 0x7fbd2806e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:13.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.149+0000 7fbd26800640 1 --2- 172.21.15.90:0/592114900 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd2815f400 0x7fbd2806eb90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fbd20009e50 tx=0x7fbd2002a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39b76844095fae28 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:13.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.149+0000 7fbd25400640 1 -- 172.21.15.90:0/592114900 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd20036070 con 0x7fbd2815f400 2024-04-25T01:17:13.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.149+0000 7fbd25400640 1 -- 172.21.15.90:0/592114900 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd20031600 con 0x7fbd2815f400 2024-04-25T01:17:13.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.149+0000 7fbd25400640 1 -- 172.21.15.90:0/592114900 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd200319d0 con 0x7fbd2815f400 2024-04-25T01:17:13.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.149+0000 7fbd2cc00640 1 -- 172.21.15.90:0/592114900 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd2815f400 msgr2=0x7fbd2806eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:13.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.149+0000 7fbd2cc00640 1 --2- 172.21.15.90:0/592114900 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd2815f400 0x7fbd2806eb90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fbd20009e50 tx=0x7fbd2002a6d0 comp rx=0 tx=0).stop 2024-04-25T01:17:13.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.149+0000 7fbd2cc00640 1 -- 172.21.15.90:0/592114900 shutdown_connections 2024-04-25T01:17:13.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.149+0000 7fbd2cc00640 1 --2- 172.21.15.90:0/592114900 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd2815f400 0x7fbd2806eb90 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:13.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.149+0000 7fbd2cc00640 1 --2- 172.21.15.90:0/592114900 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd28143940 0x7fbd2806e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:13.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.149+0000 7fbd2cc00640 1 -- 172.21.15.90:0/592114900 >> 172.21.15.90:0/592114900 conn(0x7fbd280b1820 msgr2=0x7fbd280b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:13.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.150+0000 7fbd2cc00640 1 -- 172.21.15.90:0/592114900 shutdown_connections 2024-04-25T01:17:13.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.150+0000 7fbd2cc00640 1 -- 172.21.15.90:0/592114900 wait complete. 2024-04-25T01:17:13.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.151+0000 7fbd2cc00640 1 Processor -- start 2024-04-25T01:17:13.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.151+0000 7fbd2cc00640 1 -- start start 2024-04-25T01:17:13.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.152+0000 7fbd2cc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd28143940 0x7fbd280b1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:13.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.152+0000 7fbd2cc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd28143d20 0x7fbd281f30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:13.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.152+0000 7fbd2cc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd2815f400 0x7fbd280ce040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:13.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.152+0000 7fbd2cc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbd28071fb0 con 0x7fbd28143940 2024-04-25T01:17:13.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.152+0000 7fbd2cc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbd28071e30 con 0x7fbd2815f400 2024-04-25T01:17:13.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.152+0000 7fbd2cc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbd28072130 con 0x7fbd28143d20 2024-04-25T01:17:13.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.153+0000 7fbd25e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd28143940 0x7fbd280b1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:13.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.153+0000 7fbd25e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd28143940 0x7fbd280b1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44762/0 (socket says 172.21.15.90:44762) 2024-04-25T01:17:13.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.153+0000 7fbd25e00640 1 -- 172.21.15.90:0/3176818383 learned_addr learned my addr 172.21.15.90:0/3176818383 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:13.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.153+0000 7fbd25e00640 1 -- 172.21.15.90:0/3176818383 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd28143d20 msgr2=0x7fbd281f30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:13.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.153+0000 7fbd25e00640 1 --2- 172.21.15.90:0/3176818383 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd28143d20 0x7fbd281f30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:13.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.153+0000 7fbd26800640 1 --2- 172.21.15.90:0/3176818383 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd2815f400 0x7fbd280ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:13.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.153+0000 7fbd25e00640 1 -- 172.21.15.90:0/3176818383 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd2815f400 msgr2=0x7fbd280ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:13.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.153+0000 7fbd25e00640 1 --2- 172.21.15.90:0/3176818383 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd2815f400 0x7fbd280ce040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:13.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.153+0000 7fbd25e00640 1 -- 172.21.15.90:0/3176818383 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd280afb90 con 0x7fbd28143940 2024-04-25T01:17:13.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.153+0000 7fbd25e00640 1 --2- 172.21.15.90:0/3176818383 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd28143940 0x7fbd280b1020 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fbd1800c7f0 tx=0x7fbd1800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:13.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.154+0000 7fbd1e000640 1 -- 172.21.15.90:0/3176818383 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd18015070 con 0x7fbd28143940 2024-04-25T01:17:13.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.154+0000 7fbd1e000640 1 -- 172.21.15.90:0/3176818383 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd180079d0 con 0x7fbd28143940 2024-04-25T01:17:13.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.154+0000 7fbd1e000640 1 -- 172.21.15.90:0/3176818383 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd18007da0 con 0x7fbd28143940 2024-04-25T01:17:13.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.154+0000 7fbd2cc00640 1 -- 172.21.15.90:0/3176818383 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd280afe70 con 0x7fbd28143940 2024-04-25T01:17:13.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.154+0000 7fbd2cc00640 1 -- 172.21.15.90:0/3176818383 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd280b0430 con 0x7fbd28143940 2024-04-25T01:17:13.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.155+0000 7fbd2cc00640 1 -- 172.21.15.90:0/3176818383 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd28068fb0 con 0x7fbd28143940 2024-04-25T01:17:13.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.156+0000 7fbd1e000640 1 -- 172.21.15.90:0/3176818383 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbd18018070 con 0x7fbd28143940 2024-04-25T01:17:13.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.157+0000 7fbd1e000640 1 --2- 172.21.15.90:0/3176818383 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbd0406be30 0x7fbd0406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:13.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.157+0000 7fbd1e000640 1 -- 172.21.15.90:0/3176818383 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7fbd1808b160 con 0x7fbd28143940 2024-04-25T01:17:13.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.157+0000 7fbd1fe00640 1 --2- 172.21.15.90:0/3176818383 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbd0406be30 0x7fbd0406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:13.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.158+0000 7fbd1fe00640 1 --2- 172.21.15.90:0/3176818383 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbd0406be30 0x7fbd0406e2f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbd2815de40 tx=0x7fbd100079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:13.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.162+0000 7fbd1e000640 1 -- 172.21.15.90:0/3176818383 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fbd18059790 con 0x7fbd28143940 2024-04-25T01:17:13.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.399+0000 7fbd2cc00640 1 -- 172.21.15.90:0/3176818383 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fbd2806d150 con 0x7fbd28143940 2024-04-25T01:17:13.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.400+0000 7fbd1e000640 1 -- 172.21.15.90:0/3176818383 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7fbd18059090 con 0x7fbd28143940 2024-04-25T01:17:13.384 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T01:17:13.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.402+0000 7fbd2cc00640 1 -- 172.21.15.90:0/3176818383 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbd0406be30 msgr2=0x7fbd0406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:13.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.402+0000 7fbd2cc00640 1 --2- 172.21.15.90:0/3176818383 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbd0406be30 0x7fbd0406e2f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbd2815de40 tx=0x7fbd100079b0 comp rx=0 tx=0).stop 2024-04-25T01:17:13.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.402+0000 7fbd2cc00640 1 -- 172.21.15.90:0/3176818383 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd28143940 msgr2=0x7fbd280b1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:13.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.402+0000 7fbd2cc00640 1 --2- 172.21.15.90:0/3176818383 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd28143940 0x7fbd280b1020 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fbd1800c7f0 tx=0x7fbd1800cbf0 comp rx=0 tx=0).stop 2024-04-25T01:17:13.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.403+0000 7fbd2cc00640 1 -- 172.21.15.90:0/3176818383 shutdown_connections 2024-04-25T01:17:13.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.403+0000 7fbd2cc00640 1 --2- 172.21.15.90:0/3176818383 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbd0406be30 0x7fbd0406e2f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:13.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.403+0000 7fbd2cc00640 1 --2- 172.21.15.90:0/3176818383 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd2815f400 0x7fbd280ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:13.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.403+0000 7fbd2cc00640 1 --2- 172.21.15.90:0/3176818383 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbd28143d20 0x7fbd281f30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:13.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.403+0000 7fbd2cc00640 1 --2- 172.21.15.90:0/3176818383 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd28143940 0x7fbd280b1020 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:13.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.403+0000 7fbd2cc00640 1 -- 172.21.15.90:0/3176818383 >> 172.21.15.90:0/3176818383 conn(0x7fbd280b1820 msgr2=0x7fbd280b4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:13.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.403+0000 7fbd2cc00640 1 -- 172.21.15.90:0/3176818383 shutdown_connections 2024-04-25T01:17:13.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:13.403+0000 7fbd2cc00640 1 -- 172.21.15.90:0/3176818383 wait complete. 2024-04-25T01:17:13.405 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T01:17:13.405 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T01:17:13.405 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi103.front.sepia.ceph.com 2024-04-25T01:17:13.405 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-25T01:17:13.405 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-25T01:17:13.405 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T01:17:13.405 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T01:17:13.405 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T01:17:13.406 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-25T01:17:13.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:13.406 DEBUG:teuthology.orchestra.run.smithi103:> ip addr 2024-04-25T01:17:13.462 INFO:teuthology.orchestra.run.smithi103.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T01:17:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T01:17:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T01:17:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 ::1/128 scope host 2024-04-25T01:17:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:13.463 INFO:teuthology.orchestra.run.smithi103.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:17:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c4 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:17:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f0 2024-04-25T01:17:13.463 INFO:teuthology.orchestra.run.smithi103.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:17:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c5 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:17:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f1 2024-04-25T01:17:13.463 INFO:teuthology.orchestra.run.smithi103.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:17:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0a brd ff:ff:ff:ff:ff:ff 2024-04-25T01:17:13.464 INFO:teuthology.orchestra.run.smithi103.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T01:17:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0b brd ff:ff:ff:ff:ff:ff 2024-04-25T01:17:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: inet 172.21.15.103/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-25T01:17:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft 42612sec preferred_lft 42612sec 2024-04-25T01:17:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::ec4:7aff:fe8f:cc0b/64 scope link 2024-04-25T01:17:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:13.465 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T01:17:13.465 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:17:13.465 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-25T01:17:13.465 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link add name ceph-brx type bridge 2024-04-25T01:17:13.465 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip addr flush dev ceph-brx 2024-04-25T01:17:13.465 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set ceph-brx up 2024-04-25T01:17:13.465 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T01:17:13.465 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-25T01:17:13.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:17:13.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:13.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:13.940 DEBUG:teuthology.orchestra.run.smithi103:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T01:17:14.009 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-04-25T01:17:14.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:14.013 DEBUG:teuthology.orchestra.run.smithi103:> route 2024-04-25T01:17:14.069 INFO:teuthology.orchestra.run.smithi103.stdout:Kernel IP routing table 2024-04-25T01:17:14.069 INFO:teuthology.orchestra.run.smithi103.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T01:17:14.069 INFO:teuthology.orchestra.run.smithi103.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T01:17:14.070 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T01:17:14.070 INFO:teuthology.orchestra.run.smithi103.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T01:17:14.070 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:17:14.070 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-25T01:17:14.070 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T01:17:14.070 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T01:17:14.070 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T01:17:14.070 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-25T01:17:14.170 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:17:14.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:14.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:14.804 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-04-25T01:17:14.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:14.855 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list-id 2024-04-25T01:17:14.906 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:17:14.906 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-25T01:17:14.906 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T01:17:14.906 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-25T01:17:14.906 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-25T01:17:15.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:17:15.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:15.096 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-25T01:17:15.096 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:17:15.096 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-25T01:17:15.096 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-25T01:17:15.096 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T01:17:15.097 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-25T01:17:15.097 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-25T01:17:15.097 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-25T01:17:15.097 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-25T01:17:15.198 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:17:15.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:15.388 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:17:15.388 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-25T01:17:15.388 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set brx.0 up 2024-04-25T01:17:15.389 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T01:17:15.389 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-25T01:17:15.487 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:17:15.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:15.592 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:17:15.592 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-25T01:17:15.641 INFO:teuthology.orchestra.run.smithi103.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-25T01:17:15.642 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:17:15.642 DEBUG:teuthology.orchestra.run.smithi103:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-25T01:17:15.689 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-25T01:17:15.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:15.689 DEBUG:teuthology.orchestra.run.smithi103:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-25T01:17:15.811 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-25T01:17:15.811 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-04-25T01:17:15.811 INFO:teuthology.orchestra.run.smithi103.stdout:invalid new device string format 2024-04-25T01:17:16.529 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-25T01:17:16.529 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-04-25T01:17:16.530 INFO:teuthology.orchestra.run.smithi103.stdout:invalid new device string format 2024-04-25T01:17:16.530 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: resolved to: "172.21.15.90:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-25T01:17:16.530 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-25T01:17:16.567 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-04-25T01:17:16.568 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-04-25T01:17:16.568 INFO:teuthology.orchestra.run.smithi103.stdout:invalid new device string format 2024-04-25T01:17:16.568 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: resolved to: "172.21.15.90:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-25T01:17:16.568 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-25T01:17:16.568 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: resolved to: "172.21.15.90:6789,172.21.15.103:6789,172.21.15.103:6790" 2024-04-25T01:17:16.568 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.90:6789,172.21.15.103:6789,172.21.15.103:6790:/ 2024-04-25T01:17:16.568 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21" will pass to kernel 2024-04-25T01:17:16.571 INFO:teuthology.orchestra.run.smithi103.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-04-25T01:17:16.571 INFO:teuthology.orchestra.run.smithi103.stdout: You just mounted a file system that supports labels which does not 2024-04-25T01:17:16.571 INFO:teuthology.orchestra.run.smithi103.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T01:17:16.572 INFO:teuthology.orchestra.run.smithi103.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T01:17:16.572 INFO:teuthology.orchestra.run.smithi103.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T01:17:16.577 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T01:17:16.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:16.577 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-25T01:17:16.657 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:17:16.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:16 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:17:16.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:16.781 INFO:teuthology.orchestra.run.smithi103.stdout:client4633 2024-04-25T01:17:16.782 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:17:16.782 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:17:16.832 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:17:16.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:17 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:17:16.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:16.997 INFO:teuthology.orchestra.run.smithi103.stdout:client4633 2024-04-25T01:17:16.998 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:17:17.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:17 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:17:17.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:17.114 INFO:teuthology.orchestra.run.smithi103.stdout:client4633 2024-04-25T01:17:17.114 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21.client4633/status') 2024-04-25T01:17:17.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:17:17.244 INFO:teuthology.orchestra.run.smithi103.stdout:instance: client.4633 (3)192.168.0.1:0/3418984662 2024-04-25T01:17:17.244 INFO:teuthology.orchestra.run.smithi103.stdout:blocklisted: false 2024-04-25T01:17:17.244 INFO:teuthology.orchestra.run.smithi103.stderr:0+1 records in 2024-04-25T01:17:17.245 INFO:teuthology.orchestra.run.smithi103.stderr:0+1 records out 2024-04-25T01:17:17.245 INFO:teuthology.orchestra.run.smithi103.stderr:69 bytes copied, 9.1855e-05 s, 751 kB/s 2024-04-25T01:17:17.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:17.257 INFO:tasks.kclient:client.3 config is {'syntax': 'v1'} 2024-04-25T01:17:17.258 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T01:17:17.258 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-25T01:17:17.258 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T01:17:17.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.485+0000 7f57bee00640 1 Processor -- start 2024-04-25T01:17:17.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.485+0000 7f57bee00640 1 -- start start 2024-04-25T01:17:17.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.486+0000 7f57bee00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f57b80b41f0 0x7f57b806b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:17.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.486+0000 7f57bee00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f57b80ae9d0 0x7f57b806b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:17.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.486+0000 7f57bee00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57b80adc10 con 0x7f57b815e6f0 2024-04-25T01:17:17.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.486+0000 7f57bee00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f57b815f240 con 0x7f57b80b41f0 2024-04-25T01:17:17.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.486+0000 7f57bee00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f57b80adef0 con 0x7f57b80ae9d0 2024-04-25T01:17:17.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.487+0000 7f57bda00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f57b80b41f0 0x7f57b806b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:17.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.487+0000 7f57be400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f57b815e6f0 0x7f57b815eb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50464/0 (socket says 172.21.15.90:50464) 2024-04-25T01:17:17.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.487+0000 7f57be400640 1 -- 172.21.15.90:0/3111685190 learned_addr learned my addr 172.21.15.90:0/3111685190 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:17.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.487+0000 7f57bd000640 1 --2- 172.21.15.90:0/3111685190 >> v2:172.21.15.103:3300/0 conn(0x7f57b80ae9d0 0x7f57b806b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:17.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.487+0000 7f57bda00640 1 -- 172.21.15.90:0/3111685190 >> v2:172.21.15.103:3300/0 conn(0x7f57b80ae9d0 msgr2=0x7f57b806b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:17.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.487+0000 7f57bda00640 1 --2- 172.21.15.90:0/3111685190 >> v2:172.21.15.103:3300/0 conn(0x7f57b80ae9d0 0x7f57b806b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:17.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.487+0000 7f57bda00640 1 -- 172.21.15.90:0/3111685190 >> v1:172.21.15.90:6789/0 conn(0x7f57b815e6f0 legacy=0x7f57b815eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:17.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.487+0000 7f57bda00640 1 -- 172.21.15.90:0/3111685190 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57b80ae1d0 con 0x7f57b80b41f0 2024-04-25T01:17:17.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.487+0000 7f57b7e00640 1 -- 172.21.15.90:0/3111685190 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3275262447 0 0) 0x7f57b80adc10 con 0x7f57b815e6f0 2024-04-25T01:17:17.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.488+0000 7f57bda00640 1 --2- 172.21.15.90:0/3111685190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57b80b41f0 0x7f57b806b120 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f57b80cc240 tx=0x7f57ac02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d31e05e363bf8e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:17.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.489+0000 7f57b7e00640 1 -- 172.21.15.90:0/3111685190 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ac037070 con 0x7f57b80b41f0 2024-04-25T01:17:17.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.489+0000 7f57b7e00640 1 -- 172.21.15.90:0/3111685190 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57ac032440 con 0x7f57b80b41f0 2024-04-25T01:17:17.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.489+0000 7f57b7e00640 1 -- 172.21.15.90:0/3111685190 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ac032810 con 0x7f57b80b41f0 2024-04-25T01:17:17.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.489+0000 7f57bee00640 1 -- 172.21.15.90:0/3111685190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57b80b41f0 msgr2=0x7f57b806b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:17.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.489+0000 7f57bee00640 1 --2- 172.21.15.90:0/3111685190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57b80b41f0 0x7f57b806b120 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f57b80cc240 tx=0x7f57ac02a1a0 comp rx=0 tx=0).stop 2024-04-25T01:17:17.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.489+0000 7f57bee00640 1 -- 172.21.15.90:0/3111685190 shutdown_connections 2024-04-25T01:17:17.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.489+0000 7f57bee00640 1 --2- 172.21.15.90:0/3111685190 >> v2:172.21.15.103:3300/0 conn(0x7f57b80ae9d0 0x7f57b806b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:17.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.489+0000 7f57bee00640 1 --2- 172.21.15.90:0/3111685190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57b80b41f0 0x7f57b806b120 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:17.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.489+0000 7f57bee00640 1 -- 172.21.15.90:0/3111685190 >> 172.21.15.90:0/3111685190 conn(0x7f57b80b1660 msgr2=0x7f57b80b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:17.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.490+0000 7f57bee00640 1 -- 172.21.15.90:0/3111685190 shutdown_connections 2024-04-25T01:17:17.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.490+0000 7f57bee00640 1 -- 172.21.15.90:0/3111685190 wait complete. 2024-04-25T01:17:17.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.491+0000 7f57bee00640 1 Processor -- start 2024-04-25T01:17:17.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.491+0000 7f57bee00640 1 -- start start 2024-04-25T01:17:17.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.492+0000 7f57bee00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57b80ae9d0 0x7f57b815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:17.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.492+0000 7f57bee00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57b80b41f0 0x7f57b8071d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:17.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.492+0000 7f57bee00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57b815e6f0 0x7f57b8145d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:17.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.493+0000 7f57bee00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f57b8157db0 con 0x7f57b815e6f0 2024-04-25T01:17:17.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.493+0000 7f57bee00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f57b8157c30 con 0x7f57b80b41f0 2024-04-25T01:17:17.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.493+0000 7f57bee00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f57b8157f30 con 0x7f57b80ae9d0 2024-04-25T01:17:17.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.493+0000 7f57be400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57b815e6f0 0x7f57b8145d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:17.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.493+0000 7f57be400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57b815e6f0 0x7f57b8145d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36520/0 (socket says 172.21.15.90:36520) 2024-04-25T01:17:17.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.493+0000 7f57be400640 1 -- 172.21.15.90:0/580305771 learned_addr learned my addr 172.21.15.90:0/580305771 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:17.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.493+0000 7f57bda00640 1 --2- 172.21.15.90:0/580305771 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57b80ae9d0 0x7f57b815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:17.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.493+0000 7f57be400640 1 -- 172.21.15.90:0/580305771 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57b80ae9d0 msgr2=0x7f57b815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:17.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.493+0000 7f57be400640 1 --2- 172.21.15.90:0/580305771 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57b80ae9d0 0x7f57b815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:17.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.493+0000 7f57be400640 1 -- 172.21.15.90:0/580305771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57b80b41f0 msgr2=0x7f57b8071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:17.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.493+0000 7f57be400640 1 --2- 172.21.15.90:0/580305771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57b80b41f0 0x7f57b8071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:17.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.493+0000 7f57be400640 1 -- 172.21.15.90:0/580305771 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57b815e1e0 con 0x7f57b815e6f0 2024-04-25T01:17:17.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.494+0000 7f57be400640 1 --2- 172.21.15.90:0/580305771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57b815e6f0 0x7f57b8145d40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f57b000c780 tx=0x7f57b000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:17.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.494+0000 7f57b5600640 1 -- 172.21.15.90:0/580305771 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b0018070 con 0x7f57b815e6f0 2024-04-25T01:17:17.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.494+0000 7f57b5600640 1 -- 172.21.15.90:0/580305771 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57b000a640 con 0x7f57b815e6f0 2024-04-25T01:17:17.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.494+0000 7f57b5600640 1 -- 172.21.15.90:0/580305771 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b000aa30 con 0x7f57b815e6f0 2024-04-25T01:17:17.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.494+0000 7f57bee00640 1 -- 172.21.15.90:0/580305771 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57b81f7280 con 0x7f57b815e6f0 2024-04-25T01:17:17.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.494+0000 7f57bee00640 1 -- 172.21.15.90:0/580305771 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57b81f7840 con 0x7f57b815e6f0 2024-04-25T01:17:17.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.495+0000 7f57bee00640 1 -- 172.21.15.90:0/580305771 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57b8075350 con 0x7f57b815e6f0 2024-04-25T01:17:17.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.496+0000 7f57b5600640 1 -- 172.21.15.90:0/580305771 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f57b000abd0 con 0x7f57b815e6f0 2024-04-25T01:17:17.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.497+0000 7f57b5600640 1 --2- 172.21.15.90:0/580305771 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f579806be30 0x7f579806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:17.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.497+0000 7f57b5600640 1 -- 172.21.15.90:0/580305771 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f57b008a6e0 con 0x7f57b815e6f0 2024-04-25T01:17:17.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.497+0000 7f57bda00640 1 --2- 172.21.15.90:0/580305771 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f579806be30 0x7f579806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:17.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.498+0000 7f57bda00640 1 --2- 172.21.15.90:0/580305771 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f579806be30 0x7f579806e2f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f57ac034000 tx=0x7f57ac02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:17.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.502+0000 7f57b5600640 1 -- 172.21.15.90:0/580305771 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f57b0058d10 con 0x7f57b815e6f0 2024-04-25T01:17:17.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.743+0000 7f57bee00640 1 -- 172.21.15.90:0/580305771 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f57b815ed40 con 0x7f57b815e6f0 2024-04-25T01:17:17.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.744+0000 7f57b5600640 1 -- 172.21.15.90:0/580305771 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f57b0058610 con 0x7f57b815e6f0 2024-04-25T01:17:17.728 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T01:17:17.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.747+0000 7f57bee00640 1 -- 172.21.15.90:0/580305771 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f579806be30 msgr2=0x7f579806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:17.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.747+0000 7f57bee00640 1 --2- 172.21.15.90:0/580305771 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f579806be30 0x7f579806e2f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f57ac034000 tx=0x7f57ac02e000 comp rx=0 tx=0).stop 2024-04-25T01:17:17.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.747+0000 7f57bee00640 1 -- 172.21.15.90:0/580305771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57b815e6f0 msgr2=0x7f57b8145d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:17.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.747+0000 7f57bee00640 1 --2- 172.21.15.90:0/580305771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57b815e6f0 0x7f57b8145d40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f57b000c780 tx=0x7f57b000cc40 comp rx=0 tx=0).stop 2024-04-25T01:17:17.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.747+0000 7f57bee00640 1 -- 172.21.15.90:0/580305771 shutdown_connections 2024-04-25T01:17:17.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.747+0000 7f57bee00640 1 --2- 172.21.15.90:0/580305771 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f579806be30 0x7f579806e2f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:17.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.747+0000 7f57bee00640 1 --2- 172.21.15.90:0/580305771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57b815e6f0 0x7f57b8145d40 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:17.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.747+0000 7f57bee00640 1 --2- 172.21.15.90:0/580305771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57b80b41f0 0x7f57b8071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:17.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.747+0000 7f57bee00640 1 --2- 172.21.15.90:0/580305771 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57b80ae9d0 0x7f57b815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:17.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.747+0000 7f57bee00640 1 -- 172.21.15.90:0/580305771 >> 172.21.15.90:0/580305771 conn(0x7f57b80b1660 msgr2=0x7f57b806dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:17.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.747+0000 7f57bee00640 1 -- 172.21.15.90:0/580305771 shutdown_connections 2024-04-25T01:17:17.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:17.747+0000 7f57bee00640 1 -- 172.21.15.90:0/580305771 wait complete. 2024-04-25T01:17:17.750 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T01:17:17.750 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T01:17:17.750 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi103.front.sepia.ceph.com 2024-04-25T01:17:17.750 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-04-25T01:17:17.750 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-25T01:17:17.750 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T01:17:17.750 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T01:17:17.750 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T01:17:17.750 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-25T01:17:17.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:17.750 DEBUG:teuthology.orchestra.run.smithi103:> ip addr 2024-04-25T01:17:17.812 INFO:teuthology.orchestra.run.smithi103.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T01:17:17.812 INFO:teuthology.orchestra.run.smithi103.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T01:17:17.812 INFO:teuthology.orchestra.run.smithi103.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T01:17:17.812 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:17.812 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 ::1/128 scope host 2024-04-25T01:17:17.812 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:17.812 INFO:teuthology.orchestra.run.smithi103.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:17:17.812 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c4 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:17:17.812 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f0 2024-04-25T01:17:17.812 INFO:teuthology.orchestra.run.smithi103.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:17:17.812 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c5 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:17:17.813 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f1 2024-04-25T01:17:17.813 INFO:teuthology.orchestra.run.smithi103.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:17:17.813 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0a brd ff:ff:ff:ff:ff:ff 2024-04-25T01:17:17.813 INFO:teuthology.orchestra.run.smithi103.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T01:17:17.813 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0b brd ff:ff:ff:ff:ff:ff 2024-04-25T01:17:17.813 INFO:teuthology.orchestra.run.smithi103.stdout: inet 172.21.15.103/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-25T01:17:17.813 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft 42608sec preferred_lft 42608sec 2024-04-25T01:17:17.813 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::ec4:7aff:fe8f:cc0b/64 scope link 2024-04-25T01:17:17.813 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:17.813 INFO:teuthology.orchestra.run.smithi103.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T01:17:17.813 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether da:3a:bc:2a:75:d7 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:17:17.813 INFO:teuthology.orchestra.run.smithi103.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T01:17:17.813 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:17.813 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::d402:4bff:feaa:9edc/64 scope link 2024-04-25T01:17:17.813 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:17.814 INFO:teuthology.orchestra.run.smithi103.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T01:17:17.814 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether da:3a:bc:2a:75:d7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T01:17:17.814 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::d83a:bcff:fe2a:75d7/64 scope link 2024-04-25T01:17:17.814 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:17.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:17.815 DEBUG:teuthology.orchestra.run.smithi103:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T01:17:17.889 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-04-25T01:17:17.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:17.893 DEBUG:teuthology.orchestra.run.smithi103:> route 2024-04-25T01:17:17.943 INFO:teuthology.orchestra.run.smithi103.stdout:Kernel IP routing table 2024-04-25T01:17:17.943 INFO:teuthology.orchestra.run.smithi103.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T01:17:17.943 INFO:teuthology.orchestra.run.smithi103.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T01:17:17.943 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T01:17:17.943 INFO:teuthology.orchestra.run.smithi103.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T01:17:17.943 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:17:17.943 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-25T01:17:17.944 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T01:17:17.944 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T01:17:17.944 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T01:17:17.944 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-25T01:17:18.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:17:18.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:18.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:18.150 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-04-25T01:17:18.195 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-25T01:17:18.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:18.196 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list-id 2024-04-25T01:17:18.290 INFO:teuthology.orchestra.run.smithi103.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-04-25T01:17:18.290 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:17:18.290 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-25T01:17:18.290 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T01:17:18.291 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-04-25T01:17:18.291 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-25T01:17:18.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:17:18.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:18.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:18.469 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-25T01:17:18.549 INFO:teuthology.orchestra.run.smithi103.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T01:17:18.549 INFO:teuthology.orchestra.run.smithi103.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T01:17:18.550 INFO:teuthology.orchestra.run.smithi103.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T01:17:18.550 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:18.550 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 ::1/128 scope host 2024-04-25T01:17:18.550 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:18.550 INFO:teuthology.orchestra.run.smithi103.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T01:17:18.550 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether a6:04:a4:3f:c8:f2 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-25T01:17:18.551 INFO:teuthology.orchestra.run.smithi103.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-25T01:17:18.551 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:18.551 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::a404:a4ff:fe3f:c8f2/64 scope link 2024-04-25T01:17:18.551 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:18.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:18.553 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-25T01:17:18.634 INFO:teuthology.orchestra.run.smithi103.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T01:17:18.635 INFO:teuthology.orchestra.run.smithi103.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T01:17:18.635 INFO:teuthology.orchestra.run.smithi103.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T01:17:18.635 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:18.635 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 ::1/128 scope host 2024-04-25T01:17:18.635 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:18.635 INFO:teuthology.orchestra.run.smithi103.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T01:17:18.635 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether a6:04:a4:3f:c8:f2 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-25T01:17:18.635 INFO:teuthology.orchestra.run.smithi103.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-25T01:17:18.635 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:18.635 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::a404:a4ff:fe3f:c8f2/64 scope link 2024-04-25T01:17:18.635 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:17:18.638 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-04-25T01:17:18.639 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:17:18.639 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-25T01:17:18.639 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-04-25T01:17:18.639 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-25T01:17:18.639 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-04-25T01:17:18.639 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-04-25T01:17:18.639 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-04-25T01:17:18.639 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-25T01:17:18.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:17:18.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:18.908 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:17:18.908 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-04-25T01:17:18.908 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set brx.1 up 2024-04-25T01:17:18.908 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set dev brx.1 master ceph-brx 2024-04-25T01:17:18.909 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-04-25T01:17:19.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:19 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:17:19.102 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:19.111 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:17:19.111 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-04-25T01:17:19.153 INFO:teuthology.orchestra.run.smithi103.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-04-25T01:17:19.153 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:17:19.153 DEBUG:teuthology.orchestra.run.smithi103:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-04-25T01:17:19.248 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-25T01:17:19.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:19.248 DEBUG:teuthology.orchestra.run.smithi103:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-04-25T01:17:19.377 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-04-25T01:17:19.377 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-04-25T01:17:19.377 INFO:teuthology.orchestra.run.smithi103.stdout:invalid new device string format 2024-04-25T01:17:19.398 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-04-25T01:17:19.398 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-04-25T01:17:19.398 INFO:teuthology.orchestra.run.smithi103.stdout:invalid new device string format 2024-04-25T01:17:19.398 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: resolved to: "172.21.15.90:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-25T01:17:19.399 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-25T01:17:19.428 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-04-25T01:17:19.429 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-04-25T01:17:19.429 INFO:teuthology.orchestra.run.smithi103.stdout:invalid new device string format 2024-04-25T01:17:19.429 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: resolved to: "172.21.15.90:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-25T01:17:19.429 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-25T01:17:19.429 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: resolved to: "172.21.15.90:6789,172.21.15.103:6789,172.21.15.103:6790" 2024-04-25T01:17:19.429 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.90:6789,172.21.15.103:6789,172.21.15.103:6790:/ 2024-04-25T01:17:19.429 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs,key=3,fsid=e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21" will pass to kernel 2024-04-25T01:17:19.432 INFO:teuthology.orchestra.run.smithi103.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2024-04-25T01:17:19.432 INFO:teuthology.orchestra.run.smithi103.stdout: You just mounted a file system that supports labels which does not 2024-04-25T01:17:19.433 INFO:teuthology.orchestra.run.smithi103.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T01:17:19.433 INFO:teuthology.orchestra.run.smithi103.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T01:17:19.433 INFO:teuthology.orchestra.run.smithi103.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T01:17:19.437 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T01:17:19.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:19.437 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-04-25T01:17:19.514 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:17:19.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:19 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:17:19.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:19.632 INFO:teuthology.orchestra.run.smithi103.stdout:client4890 2024-04-25T01:17:19.633 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:17:19.633 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:17:19.688 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:17:19.723 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:17:19.731 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:17:19.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:19 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:17:19.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:19.806 INFO:teuthology.orchestra.run.smithi103.stdout:client4890 2024-04-25T01:17:19.806 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:17:19.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:19 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:17:19.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:19.908 INFO:teuthology.orchestra.run.smithi103.stdout:client4890 2024-04-25T01:17:19.908 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21.client4890/status') 2024-04-25T01:17:20.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:17:20.030 INFO:teuthology.orchestra.run.smithi103.stdout:instance: client.4890 (3)172.21.15.103:0/1754347913 2024-04-25T01:17:20.030 INFO:teuthology.orchestra.run.smithi103.stdout:blocklisted: false 2024-04-25T01:17:20.030 INFO:teuthology.orchestra.run.smithi103.stderr:0+1 records in 2024-04-25T01:17:20.030 INFO:teuthology.orchestra.run.smithi103.stderr:0+1 records out 2024-04-25T01:17:20.030 INFO:teuthology.orchestra.run.smithi103.stderr:71 bytes copied, 8.1274e-05 s, 874 kB/s 2024-04-25T01:17:20.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-25T01:17:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:17:20.045 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-04-25T01:17:20.062 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:17:20.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.288+0000 7f454e200640 1 Processor -- start 2024-04-25T01:17:20.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.288+0000 7f454e200640 1 -- start start 2024-04-25T01:17:20.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.290+0000 7f454e200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4548145590 0x7f45480b4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:20.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.290+0000 7f454e200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f454815f9b0 0x7f454815fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:20.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.290+0000 7f454e200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45480ce110 0x7f45480ce5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:20.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.290+0000 7f454e200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f4548160830 con 0x7f4548145590 2024-04-25T01:17:20.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.290+0000 7f454e200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4548160350 con 0x7f454815f9b0 2024-04-25T01:17:20.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.290+0000 7f454e200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f454806cde0 con 0x7f45480ce110 2024-04-25T01:17:20.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.290+0000 7f4547400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4548145590 0x7f45480b4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:20.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.290+0000 7f4547400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4548145590 0x7f45480b4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36526/0 (socket says 172.21.15.90:36526) 2024-04-25T01:17:20.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.290+0000 7f4547400640 1 -- 172.21.15.90:0/1736468475 learned_addr learned my addr 172.21.15.90:0/1736468475 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:20.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.291+0000 7f4546a00640 1 --2- 172.21.15.90:0/1736468475 >> v2:172.21.15.103:3300/0 conn(0x7f454815f9b0 0x7f454815fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:20.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.291+0000 7f4547e00640 1 --2- 172.21.15.90:0/1736468475 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45480ce110 0x7f45480ce5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:20.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.291+0000 7f4547400640 1 -- 172.21.15.90:0/1736468475 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45480ce110 msgr2=0x7f45480ce5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:20.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.291+0000 7f4547400640 1 --2- 172.21.15.90:0/1736468475 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45480ce110 0x7f45480ce5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:20.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.291+0000 7f4547400640 1 -- 172.21.15.90:0/1736468475 >> v2:172.21.15.103:3300/0 conn(0x7f454815f9b0 msgr2=0x7f454815fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:20.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.291+0000 7f4547400640 1 --2- 172.21.15.90:0/1736468475 >> v2:172.21.15.103:3300/0 conn(0x7f454815f9b0 0x7f454815fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:20.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.291+0000 7f4547400640 1 -- 172.21.15.90:0/1736468475 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45480b4240 con 0x7f4548145590 2024-04-25T01:17:20.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.291+0000 7f4547e00640 1 --2- 172.21.15.90:0/1736468475 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45480ce110 0x7f45480ce5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:20.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.292+0000 7f4547400640 1 --2- 172.21.15.90:0/1736468475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4548145590 0x7f45480b4a20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f45480b46c0 tx=0x7f453802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75970024c2baba8f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:20.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.292+0000 7f4546000640 1 -- 172.21.15.90:0/1736468475 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4538037070 con 0x7f4548145590 2024-04-25T01:17:20.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.292+0000 7f4546000640 1 -- 172.21.15.90:0/1736468475 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4538032440 con 0x7f4548145590 2024-04-25T01:17:20.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.292+0000 7f4546000640 1 -- 172.21.15.90:0/1736468475 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4538032810 con 0x7f4548145590 2024-04-25T01:17:20.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.293+0000 7f454e200640 1 -- 172.21.15.90:0/1736468475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4548145590 msgr2=0x7f45480b4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:20.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.293+0000 7f454e200640 1 --2- 172.21.15.90:0/1736468475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4548145590 0x7f45480b4a20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f45480b46c0 tx=0x7f453802a1e0 comp rx=0 tx=0).stop 2024-04-25T01:17:20.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.293+0000 7f454e200640 1 -- 172.21.15.90:0/1736468475 shutdown_connections 2024-04-25T01:17:20.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.293+0000 7f454e200640 1 --2- 172.21.15.90:0/1736468475 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f45480ce110 0x7f45480ce5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:20.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.293+0000 7f454e200640 1 --2- 172.21.15.90:0/1736468475 >> v2:172.21.15.103:3300/0 conn(0x7f454815f9b0 0x7f454815fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:20.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.293+0000 7f454e200640 1 --2- 172.21.15.90:0/1736468475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4548145590 0x7f45480b4a20 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:20.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.293+0000 7f454e200640 1 -- 172.21.15.90:0/1736468475 >> 172.21.15.90:0/1736468475 conn(0x7f45480b1820 msgr2=0x7f45480b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:20.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.293+0000 7f454e200640 1 -- 172.21.15.90:0/1736468475 shutdown_connections 2024-04-25T01:17:20.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.293+0000 7f454e200640 1 -- 172.21.15.90:0/1736468475 wait complete. 2024-04-25T01:17:20.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.295+0000 7f454e200640 1 Processor -- start 2024-04-25T01:17:20.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.295+0000 7f454e200640 1 -- start start 2024-04-25T01:17:20.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.296+0000 7f454e200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45480ce110 0x7f454815c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:20.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.296+0000 7f454e200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4548145590 0x7f4548071f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:20.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.296+0000 7f454e200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f454815f9b0 0x7f454813a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:20.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.296+0000 7f454e200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f45480b3c80 con 0x7f454815f9b0 2024-04-25T01:17:20.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.296+0000 7f454e200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f45480b39a0 con 0x7f45480ce110 2024-04-25T01:17:20.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.296+0000 7f454e200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f45480b3f60 con 0x7f4548145590 2024-04-25T01:17:20.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.296+0000 7f4547e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f454815f9b0 0x7f454813a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:20.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.296+0000 7f4547e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f454815f9b0 0x7f454813a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36532/0 (socket says 172.21.15.90:36532) 2024-04-25T01:17:20.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.296+0000 7f4547e00640 1 -- 172.21.15.90:0/2196193646 learned_addr learned my addr 172.21.15.90:0/2196193646 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:20.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.297+0000 7f4547400640 1 --2- 172.21.15.90:0/2196193646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45480ce110 0x7f454815c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:20.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.297+0000 7f4546a00640 1 --2- 172.21.15.90:0/2196193646 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4548145590 0x7f4548071f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:20.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.297+0000 7f4547e00640 1 -- 172.21.15.90:0/2196193646 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4548145590 msgr2=0x7f4548071f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:20.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.297+0000 7f4547e00640 1 --2- 172.21.15.90:0/2196193646 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4548145590 0x7f4548071f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:20.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.297+0000 7f4547e00640 1 -- 172.21.15.90:0/2196193646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45480ce110 msgr2=0x7f454815c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:20.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.297+0000 7f4547e00640 1 --2- 172.21.15.90:0/2196193646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45480ce110 0x7f454815c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:20.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.297+0000 7f4547e00640 1 -- 172.21.15.90:0/2196193646 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f454815a700 con 0x7f454815f9b0 2024-04-25T01:17:20.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.297+0000 7f4547e00640 1 --2- 172.21.15.90:0/2196193646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f454815f9b0 0x7f454813a0f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f454000cef0 tx=0x7f454000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:20.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.297+0000 7f453f400640 1 -- 172.21.15.90:0/2196193646 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f454001a070 con 0x7f454815f9b0 2024-04-25T01:17:20.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.297+0000 7f453f400640 1 -- 172.21.15.90:0/2196193646 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f454000d840 con 0x7f454815f9b0 2024-04-25T01:17:20.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.297+0000 7f454e200640 1 -- 172.21.15.90:0/2196193646 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f454815ca50 con 0x7f454815f9b0 2024-04-25T01:17:20.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.297+0000 7f453f400640 1 -- 172.21.15.90:0/2196193646 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f454000dc30 con 0x7f454815f9b0 2024-04-25T01:17:20.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.297+0000 7f454e200640 1 -- 172.21.15.90:0/2196193646 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f454815d010 con 0x7f454815f9b0 2024-04-25T01:17:20.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.299+0000 7f454e200640 1 -- 172.21.15.90:0/2196193646 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4548068fb0 con 0x7f454815f9b0 2024-04-25T01:17:20.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.300+0000 7f453f400640 1 -- 172.21.15.90:0/2196193646 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f454001d070 con 0x7f454815f9b0 2024-04-25T01:17:20.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.301+0000 7f453f400640 1 --2- 172.21.15.90:0/2196193646 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f452806be30 0x7f452806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:20.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.301+0000 7f453f400640 1 -- 172.21.15.90:0/2196193646 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f454008ba70 con 0x7f454815f9b0 2024-04-25T01:17:20.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.301+0000 7f4547400640 1 --2- 172.21.15.90:0/2196193646 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f452806be30 0x7f452806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:20.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.302+0000 7f4547400640 1 --2- 172.21.15.90:0/2196193646 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f452806be30 0x7f452806e2f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4538034000 tx=0x7f453802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:20.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.306+0000 7f453f400640 1 -- 172.21.15.90:0/2196193646 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f454005a010 con 0x7f454815f9b0 2024-04-25T01:17:20.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.503+0000 7f454e200640 1 -- 172.21.15.90:0/2196193646 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f454806ea50 con 0x7f454815f9b0 2024-04-25T01:17:20.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.506+0000 7f453f400640 1 -- 172.21.15.90:0/2196193646 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18576 (secure 0 0 0) 0x7f4540059910 con 0x7f454815f9b0 2024-04-25T01:17:20.490 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:17:20.491 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:17:09.346008+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:16:20.094512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:16:21.078268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:16:24.670642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.e","mappings":[{"from":0,"to":7}]},{"pgid":"4.f","mappings":[{"from":7,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:17:20.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.509+0000 7f454e200640 1 -- 172.21.15.90:0/2196193646 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f452806be30 msgr2=0x7f452806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:20.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.509+0000 7f454e200640 1 --2- 172.21.15.90:0/2196193646 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f452806be30 0x7f452806e2f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4538034000 tx=0x7f453802e000 comp rx=0 tx=0).stop 2024-04-25T01:17:20.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.509+0000 7f454e200640 1 -- 172.21.15.90:0/2196193646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f454815f9b0 msgr2=0x7f454813a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:20.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.509+0000 7f454e200640 1 --2- 172.21.15.90:0/2196193646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f454815f9b0 0x7f454813a0f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f454000cef0 tx=0x7f454000a710 comp rx=0 tx=0).stop 2024-04-25T01:17:20.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.509+0000 7f454e200640 1 -- 172.21.15.90:0/2196193646 shutdown_connections 2024-04-25T01:17:20.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.509+0000 7f454e200640 1 --2- 172.21.15.90:0/2196193646 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f452806be30 0x7f452806e2f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:20.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.509+0000 7f454e200640 1 --2- 172.21.15.90:0/2196193646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f454815f9b0 0x7f454813a0f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:20.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.509+0000 7f454e200640 1 --2- 172.21.15.90:0/2196193646 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4548145590 0x7f4548071f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:20.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.509+0000 7f454e200640 1 --2- 172.21.15.90:0/2196193646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45480ce110 0x7f454815c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:20.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.509+0000 7f454e200640 1 -- 172.21.15.90:0/2196193646 >> 172.21.15.90:0/2196193646 conn(0x7f45480b1820 msgr2=0x7f45480b26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:20.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.509+0000 7f454e200640 1 -- 172.21.15.90:0/2196193646 shutdown_connections 2024-04-25T01:17:20.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.509+0000 7f454e200640 1 -- 172.21.15.90:0/2196193646 wait complete. 2024-04-25T01:17:20.512 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:16:15.894126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:16:20.094512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:16:21.078268+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:16:24.670642+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:17:20.513 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:17:20.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.734+0000 7f6808000640 1 Processor -- start 2024-04-25T01:17:20.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.734+0000 7f6808000640 1 -- start start 2024-04-25T01:17:20.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.737+0000 7f6808000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f68000b3c60 0x7f680006df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:20.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.737+0000 7f6808000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f680006d140 con 0x7f6800158680 2024-04-25T01:17:20.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.737+0000 7f6808000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f680006d420 con 0x7f6800158a60 2024-04-25T01:17:20.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.737+0000 7f6808000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f680006d700 con 0x7f68000b3c60 2024-04-25T01:17:20.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.737+0000 7f6805600640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f6800158680 0x7f680006e6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50472/0 (socket says 172.21.15.90:50472) 2024-04-25T01:17:20.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.737+0000 7f6805600640 1 -- 172.21.15.90:0/124186455 learned_addr learned my addr 172.21.15.90:0/124186455 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:20.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.737+0000 7f6806000640 1 --2- 172.21.15.90:0/124186455 >> v2:172.21.15.90:3300/0 conn(0x7f68000b3c60 0x7f680006df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:20.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.737+0000 7f6806000640 1 -- 172.21.15.90:0/124186455 >> v1:172.21.15.103:6789/0 conn(0x7f6800158a60 legacy=0x7f68000b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:20.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.737+0000 7f6806000640 1 -- 172.21.15.90:0/124186455 >> v1:172.21.15.90:6789/0 conn(0x7f6800158680 legacy=0x7f680006e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:20.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.737+0000 7f67ffe00640 1 -- 172.21.15.90:0/124186455 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 294805802 0 0) 0x7f680006d140 con 0x7f6800158680 2024-04-25T01:17:20.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.737+0000 7f6806000640 1 -- 172.21.15.90:0/124186455 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f680006d9e0 con 0x7f68000b3c60 2024-04-25T01:17:20.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.738+0000 7f6806000640 1 --2- 172.21.15.90:0/124186455 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68000b3c60 0x7f680006df00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f67f80097a0 tx=0x7f67f802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b153868d8002107c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:20.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.739+0000 7f67ffe00640 1 -- 172.21.15.90:0/124186455 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f8037070 con 0x7f68000b3c60 2024-04-25T01:17:20.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.739+0000 7f67ffe00640 1 -- 172.21.15.90:0/124186455 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67f802ae50 con 0x7f68000b3c60 2024-04-25T01:17:20.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.739+0000 7f67ffe00640 1 -- 172.21.15.90:0/124186455 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f8032660 con 0x7f68000b3c60 2024-04-25T01:17:20.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.739+0000 7f6808000640 1 -- 172.21.15.90:0/124186455 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68000b3c60 msgr2=0x7f680006df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:20.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.739+0000 7f6808000640 1 --2- 172.21.15.90:0/124186455 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68000b3c60 0x7f680006df00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f67f80097a0 tx=0x7f67f802a140 comp rx=0 tx=0).stop 2024-04-25T01:17:20.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.739+0000 7f6808000640 1 -- 172.21.15.90:0/124186455 shutdown_connections 2024-04-25T01:17:20.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.739+0000 7f6808000640 1 --2- 172.21.15.90:0/124186455 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68000b3c60 0x7f680006df00 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:20.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.739+0000 7f6808000640 1 -- 172.21.15.90:0/124186455 >> 172.21.15.90:0/124186455 conn(0x7f680006b720 msgr2=0x7f68000b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:20.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.740+0000 7f6808000640 1 -- 172.21.15.90:0/124186455 shutdown_connections 2024-04-25T01:17:20.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.740+0000 7f6808000640 1 -- 172.21.15.90:0/124186455 wait complete. 2024-04-25T01:17:20.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.741+0000 7f6808000640 1 Processor -- start 2024-04-25T01:17:20.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.741+0000 7f6808000640 1 -- start start 2024-04-25T01:17:20.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.742+0000 7f6808000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68000b3c60 0x7f6800152d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:20.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.742+0000 7f6808000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6800158680 0x7f6800151310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:20.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.742+0000 7f6808000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6800158a60 0x7f6800160d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:20.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.742+0000 7f6808000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f680015f640 con 0x7f6800158a60 2024-04-25T01:17:20.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.742+0000 7f6808000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f680015f4c0 con 0x7f6800158680 2024-04-25T01:17:20.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.742+0000 7f6808000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6800153260 con 0x7f68000b3c60 2024-04-25T01:17:20.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.743+0000 7f6806000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6800158a60 0x7f6800160d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:20.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.743+0000 7f6806000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6800158a60 0x7f6800160d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36552/0 (socket says 172.21.15.90:36552) 2024-04-25T01:17:20.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.743+0000 7f6806000640 1 -- 172.21.15.90:0/3263455333 learned_addr learned my addr 172.21.15.90:0/3263455333 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:20.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.743+0000 7f6805600640 1 --2- 172.21.15.90:0/3263455333 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68000b3c60 0x7f6800152d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:20.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.743+0000 7f6804c00640 1 --2- 172.21.15.90:0/3263455333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6800158680 0x7f6800151310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:20.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.743+0000 7f6806000640 1 -- 172.21.15.90:0/3263455333 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68000b3c60 msgr2=0x7f6800152d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:20.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.743+0000 7f6806000640 1 --2- 172.21.15.90:0/3263455333 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68000b3c60 0x7f6800152d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:20.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.743+0000 7f6806000640 1 -- 172.21.15.90:0/3263455333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6800158680 msgr2=0x7f6800151310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:20.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.743+0000 7f6806000640 1 --2- 172.21.15.90:0/3263455333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6800158680 0x7f6800151310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:20.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.743+0000 7f6806000640 1 -- 172.21.15.90:0/3263455333 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6800151c80 con 0x7f6800158a60 2024-04-25T01:17:20.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.743+0000 7f6804c00640 1 --2- 172.21.15.90:0/3263455333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6800158680 0x7f6800151310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:20.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.743+0000 7f6806000640 1 --2- 172.21.15.90:0/3263455333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6800158a60 0x7f6800160d40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f67f80097a0 tx=0x7f67f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:20.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.744+0000 7f67fd600640 1 -- 172.21.15.90:0/3263455333 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f8042070 con 0x7f6800158a60 2024-04-25T01:17:20.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.744+0000 7f67fd600640 1 -- 172.21.15.90:0/3263455333 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67f8032e10 con 0x7f6800158a60 2024-04-25T01:17:20.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.744+0000 7f67fd600640 1 -- 172.21.15.90:0/3263455333 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f8037040 con 0x7f6800158a60 2024-04-25T01:17:20.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.744+0000 7f6808000640 1 -- 172.21.15.90:0/3263455333 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6800151f60 con 0x7f6800158a60 2024-04-25T01:17:20.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.744+0000 7f6808000640 1 -- 172.21.15.90:0/3263455333 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6800152800 con 0x7f6800158a60 2024-04-25T01:17:20.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.745+0000 7f6808000640 1 -- 172.21.15.90:0/3263455333 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6800068fb0 con 0x7f6800158a60 2024-04-25T01:17:20.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.746+0000 7f67fd600640 1 -- 172.21.15.90:0/3263455333 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f67f8045070 con 0x7f6800158a60 2024-04-25T01:17:20.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.747+0000 7f67fd600640 1 --2- 172.21.15.90:0/3263455333 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f67e006be30 0x7f67e006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:20.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.747+0000 7f67fd600640 1 -- 172.21.15.90:0/3263455333 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f67f80aba90 con 0x7f6800158a60 2024-04-25T01:17:20.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.747+0000 7f6805600640 1 --2- 172.21.15.90:0/3263455333 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f67e006be30 0x7f67e006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:20.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.748+0000 7f6805600640 1 --2- 172.21.15.90:0/3263455333 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f67e006be30 0x7f67e006e2f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f67f4007ed0 tx=0x7f67f4007e60 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:20.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.752+0000 7f67fd600640 1 -- 172.21.15.90:0/3263455333 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f67f807a030 con 0x7f6800158a60 2024-04-25T01:17:20.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.953+0000 7f6808000640 1 -- 172.21.15.90:0/3263455333 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f68000712a0 con 0x7f6800158a60 2024-04-25T01:17:20.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.954+0000 7f67fd600640 1 -- 172.21.15.90:0/3263455333 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) ==== 93+0+10 (secure 0 0 0) 0x7f67f8079930 con 0x7f6800158a60 2024-04-25T01:17:20.937 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T01:17:20.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.956+0000 7f6808000640 1 -- 172.21.15.90:0/3263455333 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f67e006be30 msgr2=0x7f67e006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:20.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.956+0000 7f6808000640 1 --2- 172.21.15.90:0/3263455333 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f67e006be30 0x7f67e006e2f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f67f4007ed0 tx=0x7f67f4007e60 comp rx=0 tx=0).stop 2024-04-25T01:17:20.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.956+0000 7f6808000640 1 -- 172.21.15.90:0/3263455333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6800158a60 msgr2=0x7f6800160d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:20.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.956+0000 7f6808000640 1 --2- 172.21.15.90:0/3263455333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6800158a60 0x7f6800160d40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f67f80097a0 tx=0x7f67f80046c0 comp rx=0 tx=0).stop 2024-04-25T01:17:20.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.957+0000 7f6808000640 1 -- 172.21.15.90:0/3263455333 shutdown_connections 2024-04-25T01:17:20.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.957+0000 7f6808000640 1 --2- 172.21.15.90:0/3263455333 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f67e006be30 0x7f67e006e2f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:20.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.957+0000 7f6808000640 1 --2- 172.21.15.90:0/3263455333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6800158a60 0x7f6800160d40 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:20.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.957+0000 7f6808000640 1 --2- 172.21.15.90:0/3263455333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6800158680 0x7f6800151310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:20.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.957+0000 7f6808000640 1 --2- 172.21.15.90:0/3263455333 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68000b3c60 0x7f6800152d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:20.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.957+0000 7f6808000640 1 -- 172.21.15.90:0/3263455333 >> 172.21.15.90:0/3263455333 conn(0x7f680006b720 msgr2=0x7f68000afea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:20.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.957+0000 7f6808000640 1 -- 172.21.15.90:0/3263455333 shutdown_connections 2024-04-25T01:17:20.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:20.957+0000 7f6808000640 1 -- 172.21.15.90:0/3263455333 wait complete. 2024-04-25T01:17:20.960 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:17:21.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.176+0000 7fef69800640 1 Processor -- start 2024-04-25T01:17:21.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.176+0000 7fef69800640 1 -- start start 2024-04-25T01:17:21.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.178+0000 7fef69800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fef64158680 0x7fef6406df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:21.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.178+0000 7fef69800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fef64158a60 0x7fef6406e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:21.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.178+0000 7fef69800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef6415f4c0 0x7fef640b3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:21.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.178+0000 7fef69800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fef6406d140 con 0x7fef64158680 2024-04-25T01:17:21.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.178+0000 7fef69800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fef6406d420 con 0x7fef64158a60 2024-04-25T01:17:21.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.178+0000 7fef69800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fef6406d700 con 0x7fef6415f4c0 2024-04-25T01:17:21.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.179+0000 7fef62a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fef64158680 0x7fef6406df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:21.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.179+0000 7fef62a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fef64158680 0x7fef6406df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36560/0 (socket says 172.21.15.90:36560) 2024-04-25T01:17:21.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.179+0000 7fef62a00640 1 -- 172.21.15.90:0/1501612281 learned_addr learned my addr 172.21.15.90:0/1501612281 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:21.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.179+0000 7fef63400640 1 --2- 172.21.15.90:0/1501612281 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef6415f4c0 0x7fef640b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:21.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.179+0000 7fef5be00640 1 --2- 172.21.15.90:0/1501612281 >> v2:172.21.15.103:3300/0 conn(0x7fef64158a60 0x7fef6406e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:21.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.179+0000 7fef62a00640 1 -- 172.21.15.90:0/1501612281 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef6415f4c0 msgr2=0x7fef640b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:21.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.179+0000 7fef62a00640 1 --2- 172.21.15.90:0/1501612281 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef6415f4c0 0x7fef640b3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:21.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.179+0000 7fef62a00640 1 -- 172.21.15.90:0/1501612281 >> v2:172.21.15.103:3300/0 conn(0x7fef64158a60 msgr2=0x7fef6406e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:21.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.179+0000 7fef62a00640 1 --2- 172.21.15.90:0/1501612281 >> v2:172.21.15.103:3300/0 conn(0x7fef64158a60 0x7fef6406e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:21.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.179+0000 7fef62a00640 1 -- 172.21.15.90:0/1501612281 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef6406d9e0 con 0x7fef64158680 2024-04-25T01:17:21.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.179+0000 7fef63400640 1 --2- 172.21.15.90:0/1501612281 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef6415f4c0 0x7fef640b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:21.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.180+0000 7fef62a00640 1 --2- 172.21.15.90:0/1501612281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef64158680 0x7fef6406df00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fef64070fd0 tx=0x7fef5402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c85a86de3d4c1c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:21.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.181+0000 7fef62000640 1 -- 172.21.15.90:0/1501612281 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef54037070 con 0x7fef64158680 2024-04-25T01:17:21.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.181+0000 7fef62000640 1 -- 172.21.15.90:0/1501612281 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef54032440 con 0x7fef64158680 2024-04-25T01:17:21.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.181+0000 7fef62000640 1 -- 172.21.15.90:0/1501612281 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef54032810 con 0x7fef64158680 2024-04-25T01:17:21.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.182+0000 7fef69800640 1 -- 172.21.15.90:0/1501612281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef64158680 msgr2=0x7fef6406df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:21.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.182+0000 7fef69800640 1 --2- 172.21.15.90:0/1501612281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef64158680 0x7fef6406df00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fef64070fd0 tx=0x7fef5402a140 comp rx=0 tx=0).stop 2024-04-25T01:17:21.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.182+0000 7fef69800640 1 -- 172.21.15.90:0/1501612281 shutdown_connections 2024-04-25T01:17:21.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.182+0000 7fef69800640 1 --2- 172.21.15.90:0/1501612281 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef6415f4c0 0x7fef640b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:21.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.182+0000 7fef69800640 1 --2- 172.21.15.90:0/1501612281 >> v2:172.21.15.103:3300/0 conn(0x7fef64158a60 0x7fef6406e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:21.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.182+0000 7fef69800640 1 --2- 172.21.15.90:0/1501612281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef64158680 0x7fef6406df00 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:21.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.182+0000 7fef69800640 1 -- 172.21.15.90:0/1501612281 >> 172.21.15.90:0/1501612281 conn(0x7fef6406b720 msgr2=0x7fef640b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:21.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.182+0000 7fef69800640 1 -- 172.21.15.90:0/1501612281 shutdown_connections 2024-04-25T01:17:21.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.182+0000 7fef69800640 1 -- 172.21.15.90:0/1501612281 wait complete. 2024-04-25T01:17:21.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.184+0000 7fef69800640 1 Processor -- start 2024-04-25T01:17:21.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.184+0000 7fef69800640 1 -- start start 2024-04-25T01:17:21.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.185+0000 7fef69800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef64158680 0x7fef6414bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:21.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.185+0000 7fef69800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef64158a60 0x7fef64151270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:21.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.185+0000 7fef69800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef6415f4c0 0x7fef64160eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:21.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.185+0000 7fef69800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fef6406edb0 con 0x7fef64158a60 2024-04-25T01:17:21.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.185+0000 7fef69800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fef6406ec30 con 0x7fef6415f4c0 2024-04-25T01:17:21.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.185+0000 7fef69800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fef640b3a90 con 0x7fef64158680 2024-04-25T01:17:21.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.185+0000 7fef5be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef64158a60 0x7fef64151270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:21.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.185+0000 7fef5be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef64158a60 0x7fef64151270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36576/0 (socket says 172.21.15.90:36576) 2024-04-25T01:17:21.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.185+0000 7fef5be00640 1 -- 172.21.15.90:0/1329286289 learned_addr learned my addr 172.21.15.90:0/1329286289 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:21.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.185+0000 7fef62a00640 1 --2- 172.21.15.90:0/1329286289 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef64158680 0x7fef6414bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:21.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.185+0000 7fef63400640 1 --2- 172.21.15.90:0/1329286289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef6415f4c0 0x7fef64160eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:21.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.186+0000 7fef5be00640 1 -- 172.21.15.90:0/1329286289 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef64158680 msgr2=0x7fef6414bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:21.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.186+0000 7fef5be00640 1 --2- 172.21.15.90:0/1329286289 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef64158680 0x7fef6414bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:21.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.186+0000 7fef5be00640 1 -- 172.21.15.90:0/1329286289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef6415f4c0 msgr2=0x7fef64160eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:21.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.186+0000 7fef5be00640 1 --2- 172.21.15.90:0/1329286289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef6415f4c0 0x7fef64160eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:21.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.186+0000 7fef5be00640 1 -- 172.21.15.90:0/1329286289 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef64153c40 con 0x7fef64158a60 2024-04-25T01:17:21.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.186+0000 7fef5be00640 1 --2- 172.21.15.90:0/1329286289 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef64158a60 0x7fef64151270 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fef4c00c7f0 tx=0x7fef4c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:21.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.186+0000 7fef5aa00640 1 -- 172.21.15.90:0/1329286289 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4c015070 con 0x7fef64158a60 2024-04-25T01:17:21.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.186+0000 7fef5aa00640 1 -- 172.21.15.90:0/1329286289 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef4c004220 con 0x7fef64158a60 2024-04-25T01:17:21.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.186+0000 7fef5aa00640 1 -- 172.21.15.90:0/1329286289 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4c007620 con 0x7fef64158a60 2024-04-25T01:17:21.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.186+0000 7fef69800640 1 -- 172.21.15.90:0/1329286289 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef641533a0 con 0x7fef64158a60 2024-04-25T01:17:21.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.186+0000 7fef69800640 1 -- 172.21.15.90:0/1329286289 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef64153960 con 0x7fef64158a60 2024-04-25T01:17:21.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.187+0000 7fef69800640 1 -- 172.21.15.90:0/1329286289 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef64068fb0 con 0x7fef64158a60 2024-04-25T01:17:21.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.188+0000 7fef5aa00640 1 -- 172.21.15.90:0/1329286289 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fef4c0079d0 con 0x7fef64158a60 2024-04-25T01:17:21.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.189+0000 7fef5aa00640 1 --2- 172.21.15.90:0/1329286289 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fef4006be30 0x7fef4006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:21.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.189+0000 7fef5aa00640 1 -- 172.21.15.90:0/1329286289 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7fef4c08b5d0 con 0x7fef64158a60 2024-04-25T01:17:21.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.190+0000 7fef62a00640 1 --2- 172.21.15.90:0/1329286289 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fef4006be30 0x7fef4006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:21.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.190+0000 7fef62a00640 1 --2- 172.21.15.90:0/1329286289 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fef4006be30 0x7fef4006e2f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fef54034000 tx=0x7fef5402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:21.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.193+0000 7fef5aa00640 1 -- 172.21.15.90:0/1329286289 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fef4c059c00 con 0x7fef64158a60 2024-04-25T01:17:21.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.388+0000 7fef69800640 1 -- 172.21.15.90:0/1329286289 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fef640712a0 con 0x7fef64158a60 2024-04-25T01:17:21.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.389+0000 7fef5aa00640 1 -- 172.21.15.90:0/1329286289 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) ==== 104+0+11 (secure 0 0 0) 0x7fef4c059500 con 0x7fef64158a60 2024-04-25T01:17:21.373 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:17:21.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.392+0000 7fef69800640 1 -- 172.21.15.90:0/1329286289 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fef4006be30 msgr2=0x7fef4006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:21.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.392+0000 7fef69800640 1 --2- 172.21.15.90:0/1329286289 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fef4006be30 0x7fef4006e2f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fef54034000 tx=0x7fef5402e000 comp rx=0 tx=0).stop 2024-04-25T01:17:21.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.392+0000 7fef69800640 1 -- 172.21.15.90:0/1329286289 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef64158a60 msgr2=0x7fef64151270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:21.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.392+0000 7fef69800640 1 --2- 172.21.15.90:0/1329286289 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef64158a60 0x7fef64151270 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fef4c00c7f0 tx=0x7fef4c00ccb0 comp rx=0 tx=0).stop 2024-04-25T01:17:21.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.392+0000 7fef69800640 1 -- 172.21.15.90:0/1329286289 shutdown_connections 2024-04-25T01:17:21.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.392+0000 7fef69800640 1 --2- 172.21.15.90:0/1329286289 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fef4006be30 0x7fef4006e2f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:21.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.392+0000 7fef69800640 1 --2- 172.21.15.90:0/1329286289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef6415f4c0 0x7fef64160eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:21.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.392+0000 7fef69800640 1 --2- 172.21.15.90:0/1329286289 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef64158a60 0x7fef64151270 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:21.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.392+0000 7fef69800640 1 --2- 172.21.15.90:0/1329286289 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fef64158680 0x7fef6414bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:21.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.392+0000 7fef69800640 1 -- 172.21.15.90:0/1329286289 >> 172.21.15.90:0/1329286289 conn(0x7fef6406b720 msgr2=0x7fef64157920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:21.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.392+0000 7fef69800640 1 -- 172.21.15.90:0/1329286289 shutdown_connections 2024-04-25T01:17:21.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.392+0000 7fef69800640 1 -- 172.21.15.90:0/1329286289 wait complete. 2024-04-25T01:17:21.396 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:17:21.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.621+0000 7fb787e00640 1 Processor -- start 2024-04-25T01:17:21.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.621+0000 7fb787e00640 1 -- start start 2024-04-25T01:17:21.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.623+0000 7fb787e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb78015fc40 0x7fb7801600a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:21.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.623+0000 7fb787e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb78006d110 con 0x7fb7800722e0 2024-04-25T01:17:21.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.623+0000 7fb787e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb78006d3f0 con 0x7fb7800cedc0 2024-04-25T01:17:21.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.623+0000 7fb787e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb780160aa0 con 0x7fb78015fc40 2024-04-25T01:17:21.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.623+0000 7fb785400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fb7800722e0 0x7fb78006ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50476/0 (socket says 172.21.15.90:50476) 2024-04-25T01:17:21.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.624+0000 7fb785400640 1 -- 172.21.15.90:0/301196488 learned_addr learned my addr 172.21.15.90:0/301196488 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:21.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.624+0000 7fb784a00640 1 --2- 172.21.15.90:0/301196488 >> v2:172.21.15.103:3300/0 conn(0x7fb78015fc40 0x7fb7801600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:21.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.624+0000 7fb77fe00640 1 -- 172.21.15.90:0/301196488 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3059951257 0 0) 0x7fb78006d110 con 0x7fb7800722e0 2024-04-25T01:17:21.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.624+0000 7fb77fe00640 1 -- 172.21.15.90:0/301196488 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb768003520 con 0x7fb7800722e0 2024-04-25T01:17:21.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.624+0000 7fb77fe00640 1 -- 172.21.15.90:0/301196488 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3224240861 0 0) 0x7fb768003520 con 0x7fb7800722e0 2024-04-25T01:17:21.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.624+0000 7fb77fe00640 1 -- 172.21.15.90:0/301196488 >> v2:172.21.15.103:3300/0 conn(0x7fb78015fc40 msgr2=0x7fb7801600a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:21.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.624+0000 7fb77fe00640 1 --2- 172.21.15.90:0/301196488 >> v2:172.21.15.103:3300/0 conn(0x7fb78015fc40 0x7fb7801600a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:21.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.624+0000 7fb77fe00640 1 -- 172.21.15.90:0/301196488 >> v1:172.21.15.103:6789/0 conn(0x7fb7800cedc0 legacy=0x7fb7800cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:21.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.624+0000 7fb77fe00640 1 -- 172.21.15.90:0/301196488 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb78006d6d0 con 0x7fb7800722e0 2024-04-25T01:17:21.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.624+0000 7fb77fe00640 1 -- 172.21.15.90:0/301196488 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7fb7740019f0 con 0x7fb7800722e0 2024-04-25T01:17:21.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.625+0000 7fb77fe00640 1 -- 172.21.15.90:0/301196488 >> v1:172.21.15.90:6789/0 conn(0x7fb7800722e0 legacy=0x7fb78006ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:21.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.625+0000 7fb77fe00640 1 --2- 172.21.15.90:0/301196488 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7680039b0 0x7fb768003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:21.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.625+0000 7fb77fe00640 1 --2- 172.21.15.90:0/301196488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb768004800 0x7fb768004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:21.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.625+0000 7fb77fe00640 1 --2- 172.21.15.90:0/301196488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7680052e0 0x7fb76800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:21.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.625+0000 7fb77fe00640 1 -- 172.21.15.90:0/301196488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb780160aa0 con 0x7fb7680052e0 2024-04-25T01:17:21.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.625+0000 7fb77fe00640 1 -- 172.21.15.90:0/301196488 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb76800c1b0 con 0x7fb768004800 2024-04-25T01:17:21.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.625+0000 7fb77fe00640 1 -- 172.21.15.90:0/301196488 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb76800c3b0 con 0x7fb7680039b0 2024-04-25T01:17:21.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.625+0000 7fb785e00640 1 --2- 172.21.15.90:0/301196488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7680052e0 0x7fb76800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:21.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.625+0000 7fb785400640 1 --2- 172.21.15.90:0/301196488 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7680039b0 0x7fb768003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:21.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.625+0000 7fb784a00640 1 --2- 172.21.15.90:0/301196488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb768004800 0x7fb768004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:21.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.626+0000 7fb785e00640 1 -- 172.21.15.90:0/301196488 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7680039b0 msgr2=0x7fb768003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:21.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.626+0000 7fb785e00640 1 --2- 172.21.15.90:0/301196488 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7680039b0 0x7fb768003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:21.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.626+0000 7fb785e00640 1 -- 172.21.15.90:0/301196488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb768004800 msgr2=0x7fb768004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:21.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.626+0000 7fb785e00640 1 --2- 172.21.15.90:0/301196488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb768004800 0x7fb768004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:21.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.626+0000 7fb785e00640 1 -- 172.21.15.90:0/301196488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb768003520 con 0x7fb7680052e0 2024-04-25T01:17:21.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.626+0000 7fb784a00640 1 --2- 172.21.15.90:0/301196488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb768004800 0x7fb768004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:21.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.627+0000 7fb785e00640 1 -- 172.21.15.90:0/301196488 reap_dead start 2024-04-25T01:17:21.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.627+0000 7fb785e00640 1 --2- 172.21.15.90:0/301196488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7680052e0 0x7fb76800b9c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb778001720 tx=0x7fb768005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=602ef85756a9f90d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:21.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.627+0000 7fb77fe00640 1 -- 172.21.15.90:0/301196488 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb778001370 con 0x7fb7680052e0 2024-04-25T01:17:21.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.627+0000 7fb77fe00640 1 -- 172.21.15.90:0/301196488 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7780052d0 con 0x7fb7680052e0 2024-04-25T01:17:21.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.627+0000 7fb77fe00640 1 -- 172.21.15.90:0/301196488 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77802abc0 con 0x7fb7680052e0 2024-04-25T01:17:21.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.628+0000 7fb787e00640 1 -- 172.21.15.90:0/301196488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7680052e0 msgr2=0x7fb76800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:21.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.628+0000 7fb787e00640 1 --2- 172.21.15.90:0/301196488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7680052e0 0x7fb76800b9c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb778001720 tx=0x7fb768005830 comp rx=0 tx=0).stop 2024-04-25T01:17:21.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.628+0000 7fb787e00640 1 -- 172.21.15.90:0/301196488 shutdown_connections 2024-04-25T01:17:21.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.628+0000 7fb787e00640 1 --2- 172.21.15.90:0/301196488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7680052e0 0x7fb76800b9c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:21.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.628+0000 7fb787e00640 1 -- 172.21.15.90:0/301196488 >> 172.21.15.90:0/301196488 conn(0x7fb78006b720 msgr2=0x7fb7800b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:21.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.628+0000 7fb787e00640 1 -- 172.21.15.90:0/301196488 shutdown_connections 2024-04-25T01:17:21.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.628+0000 7fb787e00640 1 -- 172.21.15.90:0/301196488 wait complete. 2024-04-25T01:17:21.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.630+0000 7fb787e00640 1 Processor -- start 2024-04-25T01:17:21.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.630+0000 7fb787e00640 1 -- start start 2024-04-25T01:17:21.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.631+0000 7fb787e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7680052e0 0x7fb780156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:21.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.631+0000 7fb787e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7800b0a20 0x7fb780149550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:21.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.631+0000 7fb787e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb78015b2c0 0x7fb78015b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:21.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.631+0000 7fb787e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb780160430 con 0x7fb78015b2c0 2024-04-25T01:17:21.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.631+0000 7fb787e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb7801602b0 con 0x7fb7680052e0 2024-04-25T01:17:21.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.631+0000 7fb787e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb7801605b0 con 0x7fb7800b0a20 2024-04-25T01:17:21.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.631+0000 7fb785e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb78015b2c0 0x7fb78015b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:21.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.631+0000 7fb785e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb78015b2c0 0x7fb78015b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36596/0 (socket says 172.21.15.90:36596) 2024-04-25T01:17:21.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.631+0000 7fb785e00640 1 -- 172.21.15.90:0/1151874316 learned_addr learned my addr 172.21.15.90:0/1151874316 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:21.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.632+0000 7fb784a00640 1 --2- 172.21.15.90:0/1151874316 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7800b0a20 0x7fb780149550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:21.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.632+0000 7fb785400640 1 --2- 172.21.15.90:0/1151874316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7680052e0 0x7fb780156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:21.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.632+0000 7fb785e00640 1 -- 172.21.15.90:0/1151874316 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7800b0a20 msgr2=0x7fb780149550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:21.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.632+0000 7fb785e00640 1 --2- 172.21.15.90:0/1151874316 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7800b0a20 0x7fb780149550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:21.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.632+0000 7fb785e00640 1 -- 172.21.15.90:0/1151874316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7680052e0 msgr2=0x7fb780156d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:21.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.632+0000 7fb785e00640 1 --2- 172.21.15.90:0/1151874316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7680052e0 0x7fb780156d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:21.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.632+0000 7fb785e00640 1 -- 172.21.15.90:0/1151874316 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb780157540 con 0x7fb78015b2c0 2024-04-25T01:17:21.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.632+0000 7fb784a00640 1 --2- 172.21.15.90:0/1151874316 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7800b0a20 0x7fb780149550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:21.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.632+0000 7fb785e00640 1 --2- 172.21.15.90:0/1151874316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb78015b2c0 0x7fb78015b740 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb778001810 tx=0x7fb778001190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:21.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.632+0000 7fb77d600640 1 -- 172.21.15.90:0/1151874316 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb778001370 con 0x7fb78015b2c0 2024-04-25T01:17:21.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.632+0000 7fb77d600640 1 -- 172.21.15.90:0/1151874316 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb778005480 con 0x7fb78015b2c0 2024-04-25T01:17:21.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.632+0000 7fb77d600640 1 -- 172.21.15.90:0/1151874316 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb778007a30 con 0x7fb78015b2c0 2024-04-25T01:17:21.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.632+0000 7fb787e00640 1 -- 172.21.15.90:0/1151874316 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb780157820 con 0x7fb78015b2c0 2024-04-25T01:17:21.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.633+0000 7fb787e00640 1 -- 172.21.15.90:0/1151874316 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb780157de0 con 0x7fb78015b2c0 2024-04-25T01:17:21.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.634+0000 7fb787e00640 1 -- 172.21.15.90:0/1151874316 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb780068fb0 con 0x7fb78015b2c0 2024-04-25T01:17:21.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.635+0000 7fb77d600640 1 -- 172.21.15.90:0/1151874316 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb77803b020 con 0x7fb78015b2c0 2024-04-25T01:17:21.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.636+0000 7fb77d600640 1 --2- 172.21.15.90:0/1151874316 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb76006be30 0x7fb76006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:21.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.636+0000 7fb77d600640 1 -- 172.21.15.90:0/1151874316 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7fb7780aa120 con 0x7fb78015b2c0 2024-04-25T01:17:21.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.636+0000 7fb785400640 1 --2- 172.21.15.90:0/1151874316 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb76006be30 0x7fb76006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:21.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.637+0000 7fb785400640 1 --2- 172.21.15.90:0/1151874316 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb76006be30 0x7fb76006e2f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb774001580 tx=0x7fb7740014d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:21.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.640+0000 7fb77d600640 1 -- 172.21.15.90:0/1151874316 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb778044020 con 0x7fb78015b2c0 2024-04-25T01:17:21.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.830+0000 7fb787e00640 1 -- 172.21.15.90:0/1151874316 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb780072c90 con 0x7fb78015b2c0 2024-04-25T01:17:21.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.831+0000 7fb77d600640 1 -- 172.21.15.90:0/1151874316 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) ==== 100+0+11 (secure 0 0 0) 0x7fb778078750 con 0x7fb78015b2c0 2024-04-25T01:17:21.816 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-25T01:17:21.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.834+0000 7fb787e00640 1 -- 172.21.15.90:0/1151874316 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb76006be30 msgr2=0x7fb76006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:21.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.834+0000 7fb787e00640 1 --2- 172.21.15.90:0/1151874316 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb76006be30 0x7fb76006e2f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb774001580 tx=0x7fb7740014d0 comp rx=0 tx=0).stop 2024-04-25T01:17:21.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.834+0000 7fb787e00640 1 -- 172.21.15.90:0/1151874316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb78015b2c0 msgr2=0x7fb78015b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:21.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.834+0000 7fb787e00640 1 --2- 172.21.15.90:0/1151874316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb78015b2c0 0x7fb78015b740 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb778001810 tx=0x7fb778001190 comp rx=0 tx=0).stop 2024-04-25T01:17:21.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.834+0000 7fb787e00640 1 -- 172.21.15.90:0/1151874316 shutdown_connections 2024-04-25T01:17:21.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.834+0000 7fb787e00640 1 --2- 172.21.15.90:0/1151874316 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb76006be30 0x7fb76006e2f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:21.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.834+0000 7fb787e00640 1 --2- 172.21.15.90:0/1151874316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb78015b2c0 0x7fb78015b740 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:21.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.834+0000 7fb787e00640 1 --2- 172.21.15.90:0/1151874316 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb7800b0a20 0x7fb780149550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:21.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.834+0000 7fb787e00640 1 --2- 172.21.15.90:0/1151874316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7680052e0 0x7fb780156d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:21.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.834+0000 7fb787e00640 1 -- 172.21.15.90:0/1151874316 >> 172.21.15.90:0/1151874316 conn(0x7fb78006b720 msgr2=0x7fb780074750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:21.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.834+0000 7fb787e00640 1 -- 172.21.15.90:0/1151874316 shutdown_connections 2024-04-25T01:17:21.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:21.834+0000 7fb787e00640 1 -- 172.21.15.90:0/1151874316 wait complete. 2024-04-25T01:17:21.838 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:17:22.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.045+0000 7fd068c00640 1 Processor -- start 2024-04-25T01:17:22.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.045+0000 7fd068c00640 1 -- start start 2024-04-25T01:17:22.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.047+0000 7fd068c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd064155130 0x7fd06406db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:22.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.047+0000 7fd068c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd06406e0b0 0x7fd06415b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:22.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.047+0000 7fd068c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd064155510 con 0x7fd06415b780 2024-04-25T01:17:22.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.047+0000 7fd068c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd06415bfe0 con 0x7fd06406e0b0 2024-04-25T01:17:22.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.047+0000 7fd068c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd06415d1d0 con 0x7fd064155130 2024-04-25T01:17:22.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.047+0000 7fd063e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd06415b780 0x7fd0640ae620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50484/0 (socket says 172.21.15.90:50484) 2024-04-25T01:17:22.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.047+0000 7fd063e00640 1 -- 172.21.15.90:0/3828958211 learned_addr learned my addr 172.21.15.90:0/3828958211 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:22.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.048+0000 7fd062000640 1 -- 172.21.15.90:0/3828958211 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 430642278 0 0) 0x7fd064155510 con 0x7fd06415b780 2024-04-25T01:17:22.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.048+0000 7fd062000640 1 -- 172.21.15.90:0/3828958211 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd048003520 con 0x7fd06415b780 2024-04-25T01:17:22.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.048+0000 7fd062a00640 1 --2- 172.21.15.90:0/3828958211 >> v2:172.21.15.103:3300/0 conn(0x7fd06406e0b0 0x7fd06415b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:22.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.048+0000 7fd063400640 1 --2- 172.21.15.90:0/3828958211 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd064155130 0x7fd06406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:22.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.048+0000 7fd062000640 1 -- 172.21.15.90:0/3828958211 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 301005345 0 0) 0x7fd048003520 con 0x7fd06415b780 2024-04-25T01:17:22.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.048+0000 7fd062000640 1 -- 172.21.15.90:0/3828958211 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd064155130 msgr2=0x7fd06406db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:22.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.048+0000 7fd062000640 1 --2- 172.21.15.90:0/3828958211 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd064155130 0x7fd06406db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.048+0000 7fd062000640 1 -- 172.21.15.90:0/3828958211 >> v2:172.21.15.103:3300/0 conn(0x7fd06406e0b0 msgr2=0x7fd06415b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:22.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.048+0000 7fd062000640 1 --2- 172.21.15.90:0/3828958211 >> v2:172.21.15.103:3300/0 conn(0x7fd06406e0b0 0x7fd06415b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.048+0000 7fd062000640 1 -- 172.21.15.90:0/3828958211 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd064071af0 con 0x7fd06415b780 2024-04-25T01:17:22.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.048+0000 7fd062000640 1 -- 172.21.15.90:0/3828958211 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7fd05c0019f0 con 0x7fd06415b780 2024-04-25T01:17:22.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.048+0000 7fd062000640 1 -- 172.21.15.90:0/3828958211 >> v1:172.21.15.90:6789/0 conn(0x7fd06415b780 legacy=0x7fd0640ae620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:22.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.048+0000 7fd062000640 1 --2- 172.21.15.90:0/3828958211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0480039b0 0x7fd048003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:22.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.048+0000 7fd062000640 1 --2- 172.21.15.90:0/3828958211 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd048004800 0x7fd048004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:22.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.048+0000 7fd062000640 1 --2- 172.21.15.90:0/3828958211 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0480051c0 0x7fd04800dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:22.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.048+0000 7fd062000640 1 -- 172.21.15.90:0/3828958211 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd06415bfe0 con 0x7fd0480039b0 2024-04-25T01:17:22.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.048+0000 7fd062000640 1 -- 172.21.15.90:0/3828958211 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd06415d1d0 con 0x7fd048004800 2024-04-25T01:17:22.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.048+0000 7fd062000640 1 -- 172.21.15.90:0/3828958211 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd04800e2a0 con 0x7fd0480051c0 2024-04-25T01:17:22.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.048+0000 7fd063400640 1 --2- 172.21.15.90:0/3828958211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0480039b0 0x7fd048003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:22.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.049+0000 7fd062a00640 1 --2- 172.21.15.90:0/3828958211 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd048004800 0x7fd048004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:22.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.049+0000 7fd063400640 1 -- 172.21.15.90:0/3828958211 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0480051c0 msgr2=0x7fd04800dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:22.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.049+0000 7fd063400640 1 --2- 172.21.15.90:0/3828958211 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0480051c0 0x7fd04800dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.049+0000 7fd063400640 1 -- 172.21.15.90:0/3828958211 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd048004800 msgr2=0x7fd048004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:22.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.049+0000 7fd063400640 1 --2- 172.21.15.90:0/3828958211 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd048004800 0x7fd048004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.049+0000 7fd063400640 1 -- 172.21.15.90:0/3828958211 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd048003520 con 0x7fd0480039b0 2024-04-25T01:17:22.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.049+0000 7fd062a00640 1 --2- 172.21.15.90:0/3828958211 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd048004800 0x7fd048004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:22.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.050+0000 7fd063400640 1 --2- 172.21.15.90:0/3828958211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0480039b0 0x7fd048003dd0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd05400b8e0 tx=0x7fd048005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc11ac76a86031a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:22.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.050+0000 7fd062000640 1 -- 172.21.15.90:0/3828958211 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd054030070 con 0x7fd0480039b0 2024-04-25T01:17:22.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.050+0000 7fd062000640 1 -- 172.21.15.90:0/3828958211 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd05402acc0 con 0x7fd0480039b0 2024-04-25T01:17:22.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.050+0000 7fd062000640 1 -- 172.21.15.90:0/3828958211 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05400b530 con 0x7fd0480039b0 2024-04-25T01:17:22.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.051+0000 7fd068c00640 1 -- 172.21.15.90:0/3828958211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0480039b0 msgr2=0x7fd048003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:22.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.051+0000 7fd068c00640 1 --2- 172.21.15.90:0/3828958211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0480039b0 0x7fd048003dd0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd05400b8e0 tx=0x7fd048005830 comp rx=0 tx=0).stop 2024-04-25T01:17:22.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.051+0000 7fd063e00640 1 -- 172.21.15.90:0/3828958211 reap_dead start 2024-04-25T01:17:22.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.051+0000 7fd068c00640 1 -- 172.21.15.90:0/3828958211 shutdown_connections 2024-04-25T01:17:22.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.051+0000 7fd068c00640 1 -- 172.21.15.90:0/3828958211 >> 172.21.15.90:0/3828958211 conn(0x7fd06406b120 msgr2=0x7fd06406b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:22.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.051+0000 7fd068c00640 1 -- 172.21.15.90:0/3828958211 shutdown_connections 2024-04-25T01:17:22.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.051+0000 7fd068c00640 1 -- 172.21.15.90:0/3828958211 wait complete. 2024-04-25T01:17:22.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.052+0000 7fd068c00640 1 Processor -- start 2024-04-25T01:17:22.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.053+0000 7fd068c00640 1 -- start start 2024-04-25T01:17:22.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.053+0000 7fd068c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0480039b0 0x7fd064153e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:22.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.053+0000 7fd068c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0640a94d0 0x7fd0640a9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:22.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.053+0000 7fd068c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd064157b70 0x7fd064157ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:22.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.053+0000 7fd068c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd0640728b0 con 0x7fd064157b70 2024-04-25T01:17:22.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.053+0000 7fd068c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd0640720b0 con 0x7fd0480039b0 2024-04-25T01:17:22.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.053+0000 7fd068c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd064072a30 con 0x7fd0640a94d0 2024-04-25T01:17:22.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.054+0000 7fd063e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd064157b70 0x7fd064157ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:22.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.054+0000 7fd063e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd064157b70 0x7fd064157ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36624/0 (socket says 172.21.15.90:36624) 2024-04-25T01:17:22.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.054+0000 7fd063e00640 1 -- 172.21.15.90:0/817487156 learned_addr learned my addr 172.21.15.90:0/817487156 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:22.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.054+0000 7fd063400640 1 --2- 172.21.15.90:0/817487156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0480039b0 0x7fd064153e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:22.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.054+0000 7fd062a00640 1 --2- 172.21.15.90:0/817487156 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0640a94d0 0x7fd0640a9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:22.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.054+0000 7fd063e00640 1 -- 172.21.15.90:0/817487156 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0640a94d0 msgr2=0x7fd0640a9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:22.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.054+0000 7fd063e00640 1 --2- 172.21.15.90:0/817487156 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0640a94d0 0x7fd0640a9950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.054+0000 7fd063e00640 1 -- 172.21.15.90:0/817487156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0480039b0 msgr2=0x7fd064153e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:22.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.054+0000 7fd063e00640 1 --2- 172.21.15.90:0/817487156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0480039b0 0x7fd064153e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.054+0000 7fd063e00640 1 -- 172.21.15.90:0/817487156 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0640a9fb0 con 0x7fd064157b70 2024-04-25T01:17:22.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.054+0000 7fd063e00640 1 --2- 172.21.15.90:0/817487156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd064157b70 0x7fd064157ff0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd064070c60 tx=0x7fd05c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:22.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.055+0000 7fd05b400640 1 -- 172.21.15.90:0/817487156 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c004d60 con 0x7fd064157b70 2024-04-25T01:17:22.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.055+0000 7fd05b400640 1 -- 172.21.15.90:0/817487156 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd05c00a610 con 0x7fd064157b70 2024-04-25T01:17:22.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.055+0000 7fd05b400640 1 -- 172.21.15.90:0/817487156 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c00aa00 con 0x7fd064157b70 2024-04-25T01:17:22.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.055+0000 7fd068c00640 1 -- 172.21.15.90:0/817487156 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0640aa290 con 0x7fd064157b70 2024-04-25T01:17:22.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.055+0000 7fd068c00640 1 -- 172.21.15.90:0/817487156 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd064155990 con 0x7fd064157b70 2024-04-25T01:17:22.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.056+0000 7fd068c00640 1 -- 172.21.15.90:0/817487156 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd064075350 con 0x7fd064157b70 2024-04-25T01:17:22.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.057+0000 7fd05b400640 1 -- 172.21.15.90:0/817487156 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd05c011070 con 0x7fd064157b70 2024-04-25T01:17:22.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.057+0000 7fd05b400640 1 --2- 172.21.15.90:0/817487156 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd04406be30 0x7fd04406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:22.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.058+0000 7fd05b400640 1 -- 172.21.15.90:0/817487156 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7fd05c08abb0 con 0x7fd064157b70 2024-04-25T01:17:22.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.058+0000 7fd063400640 1 --2- 172.21.15.90:0/817487156 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd04406be30 0x7fd04406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:22.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.058+0000 7fd063400640 1 --2- 172.21.15.90:0/817487156 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd04406be30 0x7fd04406e2f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd054035000 tx=0x7fd054030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:22.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.062+0000 7fd05b400640 1 -- 172.21.15.90:0/817487156 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd05c01a060 con 0x7fd064157b70 2024-04-25T01:17:22.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.258+0000 7fd068c00640 1 -- 172.21.15.90:0/817487156 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd064155b60 con 0x7fd064157b70 2024-04-25T01:17:22.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.259+0000 7fd05b400640 1 -- 172.21.15.90:0/817487156 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) ==== 103+0+11 (secure 0 0 0) 0x7fd05c059150 con 0x7fd064157b70 2024-04-25T01:17:22.244 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:17:22.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.262+0000 7fd068c00640 1 -- 172.21.15.90:0/817487156 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd04406be30 msgr2=0x7fd04406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:22.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.262+0000 7fd068c00640 1 --2- 172.21.15.90:0/817487156 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd04406be30 0x7fd04406e2f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd054035000 tx=0x7fd054030000 comp rx=0 tx=0).stop 2024-04-25T01:17:22.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.262+0000 7fd068c00640 1 -- 172.21.15.90:0/817487156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd064157b70 msgr2=0x7fd064157ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:22.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.262+0000 7fd068c00640 1 --2- 172.21.15.90:0/817487156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd064157b70 0x7fd064157ff0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd064070c60 tx=0x7fd05c00cb70 comp rx=0 tx=0).stop 2024-04-25T01:17:22.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.262+0000 7fd068c00640 1 -- 172.21.15.90:0/817487156 shutdown_connections 2024-04-25T01:17:22.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.262+0000 7fd068c00640 1 --2- 172.21.15.90:0/817487156 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd04406be30 0x7fd04406e2f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.262+0000 7fd068c00640 1 --2- 172.21.15.90:0/817487156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd064157b70 0x7fd064157ff0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.262+0000 7fd068c00640 1 --2- 172.21.15.90:0/817487156 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd0640a94d0 0x7fd0640a9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.262+0000 7fd068c00640 1 --2- 172.21.15.90:0/817487156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0480039b0 0x7fd064153e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.262+0000 7fd068c00640 1 -- 172.21.15.90:0/817487156 >> 172.21.15.90:0/817487156 conn(0x7fd06406b120 msgr2=0x7fd0640cfc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:22.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.262+0000 7fd068c00640 1 -- 172.21.15.90:0/817487156 shutdown_connections 2024-04-25T01:17:22.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.262+0000 7fd068c00640 1 -- 172.21.15.90:0/817487156 wait complete. 2024-04-25T01:17:22.266 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:17:22.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.481+0000 7f9be5c00640 1 Processor -- start 2024-04-25T01:17:22.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.481+0000 7f9be5c00640 1 -- start start 2024-04-25T01:17:22.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.483+0000 7f9be5c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9be006b120 0x7f9be006b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:22.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.483+0000 7f9be5c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9be00ae730 0x7f9be00cd900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:22.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.483+0000 7f9be5c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9be006ba40 con 0x7f9be015ca10 2024-04-25T01:17:22.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.483+0000 7f9be5c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f9be00cef40 con 0x7f9be006b120 2024-04-25T01:17:22.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.483+0000 7f9be5c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9be015c580 con 0x7f9be00ae730 2024-04-25T01:17:22.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.483+0000 7f9be5200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f9be015ca10 0x7f9be00cc030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50486/0 (socket says 172.21.15.90:50486) 2024-04-25T01:17:22.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.483+0000 7f9be5200640 1 -- 172.21.15.90:0/2928843728 learned_addr learned my addr 172.21.15.90:0/2928843728 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:22.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.483+0000 7f9bdfe00640 1 --2- 172.21.15.90:0/2928843728 >> v2:172.21.15.90:3300/0 conn(0x7f9be006b120 0x7f9be006b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:22.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.484+0000 7f9bdf400640 1 --2- 172.21.15.90:0/2928843728 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9be00ae730 0x7f9be00cd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:22.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.484+0000 7f9bdfe00640 1 -- 172.21.15.90:0/2928843728 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9be00ae730 msgr2=0x7f9be00cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:22.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.484+0000 7f9bdfe00640 1 --2- 172.21.15.90:0/2928843728 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9be00ae730 0x7f9be00cd900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.484+0000 7f9bdfe00640 1 -- 172.21.15.90:0/2928843728 >> v1:172.21.15.90:6789/0 conn(0x7f9be015ca10 legacy=0x7f9be00cc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:22.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.484+0000 7f9bdea00640 1 -- 172.21.15.90:0/2928843728 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 890350302 0 0) 0x7f9be006ba40 con 0x7f9be015ca10 2024-04-25T01:17:22.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.484+0000 7f9bdfe00640 1 -- 172.21.15.90:0/2928843728 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9be015c220 con 0x7f9be006b120 2024-04-25T01:17:22.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.485+0000 7f9bdfe00640 1 --2- 172.21.15.90:0/2928843728 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9be006b120 0x7f9be006b500 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8009d90 tx=0x7f9bd802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e38d6409a11382cb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:22.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.485+0000 7f9bdea00640 1 -- 172.21.15.90:0/2928843728 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8007740 con 0x7f9be006b120 2024-04-25T01:17:22.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.485+0000 7f9bdea00640 1 -- 172.21.15.90:0/2928843728 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bd80078e0 con 0x7f9be006b120 2024-04-25T01:17:22.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.485+0000 7f9bdea00640 1 -- 172.21.15.90:0/2928843728 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8007cb0 con 0x7f9be006b120 2024-04-25T01:17:22.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.486+0000 7f9be5c00640 1 -- 172.21.15.90:0/2928843728 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9be006b120 msgr2=0x7f9be006b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:22.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.486+0000 7f9be5c00640 1 --2- 172.21.15.90:0/2928843728 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9be006b120 0x7f9be006b500 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8009d90 tx=0x7f9bd802a660 comp rx=0 tx=0).stop 2024-04-25T01:17:22.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.486+0000 7f9be5c00640 1 -- 172.21.15.90:0/2928843728 shutdown_connections 2024-04-25T01:17:22.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.486+0000 7f9be5c00640 1 --2- 172.21.15.90:0/2928843728 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9be00ae730 0x7f9be00cd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.486+0000 7f9be5c00640 1 --2- 172.21.15.90:0/2928843728 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9be006b120 0x7f9be006b500 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.486+0000 7f9be5c00640 1 -- 172.21.15.90:0/2928843728 >> 172.21.15.90:0/2928843728 conn(0x7f9be00b1660 msgr2=0x7f9be00b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:22.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.486+0000 7f9be5c00640 1 -- 172.21.15.90:0/2928843728 shutdown_connections 2024-04-25T01:17:22.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.486+0000 7f9be5c00640 1 -- 172.21.15.90:0/2928843728 wait complete. 2024-04-25T01:17:22.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.488+0000 7f9be5c00640 1 Processor -- start 2024-04-25T01:17:22.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.488+0000 7f9be5c00640 1 -- start start 2024-04-25T01:17:22.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.489+0000 7f9be5c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9be006b120 0x7f9be014c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:22.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.489+0000 7f9be5c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9be00ae730 0x7f9be01f4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:22.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.489+0000 7f9be5c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9be015ca10 0x7f9be00b3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:22.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.489+0000 7f9be5c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9be00af240 con 0x7f9be00ae730 2024-04-25T01:17:22.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.489+0000 7f9be5c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9be00aef60 con 0x7f9be006b120 2024-04-25T01:17:22.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.489+0000 7f9be5c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9be0153080 con 0x7f9be015ca10 2024-04-25T01:17:22.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.489+0000 7f9bdf400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9be00ae730 0x7f9be01f4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:22.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.489+0000 7f9bdf400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9be00ae730 0x7f9be01f4020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36640/0 (socket says 172.21.15.90:36640) 2024-04-25T01:17:22.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.489+0000 7f9bdf400640 1 -- 172.21.15.90:0/3098139698 learned_addr learned my addr 172.21.15.90:0/3098139698 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:22.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.489+0000 7f9bdfe00640 1 --2- 172.21.15.90:0/3098139698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9be006b120 0x7f9be014c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:22.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.489+0000 7f9be5200640 1 --2- 172.21.15.90:0/3098139698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9be015ca10 0x7f9be00b3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:22.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.489+0000 7f9bdf400640 1 -- 172.21.15.90:0/3098139698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9be015ca10 msgr2=0x7f9be00b3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:22.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.489+0000 7f9bdf400640 1 --2- 172.21.15.90:0/3098139698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9be015ca10 0x7f9be00b3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.489+0000 7f9bdf400640 1 -- 172.21.15.90:0/3098139698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9be006b120 msgr2=0x7f9be014c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:22.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.489+0000 7f9bdf400640 1 --2- 172.21.15.90:0/3098139698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9be006b120 0x7f9be014c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.489+0000 7f9bdf400640 1 -- 172.21.15.90:0/3098139698 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9be01f4f50 con 0x7f9be00ae730 2024-04-25T01:17:22.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.489+0000 7f9bdfe00640 1 --2- 172.21.15.90:0/3098139698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9be006b120 0x7f9be014c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:22.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.490+0000 7f9bdf400640 1 --2- 172.21.15.90:0/3098139698 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9be00ae730 0x7f9be01f4020 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc00d830 tx=0x7f9bcc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:22.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.490+0000 7f9bd7e00640 1 -- 172.21.15.90:0/3098139698 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc00b660 con 0x7f9be00ae730 2024-04-25T01:17:22.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.490+0000 7f9bd7e00640 1 -- 172.21.15.90:0/3098139698 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bcc019070 con 0x7f9be00ae730 2024-04-25T01:17:22.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.490+0000 7f9bd7e00640 1 -- 172.21.15.90:0/3098139698 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc00ba20 con 0x7f9be00ae730 2024-04-25T01:17:22.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.490+0000 7f9be5c00640 1 -- 172.21.15.90:0/3098139698 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9be01f4990 con 0x7f9be00ae730 2024-04-25T01:17:22.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.490+0000 7f9be5c00640 1 -- 172.21.15.90:0/3098139698 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9be01f5540 con 0x7f9be00ae730 2024-04-25T01:17:22.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.491+0000 7f9be5c00640 1 -- 172.21.15.90:0/3098139698 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9be0075350 con 0x7f9be00ae730 2024-04-25T01:17:22.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.492+0000 7f9bd7e00640 1 -- 172.21.15.90:0/3098139698 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9bcc01e460 con 0x7f9be00ae730 2024-04-25T01:17:22.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.493+0000 7f9bd7e00640 1 --2- 172.21.15.90:0/3098139698 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9bc006be30 0x7f9bc006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:22.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.493+0000 7f9bd7e00640 1 -- 172.21.15.90:0/3098139698 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f9bcc08b850 con 0x7f9be00ae730 2024-04-25T01:17:22.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.493+0000 7f9bdfe00640 1 --2- 172.21.15.90:0/3098139698 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9bc006be30 0x7f9bc006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:22.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.494+0000 7f9bdfe00640 1 --2- 172.21.15.90:0/3098139698 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9bc006be30 0x7f9bc006e2f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8034000 tx=0x7f9bd802f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:22.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.498+0000 7f9bd7e00640 1 -- 172.21.15.90:0/3098139698 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9bcc059e80 con 0x7f9be00ae730 2024-04-25T01:17:22.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.703+0000 7f9be5c00640 1 -- 172.21.15.90:0/3098139698 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9be00716a0 con 0x7f9be00ae730 2024-04-25T01:17:22.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.705+0000 7f9bd7e00640 1 -- 172.21.15.90:0/3098139698 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18576 (secure 0 0 0) 0x7f9bcc059780 con 0x7f9be00ae730 2024-04-25T01:17:22.690 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:17:22.690 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:17:09.346008+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:16:20.094512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:16:21.078268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:16:24.670642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.e","mappings":[{"from":0,"to":7}]},{"pgid":"4.f","mappings":[{"from":7,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:17:22.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.708+0000 7f9be5c00640 1 -- 172.21.15.90:0/3098139698 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9bc006be30 msgr2=0x7f9bc006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:22.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.708+0000 7f9be5c00640 1 --2- 172.21.15.90:0/3098139698 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9bc006be30 0x7f9bc006e2f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8034000 tx=0x7f9bd802f000 comp rx=0 tx=0).stop 2024-04-25T01:17:22.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.708+0000 7f9be5c00640 1 -- 172.21.15.90:0/3098139698 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9be00ae730 msgr2=0x7f9be01f4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:22.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.708+0000 7f9be5c00640 1 --2- 172.21.15.90:0/3098139698 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9be00ae730 0x7f9be01f4020 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc00d830 tx=0x7f9bcc00dcf0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.708+0000 7f9be5c00640 1 -- 172.21.15.90:0/3098139698 shutdown_connections 2024-04-25T01:17:22.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.708+0000 7f9be5c00640 1 --2- 172.21.15.90:0/3098139698 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9bc006be30 0x7f9bc006e2f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.708+0000 7f9be5c00640 1 --2- 172.21.15.90:0/3098139698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9be015ca10 0x7f9be00b3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.708+0000 7f9be5c00640 1 --2- 172.21.15.90:0/3098139698 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9be00ae730 0x7f9be01f4020 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.708+0000 7f9be5c00640 1 --2- 172.21.15.90:0/3098139698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9be006b120 0x7f9be014c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.708+0000 7f9be5c00640 1 -- 172.21.15.90:0/3098139698 >> 172.21.15.90:0/3098139698 conn(0x7f9be00b1660 msgr2=0x7f9be006df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:22.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.708+0000 7f9be5c00640 1 -- 172.21.15.90:0/3098139698 shutdown_connections 2024-04-25T01:17:22.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.709+0000 7f9be5c00640 1 -- 172.21.15.90:0/3098139698 wait complete. 2024-04-25T01:17:22.713 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:16:15.894126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:16:20.094512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:16:21.078268+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:16:24.670642+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:17:22.714 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:17:22.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.931+0000 7febefc00640 1 Processor -- start 2024-04-25T01:17:22.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.931+0000 7febefc00640 1 -- start start 2024-04-25T01:17:22.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.932+0000 7febefc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febe80722e0 0x7febe806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:22.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.932+0000 7febefc00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7febe815fc40 0x7febe81600c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:22.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.933+0000 7febefc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7febe80cedc0 0x7febe80cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:22.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.933+0000 7febefc00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7febe8160aa0 con 0x7febe80cedc0 2024-04-25T01:17:22.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.933+0000 7febefc00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7febe806da40 con 0x7febe815fc40 2024-04-25T01:17:22.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.933+0000 7febefc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7febe806cec0 con 0x7febe80722e0 2024-04-25T01:17:22.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.933+0000 7febedc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7febe80cedc0 0x7febe80cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:22.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.933+0000 7febedc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7febe80cedc0 0x7febe80cf250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36644/0 (socket says 172.21.15.90:36644) 2024-04-25T01:17:22.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.933+0000 7febedc00640 1 -- 172.21.15.90:0/1843943344 learned_addr learned my addr 172.21.15.90:0/1843943344 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:22.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.933+0000 7febed200640 1 --2- 172.21.15.90:0/1843943344 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febe80722e0 0x7febe806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:22.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.933+0000 7febe7e00640 1 --2- 172.21.15.90:0/1843943344 >> v2:172.21.15.103:3300/0 conn(0x7febe815fc40 0x7febe81600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:22.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.933+0000 7febedc00640 1 -- 172.21.15.90:0/1843943344 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febe80722e0 msgr2=0x7febe806ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:22.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.933+0000 7febedc00640 1 --2- 172.21.15.90:0/1843943344 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febe80722e0 0x7febe806ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.933+0000 7febedc00640 1 -- 172.21.15.90:0/1843943344 >> v2:172.21.15.103:3300/0 conn(0x7febe815fc40 msgr2=0x7febe81600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:22.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.933+0000 7febedc00640 1 --2- 172.21.15.90:0/1843943344 >> v2:172.21.15.103:3300/0 conn(0x7febe815fc40 0x7febe81600c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.933+0000 7febedc00640 1 -- 172.21.15.90:0/1843943344 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febe80cf790 con 0x7febe80cedc0 2024-04-25T01:17:22.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.934+0000 7febedc00640 1 --2- 172.21.15.90:0/1843943344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febe80cedc0 0x7febe80cf250 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7febe80b4f60 tx=0x7febe002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c495e8f6098e933f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:22.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.935+0000 7febe7400640 1 -- 172.21.15.90:0/1843943344 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe0036070 con 0x7febe80cedc0 2024-04-25T01:17:22.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.935+0000 7febe7400640 1 -- 172.21.15.90:0/1843943344 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febe0031600 con 0x7febe80cedc0 2024-04-25T01:17:22.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.935+0000 7febe7400640 1 -- 172.21.15.90:0/1843943344 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe00319d0 con 0x7febe80cedc0 2024-04-25T01:17:22.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.935+0000 7febefc00640 1 -- 172.21.15.90:0/1843943344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febe80cedc0 msgr2=0x7febe80cf250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:22.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.935+0000 7febefc00640 1 --2- 172.21.15.90:0/1843943344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febe80cedc0 0x7febe80cf250 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7febe80b4f60 tx=0x7febe002a610 comp rx=0 tx=0).stop 2024-04-25T01:17:22.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.935+0000 7febefc00640 1 -- 172.21.15.90:0/1843943344 shutdown_connections 2024-04-25T01:17:22.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.935+0000 7febefc00640 1 --2- 172.21.15.90:0/1843943344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febe80cedc0 0x7febe80cf250 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.935+0000 7febefc00640 1 --2- 172.21.15.90:0/1843943344 >> v2:172.21.15.103:3300/0 conn(0x7febe815fc40 0x7febe81600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.935+0000 7febefc00640 1 --2- 172.21.15.90:0/1843943344 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febe80722e0 0x7febe806ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.935+0000 7febefc00640 1 -- 172.21.15.90:0/1843943344 >> 172.21.15.90:0/1843943344 conn(0x7febe806b720 msgr2=0x7febe80b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:22.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.935+0000 7febefc00640 1 -- 172.21.15.90:0/1843943344 shutdown_connections 2024-04-25T01:17:22.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.936+0000 7febefc00640 1 -- 172.21.15.90:0/1843943344 wait complete. 2024-04-25T01:17:22.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.937+0000 7febefc00640 1 Processor -- start 2024-04-25T01:17:22.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.937+0000 7febefc00640 1 -- start start 2024-04-25T01:17:22.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.938+0000 7febefc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febe80722e0 0x7febe8148e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:22.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.938+0000 7febefc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febe80cedc0 0x7febe80b2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:22.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.938+0000 7febefc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febe815fc40 0x7febe815e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:22.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.938+0000 7febefc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7febe806d3f0 con 0x7febe80cedc0 2024-04-25T01:17:22.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.938+0000 7febefc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7febe806e410 con 0x7febe815fc40 2024-04-25T01:17:22.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.938+0000 7febefc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7febe806d6d0 con 0x7febe80722e0 2024-04-25T01:17:22.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.938+0000 7febe7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febe80cedc0 0x7febe80b2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:22.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.938+0000 7febe7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febe80cedc0 0x7febe80b2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36652/0 (socket says 172.21.15.90:36652) 2024-04-25T01:17:22.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.938+0000 7febe7e00640 1 -- 172.21.15.90:0/517218112 learned_addr learned my addr 172.21.15.90:0/517218112 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:22.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.939+0000 7febedc00640 1 --2- 172.21.15.90:0/517218112 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febe815fc40 0x7febe815e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:22.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.939+0000 7febed200640 1 --2- 172.21.15.90:0/517218112 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febe80722e0 0x7febe8148e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:22.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.939+0000 7febe7e00640 1 -- 172.21.15.90:0/517218112 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febe80722e0 msgr2=0x7febe8148e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:22.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.939+0000 7febe7e00640 1 --2- 172.21.15.90:0/517218112 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febe80722e0 0x7febe8148e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.939+0000 7febe7e00640 1 -- 172.21.15.90:0/517218112 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febe815fc40 msgr2=0x7febe815e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:22.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.939+0000 7febe7e00640 1 --2- 172.21.15.90:0/517218112 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febe815fc40 0x7febe815e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:22.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.939+0000 7febe7e00640 1 -- 172.21.15.90:0/517218112 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febe814d870 con 0x7febe80cedc0 2024-04-25T01:17:22.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.939+0000 7febed200640 1 --2- 172.21.15.90:0/517218112 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febe80722e0 0x7febe8148e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:22.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.939+0000 7febedc00640 1 --2- 172.21.15.90:0/517218112 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febe815fc40 0x7febe815e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:22.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.939+0000 7febe7e00640 1 --2- 172.21.15.90:0/517218112 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febe80cedc0 0x7febe80b2930 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7febd400d620 tx=0x7febd400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:22.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.939+0000 7febe4c00640 1 -- 172.21.15.90:0/517218112 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd400b620 con 0x7febe80cedc0 2024-04-25T01:17:22.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.939+0000 7febe4c00640 1 -- 172.21.15.90:0/517218112 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febd4019070 con 0x7febe80cedc0 2024-04-25T01:17:22.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.939+0000 7febefc00640 1 -- 172.21.15.90:0/517218112 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febe81501e0 con 0x7febe80cedc0 2024-04-25T01:17:22.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.940+0000 7febe4c00640 1 -- 172.21.15.90:0/517218112 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd400ba00 con 0x7febe80cedc0 2024-04-25T01:17:22.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.940+0000 7febefc00640 1 -- 172.21.15.90:0/517218112 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febe814db10 con 0x7febe80cedc0 2024-04-25T01:17:22.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.941+0000 7febefc00640 1 -- 172.21.15.90:0/517218112 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febe8068fb0 con 0x7febe80cedc0 2024-04-25T01:17:22.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.942+0000 7febe4c00640 1 -- 172.21.15.90:0/517218112 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7febd400bba0 con 0x7febe80cedc0 2024-04-25T01:17:22.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.943+0000 7febe4c00640 1 --2- 172.21.15.90:0/517218112 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7febc806be60 0x7febc806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:22.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.944+0000 7febe4c00640 1 -- 172.21.15.90:0/517218112 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7febd408b5c0 con 0x7febe80cedc0 2024-04-25T01:17:22.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.944+0000 7febed200640 1 --2- 172.21.15.90:0/517218112 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7febc806be60 0x7febc806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:22.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.945+0000 7febed200640 1 --2- 172.21.15.90:0/517218112 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7febc806be60 0x7febc806e320 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7febdc006fd0 tx=0x7febdc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:22.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:22.948+0000 7febe4c00640 1 -- 172.21.15.90:0/517218112 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7febd4059bf0 con 0x7febe80cedc0 2024-04-25T01:17:23.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.146+0000 7febefc00640 1 -- 172.21.15.90:0/517218112 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7febe8072c90 con 0x7febe80cedc0 2024-04-25T01:17:23.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.147+0000 7febe4c00640 1 -- 172.21.15.90:0/517218112 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) ==== 93+0+10 (secure 0 0 0) 0x7febd40594f0 con 0x7febe80cedc0 2024-04-25T01:17:23.132 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T01:17:23.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.150+0000 7febefc00640 1 -- 172.21.15.90:0/517218112 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7febc806be60 msgr2=0x7febc806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:23.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.150+0000 7febefc00640 1 --2- 172.21.15.90:0/517218112 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7febc806be60 0x7febc806e320 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7febdc006fd0 tx=0x7febdc008040 comp rx=0 tx=0).stop 2024-04-25T01:17:23.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.150+0000 7febefc00640 1 -- 172.21.15.90:0/517218112 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febe80cedc0 msgr2=0x7febe80b2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:23.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.150+0000 7febefc00640 1 --2- 172.21.15.90:0/517218112 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febe80cedc0 0x7febe80b2930 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7febd400d620 tx=0x7febd400dae0 comp rx=0 tx=0).stop 2024-04-25T01:17:23.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.150+0000 7febefc00640 1 -- 172.21.15.90:0/517218112 shutdown_connections 2024-04-25T01:17:23.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.150+0000 7febefc00640 1 --2- 172.21.15.90:0/517218112 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7febc806be60 0x7febc806e320 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:23.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.150+0000 7febefc00640 1 --2- 172.21.15.90:0/517218112 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febe815fc40 0x7febe815e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:23.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.150+0000 7febefc00640 1 --2- 172.21.15.90:0/517218112 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febe80cedc0 0x7febe80b2930 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:23.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.150+0000 7febefc00640 1 --2- 172.21.15.90:0/517218112 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7febe80722e0 0x7febe8148e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:23.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.150+0000 7febefc00640 1 -- 172.21.15.90:0/517218112 >> 172.21.15.90:0/517218112 conn(0x7febe806b720 msgr2=0x7febe8074620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:23.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.150+0000 7febefc00640 1 -- 172.21.15.90:0/517218112 shutdown_connections 2024-04-25T01:17:23.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.150+0000 7febefc00640 1 -- 172.21.15.90:0/517218112 wait complete. 2024-04-25T01:17:23.154 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:17:23.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.360+0000 7f3128a00640 1 Processor -- start 2024-04-25T01:17:23.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.360+0000 7f3128a00640 1 -- start start 2024-04-25T01:17:23.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.362+0000 7f3128a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f31240722e0 0x7f312406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:23.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.362+0000 7f3128a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f312415fc40 0x7f31241600c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:23.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.362+0000 7f3128a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f312406e410 con 0x7f31240cedc0 2024-04-25T01:17:23.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.362+0000 7f3128a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f3124160aa0 con 0x7f312415fc40 2024-04-25T01:17:23.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.362+0000 7f3128a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f312406da40 con 0x7f31240722e0 2024-04-25T01:17:23.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.362+0000 7f3121400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f312415fc40 0x7f31241600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:23.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.362+0000 7f3121400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f312415fc40 0x7f31241600c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36660/0 (socket says 172.21.15.90:36660) 2024-04-25T01:17:23.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.362+0000 7f3121400640 1 -- 172.21.15.90:0/1160297587 learned_addr learned my addr 172.21.15.90:0/1160297587 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:23.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.363+0000 7f3121400640 1 -- 172.21.15.90:0/1160297587 >> v2:172.21.15.103:3300/0 conn(0x7f31240722e0 msgr2=0x7f312406ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:23.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.363+0000 7f3121400640 1 --2- 172.21.15.90:0/1160297587 >> v2:172.21.15.103:3300/0 conn(0x7f31240722e0 0x7f312406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:23.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.363+0000 7f3121400640 1 -- 172.21.15.90:0/1160297587 >> v1:172.21.15.103:6789/0 conn(0x7f31240cedc0 legacy=0x7f31240cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:23.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.363+0000 7f3121400640 1 -- 172.21.15.90:0/1160297587 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f312406d110 con 0x7f312415fc40 2024-04-25T01:17:23.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.364+0000 7f3121400640 1 --2- 172.21.15.90:0/1160297587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f312415fc40 0x7f31241600c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f31240727b0 tx=0x7f311002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f830d2a9b9edb3dc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:23.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.364+0000 7f3120a00640 1 -- 172.21.15.90:0/1160297587 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3110037070 con 0x7f312415fc40 2024-04-25T01:17:23.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.364+0000 7f3120a00640 1 -- 172.21.15.90:0/1160297587 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3110032440 con 0x7f312415fc40 2024-04-25T01:17:23.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.364+0000 7f3120a00640 1 -- 172.21.15.90:0/1160297587 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3110032810 con 0x7f312415fc40 2024-04-25T01:17:23.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.365+0000 7f3128a00640 1 -- 172.21.15.90:0/1160297587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f312415fc40 msgr2=0x7f31241600c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:23.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.365+0000 7f3128a00640 1 --2- 172.21.15.90:0/1160297587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f312415fc40 0x7f31241600c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f31240727b0 tx=0x7f311002a1a0 comp rx=0 tx=0).stop 2024-04-25T01:17:23.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.365+0000 7f3128a00640 1 -- 172.21.15.90:0/1160297587 shutdown_connections 2024-04-25T01:17:23.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.365+0000 7f3128a00640 1 --2- 172.21.15.90:0/1160297587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f312415fc40 0x7f31241600c0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:23.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.365+0000 7f3128a00640 1 --2- 172.21.15.90:0/1160297587 >> v2:172.21.15.103:3300/0 conn(0x7f31240722e0 0x7f312406ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:23.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.365+0000 7f3128a00640 1 -- 172.21.15.90:0/1160297587 >> 172.21.15.90:0/1160297587 conn(0x7f312406b720 msgr2=0x7f31240b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:23.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.365+0000 7f3128a00640 1 -- 172.21.15.90:0/1160297587 shutdown_connections 2024-04-25T01:17:23.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.365+0000 7f3128a00640 1 -- 172.21.15.90:0/1160297587 wait complete. 2024-04-25T01:17:23.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.367+0000 7f3128a00640 1 Processor -- start 2024-04-25T01:17:23.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.367+0000 7f3128a00640 1 -- start start 2024-04-25T01:17:23.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.368+0000 7f3128a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31240722e0 0x7f3124143530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:23.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.368+0000 7f3128a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31240cedc0 0x7f31241e9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:23.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.368+0000 7f3128a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f312415fc40 0x7f31241ea350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:23.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.368+0000 7f3128a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f312406d6d0 con 0x7f312415fc40 2024-04-25T01:17:23.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.368+0000 7f3128a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f312406cec0 con 0x7f31240cedc0 2024-04-25T01:17:23.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.368+0000 7f3128a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3124070b50 con 0x7f31240722e0 2024-04-25T01:17:23.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.369+0000 7f3122800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f312415fc40 0x7f31241ea350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:23.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.369+0000 7f3122800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f312415fc40 0x7f31241ea350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36664/0 (socket says 172.21.15.90:36664) 2024-04-25T01:17:23.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.369+0000 7f3122800640 1 -- 172.21.15.90:0/2720446680 learned_addr learned my addr 172.21.15.90:0/2720446680 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:23.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.369+0000 7f3121e00640 1 --2- 172.21.15.90:0/2720446680 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31240722e0 0x7f3124143530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:23.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.369+0000 7f3121400640 1 --2- 172.21.15.90:0/2720446680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31240cedc0 0x7f31241e9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:23.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.369+0000 7f3122800640 1 -- 172.21.15.90:0/2720446680 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31240722e0 msgr2=0x7f3124143530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:23.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.369+0000 7f3122800640 1 --2- 172.21.15.90:0/2720446680 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31240722e0 0x7f3124143530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:23.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.369+0000 7f3122800640 1 -- 172.21.15.90:0/2720446680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31240cedc0 msgr2=0x7f31241e9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:23.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.369+0000 7f3122800640 1 --2- 172.21.15.90:0/2720446680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31240cedc0 0x7f31241e9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:23.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.369+0000 7f3122800640 1 -- 172.21.15.90:0/2720446680 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31241e6cf0 con 0x7f312415fc40 2024-04-25T01:17:23.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.369+0000 7f3122800640 1 --2- 172.21.15.90:0/2720446680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f312415fc40 0x7f31241ea350 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f311c00c780 tx=0x7f311c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:23.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.370+0000 7f311a000640 1 -- 172.21.15.90:0/2720446680 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311c017070 con 0x7f312415fc40 2024-04-25T01:17:23.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.370+0000 7f311a000640 1 -- 172.21.15.90:0/2720446680 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f311c003da0 con 0x7f312415fc40 2024-04-25T01:17:23.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.370+0000 7f311a000640 1 -- 172.21.15.90:0/2720446680 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311c0052b0 con 0x7f312415fc40 2024-04-25T01:17:23.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.370+0000 7f3128a00640 1 -- 172.21.15.90:0/2720446680 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31241e6fd0 con 0x7f312415fc40 2024-04-25T01:17:23.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.370+0000 7f3128a00640 1 -- 172.21.15.90:0/2720446680 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31241e7870 con 0x7f312415fc40 2024-04-25T01:17:23.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.371+0000 7f3128a00640 1 -- 172.21.15.90:0/2720446680 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31241e71a0 con 0x7f312415fc40 2024-04-25T01:17:23.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.372+0000 7f311a000640 1 -- 172.21.15.90:0/2720446680 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f311c005450 con 0x7f312415fc40 2024-04-25T01:17:23.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.373+0000 7f311a000640 1 --2- 172.21.15.90:0/2720446680 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f310406be30 0x7f310406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:23.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.373+0000 7f311a000640 1 -- 172.21.15.90:0/2720446680 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f311c08a7c0 con 0x7f312415fc40 2024-04-25T01:17:23.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.373+0000 7f3121e00640 1 --2- 172.21.15.90:0/2720446680 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f310406be30 0x7f310406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:23.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.374+0000 7f3121e00640 1 --2- 172.21.15.90:0/2720446680 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f310406be30 0x7f310406e2f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3114009890 tx=0x7f3114008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:23.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.377+0000 7f311a000640 1 -- 172.21.15.90:0/2720446680 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f311c058d60 con 0x7f312415fc40 2024-04-25T01:17:23.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.574+0000 7f3128a00640 1 -- 172.21.15.90:0/2720446680 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f31240726c0 con 0x7f312415fc40 2024-04-25T01:17:23.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.575+0000 7f311a000640 1 -- 172.21.15.90:0/2720446680 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) ==== 104+0+11 (secure 0 0 0) 0x7f31240726c0 con 0x7f312415fc40 2024-04-25T01:17:23.559 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:17:23.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.577+0000 7f3128a00640 1 -- 172.21.15.90:0/2720446680 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f310406be30 msgr2=0x7f310406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:23.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.577+0000 7f3128a00640 1 --2- 172.21.15.90:0/2720446680 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f310406be30 0x7f310406e2f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3114009890 tx=0x7f3114008040 comp rx=0 tx=0).stop 2024-04-25T01:17:23.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.577+0000 7f3128a00640 1 -- 172.21.15.90:0/2720446680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f312415fc40 msgr2=0x7f31241ea350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:23.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.577+0000 7f3128a00640 1 --2- 172.21.15.90:0/2720446680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f312415fc40 0x7f31241ea350 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f311c00c780 tx=0x7f311c00cc40 comp rx=0 tx=0).stop 2024-04-25T01:17:23.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.578+0000 7f3128a00640 1 -- 172.21.15.90:0/2720446680 shutdown_connections 2024-04-25T01:17:23.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.578+0000 7f3128a00640 1 --2- 172.21.15.90:0/2720446680 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f310406be30 0x7f310406e2f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:23.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.578+0000 7f3128a00640 1 --2- 172.21.15.90:0/2720446680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f312415fc40 0x7f31241ea350 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:23.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.578+0000 7f3128a00640 1 --2- 172.21.15.90:0/2720446680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31240cedc0 0x7f31241e9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:23.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.578+0000 7f3128a00640 1 --2- 172.21.15.90:0/2720446680 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f31240722e0 0x7f3124143530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:23.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.578+0000 7f3128a00640 1 -- 172.21.15.90:0/2720446680 >> 172.21.15.90:0/2720446680 conn(0x7f312406b720 msgr2=0x7f31240af7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:23.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.578+0000 7f3128a00640 1 -- 172.21.15.90:0/2720446680 shutdown_connections 2024-04-25T01:17:23.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.578+0000 7f3128a00640 1 -- 172.21.15.90:0/2720446680 wait complete. 2024-04-25T01:17:23.581 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:17:23.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.801+0000 7efdf5a00640 1 Processor -- start 2024-04-25T01:17:23.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.801+0000 7efdf5a00640 1 -- start start 2024-04-25T01:17:23.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.803+0000 7efdf5a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7efdf006b120 0x7efdf00af290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:23.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.803+0000 7efdf5a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7efdf00b2790 0x7efdf006b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:23.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.803+0000 7efdf5a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdf00d01b0 con 0x7efdf00ccbb0 2024-04-25T01:17:23.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.803+0000 7efdf5a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7efdf00b2bf0 con 0x7efdf006b120 2024-04-25T01:17:23.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.803+0000 7efdf5a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7efdf00d0470 con 0x7efdf00b2790 2024-04-25T01:17:23.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.803+0000 7efdefe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7efdf006b120 0x7efdf00af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:23.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.803+0000 7efdefe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7efdf006b120 0x7efdf00af290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36672/0 (socket says 172.21.15.90:36672) 2024-04-25T01:17:23.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.803+0000 7efdefe00640 1 -- 172.21.15.90:0/1552826708 learned_addr learned my addr 172.21.15.90:0/1552826708 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:23.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.803+0000 7efdefe00640 1 -- 172.21.15.90:0/1552826708 >> v2:172.21.15.103:3300/0 conn(0x7efdf00b2790 msgr2=0x7efdf006b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:23.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.803+0000 7efdefe00640 1 --2- 172.21.15.90:0/1552826708 >> v2:172.21.15.103:3300/0 conn(0x7efdf00b2790 0x7efdf006b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:23.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.803+0000 7efdefe00640 1 -- 172.21.15.90:0/1552826708 >> v1:172.21.15.103:6789/0 conn(0x7efdf00ccbb0 legacy=0x7efdf00cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:23.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.803+0000 7efdefe00640 1 -- 172.21.15.90:0/1552826708 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdf00d0750 con 0x7efdf006b120 2024-04-25T01:17:23.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.804+0000 7efdefe00640 1 --2- 172.21.15.90:0/1552826708 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efdf006b120 0x7efdf00af290 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7efdf0071710 tx=0x7efde002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4bdc9a03d0bde6c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:23.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.805+0000 7efdef400640 1 -- 172.21.15.90:0/1552826708 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde0037070 con 0x7efdf006b120 2024-04-25T01:17:23.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.805+0000 7efdef400640 1 -- 172.21.15.90:0/1552826708 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efde0032440 con 0x7efdf006b120 2024-04-25T01:17:23.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.805+0000 7efdef400640 1 -- 172.21.15.90:0/1552826708 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde0032810 con 0x7efdf006b120 2024-04-25T01:17:23.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.805+0000 7efdf5a00640 1 -- 172.21.15.90:0/1552826708 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efdf006b120 msgr2=0x7efdf00af290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:23.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.805+0000 7efdf5a00640 1 --2- 172.21.15.90:0/1552826708 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efdf006b120 0x7efdf00af290 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7efdf0071710 tx=0x7efde002a1a0 comp rx=0 tx=0).stop 2024-04-25T01:17:23.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.805+0000 7efdf5a00640 1 -- 172.21.15.90:0/1552826708 shutdown_connections 2024-04-25T01:17:23.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.805+0000 7efdf5a00640 1 --2- 172.21.15.90:0/1552826708 >> v2:172.21.15.103:3300/0 conn(0x7efdf00b2790 0x7efdf006b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:23.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.805+0000 7efdf5a00640 1 --2- 172.21.15.90:0/1552826708 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efdf006b120 0x7efdf00af290 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:23.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.805+0000 7efdf5a00640 1 -- 172.21.15.90:0/1552826708 >> 172.21.15.90:0/1552826708 conn(0x7efdf00717a0 msgr2=0x7efdf00b4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:23.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.805+0000 7efdf5a00640 1 -- 172.21.15.90:0/1552826708 shutdown_connections 2024-04-25T01:17:23.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.806+0000 7efdf5a00640 1 -- 172.21.15.90:0/1552826708 wait complete. 2024-04-25T01:17:23.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.807+0000 7efdf5a00640 1 Processor -- start 2024-04-25T01:17:23.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.807+0000 7efdf5a00640 1 -- start start 2024-04-25T01:17:23.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.808+0000 7efdf5a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efdf006b120 0x7efdf01f3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:23.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.808+0000 7efdf5a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdf00b2790 0x7efdf00b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:23.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.808+0000 7efdf5a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efdf00ccbb0 0x7efdf00b1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:23.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.808+0000 7efdf5a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7efdf015e460 con 0x7efdf00ccbb0 2024-04-25T01:17:23.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.808+0000 7efdf5a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efdf006ba40 con 0x7efdf00b2790 2024-04-25T01:17:23.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.808+0000 7efdf5a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efdf00cd750 con 0x7efdf006b120 2024-04-25T01:17:23.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.809+0000 7efdf5000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efdf00ccbb0 0x7efdf00b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:23.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.809+0000 7efdf5000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efdf00ccbb0 0x7efdf00b1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36680/0 (socket says 172.21.15.90:36680) 2024-04-25T01:17:23.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.809+0000 7efdf5000640 1 -- 172.21.15.90:0/3698300831 learned_addr learned my addr 172.21.15.90:0/3698300831 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:23.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.809+0000 7efde7e00640 1 --2- 172.21.15.90:0/3698300831 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdf00b2790 0x7efdf00b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:23.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.809+0000 7efdefe00640 1 --2- 172.21.15.90:0/3698300831 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efdf006b120 0x7efdf01f3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:23.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.809+0000 7efdf5000640 1 -- 172.21.15.90:0/3698300831 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efdf006b120 msgr2=0x7efdf01f3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:23.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.809+0000 7efdf5000640 1 --2- 172.21.15.90:0/3698300831 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efdf006b120 0x7efdf01f3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:23.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.809+0000 7efdf5000640 1 -- 172.21.15.90:0/3698300831 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdf00b2790 msgr2=0x7efdf00b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:23.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.809+0000 7efdf5000640 1 --2- 172.21.15.90:0/3698300831 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdf00b2790 0x7efdf00b11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:23.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.809+0000 7efdf5000640 1 -- 172.21.15.90:0/3698300831 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdf01f3e50 con 0x7efdf00ccbb0 2024-04-25T01:17:23.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.809+0000 7efdf5000640 1 --2- 172.21.15.90:0/3698300831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efdf00ccbb0 0x7efdf00b1710 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7efde800b9c0 tx=0x7efde800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:23.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.810+0000 7efdecc00640 1 -- 172.21.15.90:0/3698300831 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde8019070 con 0x7efdf00ccbb0 2024-04-25T01:17:23.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.810+0000 7efdecc00640 1 -- 172.21.15.90:0/3698300831 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efde800cba0 con 0x7efdf00ccbb0 2024-04-25T01:17:23.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.810+0000 7efdf5a00640 1 -- 172.21.15.90:0/3698300831 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdf015fb90 con 0x7efdf00ccbb0 2024-04-25T01:17:23.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.810+0000 7efdecc00640 1 -- 172.21.15.90:0/3698300831 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde8014440 con 0x7efdf00ccbb0 2024-04-25T01:17:23.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.810+0000 7efdf5a00640 1 -- 172.21.15.90:0/3698300831 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdf0160150 con 0x7efdf00ccbb0 2024-04-25T01:17:23.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.811+0000 7efdf5a00640 1 -- 172.21.15.90:0/3698300831 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdf0075350 con 0x7efdf00ccbb0 2024-04-25T01:17:23.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.812+0000 7efdecc00640 1 -- 172.21.15.90:0/3698300831 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7efde801c070 con 0x7efdf00ccbb0 2024-04-25T01:17:23.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.813+0000 7efdecc00640 1 --2- 172.21.15.90:0/3698300831 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7efdcc06be30 0x7efdcc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:23.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.813+0000 7efdecc00640 1 -- 172.21.15.90:0/3698300831 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7efde808af10 con 0x7efdf00ccbb0 2024-04-25T01:17:23.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.813+0000 7efdefe00640 1 --2- 172.21.15.90:0/3698300831 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7efdcc06be30 0x7efdcc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:23.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.814+0000 7efdefe00640 1 --2- 172.21.15.90:0/3698300831 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7efdcc06be30 0x7efdcc06e2f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7efde0034000 tx=0x7efde002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:23.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:23.818+0000 7efdecc00640 1 -- 172.21.15.90:0/3698300831 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7efde80595e0 con 0x7efdf00ccbb0 2024-04-25T01:17:23.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.013+0000 7efdf5a00640 1 -- 172.21.15.90:0/3698300831 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7efdf00b1f20 con 0x7efdf00ccbb0 2024-04-25T01:17:23.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.014+0000 7efdecc00640 1 -- 172.21.15.90:0/3698300831 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) ==== 100+0+11 (secure 0 0 0) 0x7efde8058ee0 con 0x7efdf00ccbb0 2024-04-25T01:17:23.999 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-25T01:17:24.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.016+0000 7efdf5a00640 1 -- 172.21.15.90:0/3698300831 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7efdcc06be30 msgr2=0x7efdcc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:24.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.016+0000 7efdf5a00640 1 --2- 172.21.15.90:0/3698300831 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7efdcc06be30 0x7efdcc06e2f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7efde0034000 tx=0x7efde002e000 comp rx=0 tx=0).stop 2024-04-25T01:17:24.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.017+0000 7efdf5a00640 1 -- 172.21.15.90:0/3698300831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efdf00ccbb0 msgr2=0x7efdf00b1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:24.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.017+0000 7efdf5a00640 1 --2- 172.21.15.90:0/3698300831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efdf00ccbb0 0x7efdf00b1710 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7efde800b9c0 tx=0x7efde800be80 comp rx=0 tx=0).stop 2024-04-25T01:17:24.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.017+0000 7efdf5a00640 1 -- 172.21.15.90:0/3698300831 shutdown_connections 2024-04-25T01:17:24.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.017+0000 7efdf5a00640 1 --2- 172.21.15.90:0/3698300831 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7efdcc06be30 0x7efdcc06e2f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.017+0000 7efdf5a00640 1 --2- 172.21.15.90:0/3698300831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efdf00ccbb0 0x7efdf00b1710 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.017+0000 7efdf5a00640 1 --2- 172.21.15.90:0/3698300831 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdf00b2790 0x7efdf00b11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.017+0000 7efdf5a00640 1 --2- 172.21.15.90:0/3698300831 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efdf006b120 0x7efdf01f3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.017+0000 7efdf5a00640 1 -- 172.21.15.90:0/3698300831 >> 172.21.15.90:0/3698300831 conn(0x7efdf00717a0 msgr2=0x7efdf0072650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:24.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.017+0000 7efdf5a00640 1 -- 172.21.15.90:0/3698300831 shutdown_connections 2024-04-25T01:17:24.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.017+0000 7efdf5a00640 1 -- 172.21.15.90:0/3698300831 wait complete. 2024-04-25T01:17:24.019 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:17:24.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.233+0000 7f74c3e00640 1 Processor -- start 2024-04-25T01:17:24.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.233+0000 7f74c3e00640 1 -- start start 2024-04-25T01:17:24.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.235+0000 7f74c3e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f74c40713e0 0x7f74c406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:24.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.235+0000 7f74c3e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74c40b4dc0 con 0x7f74c40cf510 2024-04-25T01:17:24.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.235+0000 7f74c3e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74c4071ac0 con 0x7f74c40b44b0 2024-04-25T01:17:24.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.235+0000 7f74c3e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f74c40b2a90 con 0x7f74c40713e0 2024-04-25T01:17:24.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.235+0000 7f74c2a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f74c40cf510 0x7f74c40cf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50492/0 (socket says 172.21.15.90:50492) 2024-04-25T01:17:24.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.235+0000 7f74c2a00640 1 -- 172.21.15.90:0/2986444495 learned_addr learned my addr 172.21.15.90:0/2986444495 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:24.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.236+0000 7f74c3400640 1 --2- 172.21.15.90:0/2986444495 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f74c40713e0 0x7f74c406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:24.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.236+0000 7f74c2000640 1 -- 172.21.15.90:0/2986444495 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1957922764 0 0) 0x7f74b4001110 con 0x7f74c40cf510 2024-04-25T01:17:24.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.236+0000 7f74c2000640 1 -- 172.21.15.90:0/2986444495 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74a8003520 con 0x7f74c40cf510 2024-04-25T01:17:24.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.236+0000 7f74c2000640 1 -- 172.21.15.90:0/2986444495 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3459981539 0 0) 0x7f74a8003520 con 0x7f74c40cf510 2024-04-25T01:17:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.236+0000 7f74c2000640 1 -- 172.21.15.90:0/2986444495 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f74c40713e0 msgr2=0x7f74c406d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.236+0000 7f74c2000640 1 --2- 172.21.15.90:0/2986444495 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f74c40713e0 0x7f74c406d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.236+0000 7f74c2000640 1 -- 172.21.15.90:0/2986444495 >> v1:172.21.15.103:6789/0 conn(0x7f74c40b44b0 legacy=0x7f74c406d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.236+0000 7f74c2000640 1 -- 172.21.15.90:0/2986444495 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74c40b3630 con 0x7f74c40cf510 2024-04-25T01:17:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.236+0000 7f74c2000640 1 -- 172.21.15.90:0/2986444495 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f74b4001840 con 0x7f74c40cf510 2024-04-25T01:17:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.236+0000 7f74c2000640 1 -- 172.21.15.90:0/2986444495 >> v1:172.21.15.90:6789/0 conn(0x7f74c40cf510 legacy=0x7f74c40cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.236+0000 7f74c2000640 1 --2- 172.21.15.90:0/2986444495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74a80039e0 0x7f74a8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.236+0000 7f74c2000640 1 --2- 172.21.15.90:0/2986444495 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f74c40713e0 0x7f74a8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.236+0000 7f74c2000640 1 --2- 172.21.15.90:0/2986444495 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74a8004d40 0x7f74a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.236+0000 7f74c2000640 1 -- 172.21.15.90:0/2986444495 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f74a800bc30 con 0x7f74a8004d40 2024-04-25T01:17:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.236+0000 7f74c2000640 1 -- 172.21.15.90:0/2986444495 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f74a800bde0 con 0x7f74a80039e0 2024-04-25T01:17:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.236+0000 7f74c2000640 1 -- 172.21.15.90:0/2986444495 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f74a800bfe0 con 0x7f74c40713e0 2024-04-25T01:17:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.237+0000 7f74c3400640 1 --2- 172.21.15.90:0/2986444495 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74a8004d40 0x7f74a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.237+0000 7f74ba000640 1 --2- 172.21.15.90:0/2986444495 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f74c40713e0 0x7f74a8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.237+0000 7f74c2a00640 1 --2- 172.21.15.90:0/2986444495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74a80039e0 0x7f74a8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:24.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.237+0000 7f74c3400640 1 -- 172.21.15.90:0/2986444495 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f74c40713e0 msgr2=0x7f74a8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:24.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.237+0000 7f74c3400640 1 --2- 172.21.15.90:0/2986444495 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f74c40713e0 0x7f74a8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.237+0000 7f74c3400640 1 -- 172.21.15.90:0/2986444495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74a80039e0 msgr2=0x7f74a8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:24.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.237+0000 7f74c3400640 1 --2- 172.21.15.90:0/2986444495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74a80039e0 0x7f74a8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.237+0000 7f74c3400640 1 -- 172.21.15.90:0/2986444495 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74a8003520 con 0x7f74a8004d40 2024-04-25T01:17:24.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.238+0000 7f74c3400640 1 --2- 172.21.15.90:0/2986444495 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74a8004d40 0x7f74a80051c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f74bc007360 tx=0x7f74a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a54b83376bff60c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:24.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.238+0000 7f74c2000640 1 -- 172.21.15.90:0/2986444495 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74bc009be0 con 0x7f74a8004d40 2024-04-25T01:17:24.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.238+0000 7f74c2000640 1 -- 172.21.15.90:0/2986444495 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74bc036600 con 0x7f74a8004d40 2024-04-25T01:17:24.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.238+0000 7f74c2000640 1 -- 172.21.15.90:0/2986444495 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74bc0369d0 con 0x7f74a8004d40 2024-04-25T01:17:24.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.239+0000 7f74c3e00640 1 -- 172.21.15.90:0/2986444495 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74a8004d40 msgr2=0x7f74a80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:24.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.239+0000 7f74c3e00640 1 --2- 172.21.15.90:0/2986444495 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74a8004d40 0x7f74a80051c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f74bc007360 tx=0x7f74a8005830 comp rx=0 tx=0).stop 2024-04-25T01:17:24.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.239+0000 7f74c3400640 1 -- 172.21.15.90:0/2986444495 reap_dead start 2024-04-25T01:17:24.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.239+0000 7f74c3e00640 1 -- 172.21.15.90:0/2986444495 shutdown_connections 2024-04-25T01:17:24.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.239+0000 7f74c3e00640 1 -- 172.21.15.90:0/2986444495 >> 172.21.15.90:0/2986444495 conn(0x7f74c406b120 msgr2=0x7f74c406b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:24.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.239+0000 7f74c3e00640 1 -- 172.21.15.90:0/2986444495 shutdown_connections 2024-04-25T01:17:24.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.239+0000 7f74c3e00640 1 -- 172.21.15.90:0/2986444495 wait complete. 2024-04-25T01:17:24.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.241+0000 7f74c3e00640 1 Processor -- start 2024-04-25T01:17:24.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.241+0000 7f74c3e00640 1 -- start start 2024-04-25T01:17:24.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.242+0000 7f74c3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74a8004d40 0x7f74c41e4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:24.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.242+0000 7f74c3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74c4150d30 0x7f74c4143290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:24.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.242+0000 7f74c3e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f74c41522d0 0x7f74c4152760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:24.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.242+0000 7f74c3e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f74c40b3bf0 con 0x7f74c4150d30 2024-04-25T01:17:24.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.242+0000 7f74c3e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f74c4071870 con 0x7f74a8004d40 2024-04-25T01:17:24.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.242+0000 7f74c3e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f74c40b3ed0 con 0x7f74c41522d0 2024-04-25T01:17:24.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.242+0000 7f74ba000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74c4150d30 0x7f74c4143290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:24.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.242+0000 7f74ba000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74c4150d30 0x7f74c4143290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36706/0 (socket says 172.21.15.90:36706) 2024-04-25T01:17:24.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.242+0000 7f74ba000640 1 -- 172.21.15.90:0/2520902323 learned_addr learned my addr 172.21.15.90:0/2520902323 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:24.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.242+0000 7f74c2a00640 1 --2- 172.21.15.90:0/2520902323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74a8004d40 0x7f74c41e4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:24.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.242+0000 7f74ba000640 1 -- 172.21.15.90:0/2520902323 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f74c41522d0 msgr2=0x7f74c4152760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:24.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.242+0000 7f74ba000640 1 --2- 172.21.15.90:0/2520902323 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f74c41522d0 0x7f74c4152760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.242+0000 7f74ba000640 1 -- 172.21.15.90:0/2520902323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74a8004d40 msgr2=0x7f74c41e4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:24.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.242+0000 7f74ba000640 1 --2- 172.21.15.90:0/2520902323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74a8004d40 0x7f74c41e4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.242+0000 7f74ba000640 1 -- 172.21.15.90:0/2520902323 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74c41e4bc0 con 0x7f74c4150d30 2024-04-25T01:17:24.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.243+0000 7f74ba000640 1 --2- 172.21.15.90:0/2520902323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74c4150d30 0x7f74c4143290 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f74ac00e4d0 tx=0x7f74ac00e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:24.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.243+0000 7f74bb400640 1 -- 172.21.15.90:0/2520902323 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74ac0052e0 con 0x7f74c4150d30 2024-04-25T01:17:24.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.243+0000 7f74bb400640 1 -- 172.21.15.90:0/2520902323 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74ac018070 con 0x7f74c4150d30 2024-04-25T01:17:24.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.243+0000 7f74c3e00640 1 -- 172.21.15.90:0/2520902323 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74c41437d0 con 0x7f74c4150d30 2024-04-25T01:17:24.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.243+0000 7f74bb400640 1 -- 172.21.15.90:0/2520902323 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74ac003810 con 0x7f74c4150d30 2024-04-25T01:17:24.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.243+0000 7f74c3e00640 1 -- 172.21.15.90:0/2520902323 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74c4151510 con 0x7f74c4150d30 2024-04-25T01:17:24.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.244+0000 7f74c3e00640 1 -- 172.21.15.90:0/2520902323 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74c4075350 con 0x7f74c4150d30 2024-04-25T01:17:24.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.245+0000 7f74bb400640 1 -- 172.21.15.90:0/2520902323 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f74ac0039b0 con 0x7f74c4150d30 2024-04-25T01:17:24.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.246+0000 7f74bb400640 1 --2- 172.21.15.90:0/2520902323 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f74a006be30 0x7f74a006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:24.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.246+0000 7f74bb400640 1 -- 172.21.15.90:0/2520902323 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f74ac08aec0 con 0x7f74c4150d30 2024-04-25T01:17:24.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.246+0000 7f74c2a00640 1 --2- 172.21.15.90:0/2520902323 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f74a006be30 0x7f74a006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:24.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.247+0000 7f74c2a00640 1 --2- 172.21.15.90:0/2520902323 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f74a006be30 0x7f74a006e2f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f74b40012b0 tx=0x7f74b4000ef0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:24.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.250+0000 7f74bb400640 1 -- 172.21.15.90:0/2520902323 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f74ac059460 con 0x7f74c4150d30 2024-04-25T01:17:24.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.445+0000 7f74c3e00640 1 -- 172.21.15.90:0/2520902323 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f74c4072000 con 0x7f74c4150d30 2024-04-25T01:17:24.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.447+0000 7f74bb400640 1 -- 172.21.15.90:0/2520902323 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) ==== 103+0+11 (secure 0 0 0) 0x7f74ac058d60 con 0x7f74c4150d30 2024-04-25T01:17:24.514 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:17:24.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.449+0000 7f74c3e00640 1 -- 172.21.15.90:0/2520902323 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f74a006be30 msgr2=0x7f74a006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:24.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.449+0000 7f74c3e00640 1 --2- 172.21.15.90:0/2520902323 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f74a006be30 0x7f74a006e2f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f74b40012b0 tx=0x7f74b4000ef0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.449+0000 7f74c3e00640 1 -- 172.21.15.90:0/2520902323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74c4150d30 msgr2=0x7f74c4143290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:24.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.449+0000 7f74c3e00640 1 --2- 172.21.15.90:0/2520902323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74c4150d30 0x7f74c4143290 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f74ac00e4d0 tx=0x7f74ac00e990 comp rx=0 tx=0).stop 2024-04-25T01:17:24.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.449+0000 7f74c3e00640 1 -- 172.21.15.90:0/2520902323 shutdown_connections 2024-04-25T01:17:24.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.449+0000 7f74c3e00640 1 --2- 172.21.15.90:0/2520902323 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f74a006be30 0x7f74a006e2f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.449+0000 7f74c3e00640 1 --2- 172.21.15.90:0/2520902323 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f74c41522d0 0x7f74c4152760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.449+0000 7f74c3e00640 1 --2- 172.21.15.90:0/2520902323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74c4150d30 0x7f74c4143290 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.449+0000 7f74c3e00640 1 --2- 172.21.15.90:0/2520902323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74a8004d40 0x7f74c41e4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.449+0000 7f74c3e00640 1 -- 172.21.15.90:0/2520902323 >> 172.21.15.90:0/2520902323 conn(0x7f74c406b120 msgr2=0x7f74c40ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:24.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.449+0000 7f74c3e00640 1 -- 172.21.15.90:0/2520902323 shutdown_connections 2024-04-25T01:17:24.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.449+0000 7f74c3e00640 1 -- 172.21.15.90:0/2520902323 wait complete. 2024-04-25T01:17:24.516 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:17:24.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.735+0000 7f897be00640 1 Processor -- start 2024-04-25T01:17:24.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.735+0000 7f897be00640 1 -- start start 2024-04-25T01:17:24.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.736+0000 7f897be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f897c0cf520 0x7f897c0ae730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:24.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.736+0000 7f897be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f897c06d910 0x7f897c0aec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:24.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.736+0000 7f897be00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f897c06ba90 con 0x7f897c15ad40 2024-04-25T01:17:24.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.736+0000 7f897be00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f897c06b830 con 0x7f897c06d910 2024-04-25T01:17:24.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.736+0000 7f897be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f897c06cb50 con 0x7f897c0cf520 2024-04-25T01:17:24.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.736+0000 7f897b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f897c06d910 0x7f897c0aec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:24.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.736+0000 7f897b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f897c06d910 0x7f897c0aec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36720/0 (socket says 172.21.15.90:36720) 2024-04-25T01:17:24.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.736+0000 7f897b400640 1 -- 172.21.15.90:0/272550428 learned_addr learned my addr 172.21.15.90:0/272550428 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:24.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.737+0000 7f897a000640 1 --2- 172.21.15.90:0/272550428 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f897c0cf520 0x7f897c0ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:24.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.737+0000 7f897b400640 1 -- 172.21.15.90:0/272550428 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f897c0cf520 msgr2=0x7f897c0ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:24.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.737+0000 7f897b400640 1 --2- 172.21.15.90:0/272550428 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f897c0cf520 0x7f897c0ae730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.737+0000 7f897b400640 1 -- 172.21.15.90:0/272550428 >> v1:172.21.15.103:6789/0 conn(0x7f897c15ad40 legacy=0x7f897c06b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:24.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.737+0000 7f897b400640 1 -- 172.21.15.90:0/272550428 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f897c06ce30 con 0x7f897c06d910 2024-04-25T01:17:24.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.737+0000 7f897a000640 1 --2- 172.21.15.90:0/272550428 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f897c0cf520 0x7f897c0ae730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:24.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.738+0000 7f897b400640 1 --2- 172.21.15.90:0/272550428 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897c06d910 0x7f897c0aec70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8974009e80 tx=0x7f897402a750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b95808eaa005a03d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:24.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.738+0000 7f8979600640 1 -- 172.21.15.90:0/272550428 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8974036070 con 0x7f897c06d910 2024-04-25T01:17:24.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.738+0000 7f8979600640 1 -- 172.21.15.90:0/272550428 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8974031600 con 0x7f897c06d910 2024-04-25T01:17:24.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.738+0000 7f8979600640 1 -- 172.21.15.90:0/272550428 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89740319d0 con 0x7f897c06d910 2024-04-25T01:17:24.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.739+0000 7f897be00640 1 -- 172.21.15.90:0/272550428 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897c06d910 msgr2=0x7f897c0aec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:24.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.739+0000 7f897be00640 1 --2- 172.21.15.90:0/272550428 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897c06d910 0x7f897c0aec70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8974009e80 tx=0x7f897402a750 comp rx=0 tx=0).stop 2024-04-25T01:17:24.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.739+0000 7f897be00640 1 -- 172.21.15.90:0/272550428 shutdown_connections 2024-04-25T01:17:24.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.739+0000 7f897be00640 1 --2- 172.21.15.90:0/272550428 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897c06d910 0x7f897c0aec70 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.739+0000 7f897be00640 1 --2- 172.21.15.90:0/272550428 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f897c0cf520 0x7f897c0ae730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.739+0000 7f897be00640 1 -- 172.21.15.90:0/272550428 >> 172.21.15.90:0/272550428 conn(0x7f897c0b1660 msgr2=0x7f897c0b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:24.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.739+0000 7f897be00640 1 -- 172.21.15.90:0/272550428 shutdown_connections 2024-04-25T01:17:24.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.739+0000 7f897be00640 1 -- 172.21.15.90:0/272550428 wait complete. 2024-04-25T01:17:24.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.741+0000 7f897be00640 1 Processor -- start 2024-04-25T01:17:24.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.741+0000 7f897be00640 1 -- start start 2024-04-25T01:17:24.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.742+0000 7f897be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f897c06d910 0x7f897c15e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:24.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.742+0000 7f897be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897c0cf520 0x7f897c151320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:24.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.742+0000 7f897be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f897c15ad40 0x7f897c1520b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:24.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.742+0000 7f897be00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f897c0cd870 con 0x7f897c0cf520 2024-04-25T01:17:24.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.742+0000 7f897be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f897c06d3f0 con 0x7f897c15ad40 2024-04-25T01:17:24.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.742+0000 7f897be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f897c0cd9f0 con 0x7f897c06d910 2024-04-25T01:17:24.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.742+0000 7f897a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897c0cf520 0x7f897c151320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:24.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.742+0000 7f897a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897c0cf520 0x7f897c151320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36722/0 (socket says 172.21.15.90:36722) 2024-04-25T01:17:24.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.742+0000 7f897a000640 1 -- 172.21.15.90:0/103717238 learned_addr learned my addr 172.21.15.90:0/103717238 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:24.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.742+0000 7f897aa00640 1 --2- 172.21.15.90:0/103717238 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f897c06d910 0x7f897c15e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:24.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.742+0000 7f897a000640 1 -- 172.21.15.90:0/103717238 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f897c06d910 msgr2=0x7f897c15e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:24.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.742+0000 7f897a000640 1 --2- 172.21.15.90:0/103717238 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f897c06d910 0x7f897c15e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.742+0000 7f897a000640 1 -- 172.21.15.90:0/103717238 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f897c15ad40 msgr2=0x7f897c1520b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:24.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.742+0000 7f897a000640 1 --2- 172.21.15.90:0/103717238 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f897c15ad40 0x7f897c1520b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.742+0000 7f897a000640 1 -- 172.21.15.90:0/103717238 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f897c15e550 con 0x7f897c0cf520 2024-04-25T01:17:24.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.743+0000 7f897a000640 1 --2- 172.21.15.90:0/103717238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897c0cf520 0x7f897c151320 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f896800c7f0 tx=0x7f896800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:24.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.743+0000 7f8972a00640 1 -- 172.21.15.90:0/103717238 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8968015070 con 0x7f897c0cf520 2024-04-25T01:17:24.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.743+0000 7f8972a00640 1 -- 172.21.15.90:0/103717238 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8968007a30 con 0x7f897c0cf520 2024-04-25T01:17:24.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.743+0000 7f897be00640 1 -- 172.21.15.90:0/103717238 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f897c166590 con 0x7f897c0cf520 2024-04-25T01:17:24.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.743+0000 7f8972a00640 1 -- 172.21.15.90:0/103717238 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8968007e00 con 0x7f897c0cf520 2024-04-25T01:17:24.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.743+0000 7f897be00640 1 -- 172.21.15.90:0/103717238 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f897c166b50 con 0x7f897c0cf520 2024-04-25T01:17:24.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.744+0000 7f897be00640 1 -- 172.21.15.90:0/103717238 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f897c151d90 con 0x7f897c0cf520 2024-04-25T01:17:24.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.745+0000 7f8972a00640 1 -- 172.21.15.90:0/103717238 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8968018070 con 0x7f897c0cf520 2024-04-25T01:17:24.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.746+0000 7f8972a00640 1 --2- 172.21.15.90:0/103717238 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f895c06be30 0x7f895c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:24.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.746+0000 7f8972a00640 1 -- 172.21.15.90:0/103717238 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f896808b130 con 0x7f897c0cf520 2024-04-25T01:17:24.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.746+0000 7f897aa00640 1 --2- 172.21.15.90:0/103717238 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f895c06be30 0x7f895c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:24.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.747+0000 7f897aa00640 1 --2- 172.21.15.90:0/103717238 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f895c06be30 0x7f895c06e2f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f897c0ce770 tx=0x7f896c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:24.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.750+0000 7f8972a00640 1 -- 172.21.15.90:0/103717238 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f897c151d90 con 0x7f897c0cf520 2024-04-25T01:17:24.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.937+0000 7f897be00640 1 -- 172.21.15.90:0/103717238 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f897c075350 con 0x7f897c0cf520 2024-04-25T01:17:24.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.940+0000 7f8972a00640 1 -- 172.21.15.90:0/103717238 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18576 (secure 0 0 0) 0x7f8968059800 con 0x7f897c0cf520 2024-04-25T01:17:24.926 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:17:24.926 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:17:09.346008+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:16:20.094512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:16:21.078268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:16:24.670642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.e","mappings":[{"from":0,"to":7}]},{"pgid":"4.f","mappings":[{"from":7,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:17:24.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.943+0000 7f897be00640 1 -- 172.21.15.90:0/103717238 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f895c06be30 msgr2=0x7f895c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:24.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.943+0000 7f897be00640 1 --2- 172.21.15.90:0/103717238 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f895c06be30 0x7f895c06e2f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f897c0ce770 tx=0x7f896c009040 comp rx=0 tx=0).stop 2024-04-25T01:17:24.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.943+0000 7f897be00640 1 -- 172.21.15.90:0/103717238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897c0cf520 msgr2=0x7f897c151320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:24.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.943+0000 7f897be00640 1 --2- 172.21.15.90:0/103717238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897c0cf520 0x7f897c151320 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f896800c7f0 tx=0x7f896800cbf0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.943+0000 7f897be00640 1 -- 172.21.15.90:0/103717238 shutdown_connections 2024-04-25T01:17:24.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.943+0000 7f897be00640 1 --2- 172.21.15.90:0/103717238 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f895c06be30 0x7f895c06e2f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.943+0000 7f897be00640 1 --2- 172.21.15.90:0/103717238 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f897c15ad40 0x7f897c1520b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.943+0000 7f897be00640 1 --2- 172.21.15.90:0/103717238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897c0cf520 0x7f897c151320 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.943+0000 7f897be00640 1 --2- 172.21.15.90:0/103717238 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f897c06d910 0x7f897c15e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:24.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.943+0000 7f897be00640 1 -- 172.21.15.90:0/103717238 >> 172.21.15.90:0/103717238 conn(0x7f897c0b1660 msgr2=0x7f897c072180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:24.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.943+0000 7f897be00640 1 -- 172.21.15.90:0/103717238 shutdown_connections 2024-04-25T01:17:24.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:24.943+0000 7f897be00640 1 -- 172.21.15.90:0/103717238 wait complete. 2024-04-25T01:17:24.945 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:16:15.894126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:16:20.094512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:16:21.078268+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:16:24.670642+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:17:24.946 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:17:25.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.157+0000 7f2d42e00640 1 Processor -- start 2024-04-25T01:17:25.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.158+0000 7f2d42e00640 1 -- start start 2024-04-25T01:17:25.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.159+0000 7f2d42e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d3c06e280 0x7f2d3c15b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:25.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.159+0000 7f2d42e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2d3c15b780 0x7f2d3c0ae620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:25.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.159+0000 7f2d42e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d3c155510 con 0x7f2d3c155130 2024-04-25T01:17:25.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.159+0000 7f2d42e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f2d3c15bfe0 con 0x7f2d3c15b780 2024-04-25T01:17:25.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.159+0000 7f2d42e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2d3c15d1d0 con 0x7f2d3c06e280 2024-04-25T01:17:25.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.159+0000 7f2d42400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2d3c15b780 0x7f2d3c0ae620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:25.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.159+0000 7f2d41a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f2d3c155130 0x7f2d3c06db70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47252/0 (socket says 172.21.15.90:47252) 2024-04-25T01:17:25.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.159+0000 7f2d41a00640 1 -- 172.21.15.90:0/2804792944 learned_addr learned my addr 172.21.15.90:0/2804792944 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:25.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.160+0000 7f2d41000640 1 --2- 172.21.15.90:0/2804792944 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d3c06e280 0x7f2d3c15b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:25.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.160+0000 7f2d42400640 1 -- 172.21.15.90:0/2804792944 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d3c06e280 msgr2=0x7f2d3c15b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:25.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.160+0000 7f2d42400640 1 --2- 172.21.15.90:0/2804792944 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d3c06e280 0x7f2d3c15b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:25.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.160+0000 7f2d42400640 1 -- 172.21.15.90:0/2804792944 >> v1:172.21.15.90:6789/0 conn(0x7f2d3c155130 legacy=0x7f2d3c06db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:25.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.160+0000 7f2d42400640 1 -- 172.21.15.90:0/2804792944 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d3c071af0 con 0x7f2d3c15b780 2024-04-25T01:17:25.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.160+0000 7f2d41000640 1 --2- 172.21.15.90:0/2804792944 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d3c06e280 0x7f2d3c15b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:25.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.161+0000 7f2d42400640 1 --2- 172.21.15.90:0/2804792944 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d3c15b780 0x7f2d3c0ae620 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2d34009d90 tx=0x7f2d3402a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a52832194f43fcd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:25.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.161+0000 7f2d3be00640 1 -- 172.21.15.90:0/2804792944 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d34031600 con 0x7f2d3c15b780 2024-04-25T01:17:25.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.161+0000 7f2d3be00640 1 -- 172.21.15.90:0/2804792944 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d340317a0 con 0x7f2d3c15b780 2024-04-25T01:17:25.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.161+0000 7f2d3be00640 1 -- 172.21.15.90:0/2804792944 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d34031b70 con 0x7f2d3c15b780 2024-04-25T01:17:25.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.162+0000 7f2d42e00640 1 -- 172.21.15.90:0/2804792944 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d3c15b780 msgr2=0x7f2d3c0ae620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:25.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.162+0000 7f2d42e00640 1 --2- 172.21.15.90:0/2804792944 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d3c15b780 0x7f2d3c0ae620 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2d34009d90 tx=0x7f2d3402a5d0 comp rx=0 tx=0).stop 2024-04-25T01:17:25.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.162+0000 7f2d42e00640 1 -- 172.21.15.90:0/2804792944 shutdown_connections 2024-04-25T01:17:25.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.162+0000 7f2d42e00640 1 --2- 172.21.15.90:0/2804792944 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d3c15b780 0x7f2d3c0ae620 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:25.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.162+0000 7f2d42e00640 1 --2- 172.21.15.90:0/2804792944 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d3c06e280 0x7f2d3c15b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:25.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.162+0000 7f2d42e00640 1 -- 172.21.15.90:0/2804792944 >> 172.21.15.90:0/2804792944 conn(0x7f2d3c06b120 msgr2=0x7f2d3c06b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:25.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.162+0000 7f2d42e00640 1 -- 172.21.15.90:0/2804792944 shutdown_connections 2024-04-25T01:17:25.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.162+0000 7f2d42e00640 1 -- 172.21.15.90:0/2804792944 wait complete. 2024-04-25T01:17:25.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.164+0000 7f2d42e00640 1 Processor -- start 2024-04-25T01:17:25.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.164+0000 7f2d42e00640 1 -- start start 2024-04-25T01:17:25.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.165+0000 7f2d42e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d3c06e280 0x7f2d3c14bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:25.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.165+0000 7f2d42e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d3c155130 0x7f2d3c148720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:25.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.165+0000 7f2d42e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d3c15b780 0x7f2d3c0ce4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:25.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.165+0000 7f2d42e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2d3c0728b0 con 0x7f2d3c155130 2024-04-25T01:17:25.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.165+0000 7f2d42e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2d3c0720b0 con 0x7f2d3c06e280 2024-04-25T01:17:25.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.165+0000 7f2d42e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2d3c072a30 con 0x7f2d3c15b780 2024-04-25T01:17:25.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.165+0000 7f2d41000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d3c155130 0x7f2d3c148720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:25.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.165+0000 7f2d41000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d3c155130 0x7f2d3c148720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33178/0 (socket says 172.21.15.90:33178) 2024-04-25T01:17:25.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.165+0000 7f2d41000640 1 -- 172.21.15.90:0/4069091108 learned_addr learned my addr 172.21.15.90:0/4069091108 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:25.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.165+0000 7f2d41000640 1 -- 172.21.15.90:0/4069091108 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d3c15b780 msgr2=0x7f2d3c0ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:25.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.165+0000 7f2d41000640 1 --2- 172.21.15.90:0/4069091108 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d3c15b780 0x7f2d3c0ce4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:25.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.165+0000 7f2d41000640 1 -- 172.21.15.90:0/4069091108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d3c06e280 msgr2=0x7f2d3c14bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:25.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.165+0000 7f2d41a00640 1 --2- 172.21.15.90:0/4069091108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d3c06e280 0x7f2d3c14bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:25.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.166+0000 7f2d41000640 1 --2- 172.21.15.90:0/4069091108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d3c06e280 0x7f2d3c14bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:25.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.166+0000 7f2d41000640 1 -- 172.21.15.90:0/4069091108 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d3c0ced00 con 0x7f2d3c155130 2024-04-25T01:17:25.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.166+0000 7f2d41a00640 1 --2- 172.21.15.90:0/4069091108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d3c06e280 0x7f2d3c14bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:25.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.166+0000 7f2d41000640 1 --2- 172.21.15.90:0/4069091108 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d3c155130 0x7f2d3c148720 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2d2800c820 tx=0x7f2d2800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:25.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.166+0000 7f2d39600640 1 -- 172.21.15.90:0/4069091108 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d28015070 con 0x7f2d3c155130 2024-04-25T01:17:25.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.166+0000 7f2d39600640 1 -- 172.21.15.90:0/4069091108 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d28007a30 con 0x7f2d3c155130 2024-04-25T01:17:25.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.166+0000 7f2d39600640 1 -- 172.21.15.90:0/4069091108 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d28007de0 con 0x7f2d3c155130 2024-04-25T01:17:25.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.166+0000 7f2d42e00640 1 -- 172.21.15.90:0/4069091108 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d3c0cefe0 con 0x7f2d3c155130 2024-04-25T01:17:25.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.166+0000 7f2d42e00640 1 -- 172.21.15.90:0/4069091108 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d3c0cf5a0 con 0x7f2d3c155130 2024-04-25T01:17:25.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.168+0000 7f2d42e00640 1 -- 172.21.15.90:0/4069091108 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d3c075350 con 0x7f2d3c155130 2024-04-25T01:17:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.169+0000 7f2d39600640 1 -- 172.21.15.90:0/4069091108 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2d28018070 con 0x7f2d3c155130 2024-04-25T01:17:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.169+0000 7f2d39600640 1 --2- 172.21.15.90:0/4069091108 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2d1c06be30 0x7f2d1c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.169+0000 7f2d39600640 1 -- 172.21.15.90:0/4069091108 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f2d2808b0f0 con 0x7f2d3c155130 2024-04-25T01:17:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.170+0000 7f2d41a00640 1 --2- 172.21.15.90:0/4069091108 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2d1c06be30 0x7f2d1c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.170+0000 7f2d41a00640 1 --2- 172.21.15.90:0/4069091108 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2d1c06be30 0x7f2d1c06e2f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2d30000f60 tx=0x7f2d30009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:25.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.174+0000 7f2d39600640 1 -- 172.21.15.90:0/4069091108 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2d28059730 con 0x7f2d3c155130 2024-04-25T01:17:25.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.361+0000 7f2d42e00640 1 -- 172.21.15.90:0/4069091108 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2d3c15cf90 con 0x7f2d3c155130 2024-04-25T01:17:25.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.362+0000 7f2d39600640 1 -- 172.21.15.90:0/4069091108 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) ==== 93+0+10 (secure 0 0 0) 0x7f2d28059030 con 0x7f2d3c155130 2024-04-25T01:17:25.346 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T01:17:25.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.364+0000 7f2d42e00640 1 -- 172.21.15.90:0/4069091108 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2d1c06be30 msgr2=0x7f2d1c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:25.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.364+0000 7f2d42e00640 1 --2- 172.21.15.90:0/4069091108 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2d1c06be30 0x7f2d1c06e2f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2d30000f60 tx=0x7f2d30009040 comp rx=0 tx=0).stop 2024-04-25T01:17:25.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.364+0000 7f2d42e00640 1 -- 172.21.15.90:0/4069091108 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d3c155130 msgr2=0x7f2d3c148720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:25.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.364+0000 7f2d42e00640 1 --2- 172.21.15.90:0/4069091108 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d3c155130 0x7f2d3c148720 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2d2800c820 tx=0x7f2d2800cce0 comp rx=0 tx=0).stop 2024-04-25T01:17:25.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.364+0000 7f2d42e00640 1 -- 172.21.15.90:0/4069091108 shutdown_connections 2024-04-25T01:17:25.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.364+0000 7f2d42e00640 1 --2- 172.21.15.90:0/4069091108 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2d1c06be30 0x7f2d1c06e2f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:25.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.364+0000 7f2d42e00640 1 --2- 172.21.15.90:0/4069091108 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d3c15b780 0x7f2d3c0ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:25.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.364+0000 7f2d42e00640 1 --2- 172.21.15.90:0/4069091108 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d3c155130 0x7f2d3c148720 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:25.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.364+0000 7f2d42e00640 1 --2- 172.21.15.90:0/4069091108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d3c06e280 0x7f2d3c14bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:25.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.364+0000 7f2d42e00640 1 -- 172.21.15.90:0/4069091108 >> 172.21.15.90:0/4069091108 conn(0x7f2d3c06b120 msgr2=0x7f2d3c15d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:25.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.364+0000 7f2d42e00640 1 -- 172.21.15.90:0/4069091108 shutdown_connections 2024-04-25T01:17:25.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.364+0000 7f2d42e00640 1 -- 172.21.15.90:0/4069091108 wait complete. 2024-04-25T01:17:25.366 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:17:25.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.582+0000 7f58dac00640 1 Processor -- start 2024-04-25T01:17:25.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.582+0000 7f58dac00640 1 -- start start 2024-04-25T01:17:25.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.584+0000 7f58dac00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f58d406e5a0 0x7f58d40cf7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:25.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.584+0000 7f58dac00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f58d40b2880 0x7f58d40cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:25.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.584+0000 7f58dac00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58d415c200 con 0x7f58d41596c0 2024-04-25T01:17:25.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.584+0000 7f58dac00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f58d4159aa0 con 0x7f58d40b2880 2024-04-25T01:17:25.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.584+0000 7f58dac00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f58d415c470 con 0x7f58d406e5a0 2024-04-25T01:17:25.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.584+0000 7f58d9800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f58d41596c0 0x7f58d406de90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47264/0 (socket says 172.21.15.90:47264) 2024-04-25T01:17:25.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.584+0000 7f58d9800640 1 -- 172.21.15.90:0/4066331440 learned_addr learned my addr 172.21.15.90:0/4066331440 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:25.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.584+0000 7f58da200640 1 --2- 172.21.15.90:0/4066331440 >> v2:172.21.15.90:3300/0 conn(0x7f58d40b2880 0x7f58d40cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:25.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.584+0000 7f58d8e00640 1 --2- 172.21.15.90:0/4066331440 >> v2:172.21.15.103:3300/0 conn(0x7f58d406e5a0 0x7f58d40cf7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:25.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.584+0000 7f58d3e00640 1 -- 172.21.15.90:0/4066331440 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 771874247 0 0) 0x7f58d415c200 con 0x7f58d41596c0 2024-04-25T01:17:25.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.584+0000 7f58d3e00640 1 -- 172.21.15.90:0/4066331440 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58bc003520 con 0x7f58d41596c0 2024-04-25T01:17:25.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.585+0000 7f58da200640 1 -- 172.21.15.90:0/4066331440 >> v2:172.21.15.103:3300/0 conn(0x7f58d406e5a0 msgr2=0x7f58d40cf7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:25.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.585+0000 7f58da200640 1 --2- 172.21.15.90:0/4066331440 >> v2:172.21.15.103:3300/0 conn(0x7f58d406e5a0 0x7f58d40cf7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:25.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.585+0000 7f58da200640 1 -- 172.21.15.90:0/4066331440 >> v1:172.21.15.90:6789/0 conn(0x7f58d41596c0 legacy=0x7f58d406de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:25.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.585+0000 7f58da200640 1 -- 172.21.15.90:0/4066331440 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58d415c750 con 0x7f58d40b2880 2024-04-25T01:17:25.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.586+0000 7f58da200640 1 --2- 172.21.15.90:0/4066331440 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58d40b2880 0x7f58d40cfd30 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f58d415c130 tx=0x7f58cc02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8595e84ed337a6c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:25.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.586+0000 7f58d3e00640 1 -- 172.21.15.90:0/4066331440 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58cc007740 con 0x7f58d40b2880 2024-04-25T01:17:25.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.586+0000 7f58d3e00640 1 -- 172.21.15.90:0/4066331440 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58cc0078e0 con 0x7f58d40b2880 2024-04-25T01:17:25.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.586+0000 7f58d3e00640 1 -- 172.21.15.90:0/4066331440 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58cc007cb0 con 0x7f58d40b2880 2024-04-25T01:17:25.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.586+0000 7f58dac00640 1 -- 172.21.15.90:0/4066331440 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58d40b2880 msgr2=0x7f58d40cfd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:25.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.586+0000 7f58dac00640 1 --2- 172.21.15.90:0/4066331440 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58d40b2880 0x7f58d40cfd30 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f58d415c130 tx=0x7f58cc02a680 comp rx=0 tx=0).stop 2024-04-25T01:17:25.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.587+0000 7f58dac00640 1 -- 172.21.15.90:0/4066331440 shutdown_connections 2024-04-25T01:17:25.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.587+0000 7f58dac00640 1 --2- 172.21.15.90:0/4066331440 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58d40b2880 0x7f58d40cfd30 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:25.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.587+0000 7f58dac00640 1 --2- 172.21.15.90:0/4066331440 >> v2:172.21.15.103:3300/0 conn(0x7f58d406e5a0 0x7f58d40cf7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:25.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.587+0000 7f58dac00640 1 -- 172.21.15.90:0/4066331440 >> 172.21.15.90:0/4066331440 conn(0x7f58d406b120 msgr2=0x7f58d406b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:25.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.587+0000 7f58dac00640 1 -- 172.21.15.90:0/4066331440 shutdown_connections 2024-04-25T01:17:25.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.587+0000 7f58dac00640 1 -- 172.21.15.90:0/4066331440 wait complete. 2024-04-25T01:17:25.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.588+0000 7f58dac00640 1 Processor -- start 2024-04-25T01:17:25.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.588+0000 7f58dac00640 1 -- start start 2024-04-25T01:17:25.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.589+0000 7f58dac00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58d406e5a0 0x7f58d4150710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:25.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.589+0000 7f58dac00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58d40b2880 0x7f58d4157b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:25.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.589+0000 7f58dac00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f58d41596c0 0x7f58d415dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:25.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.589+0000 7f58dac00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f58d40d0270 con 0x7f58d406e5a0 2024-04-25T01:17:25.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.589+0000 7f58dac00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f58d415cd10 con 0x7f58d40b2880 2024-04-25T01:17:25.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.589+0000 7f58dac00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f58d415f770 con 0x7f58d41596c0 2024-04-25T01:17:25.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.590+0000 7f58d9800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58d406e5a0 0x7f58d4150710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:25.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.590+0000 7f58d9800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58d406e5a0 0x7f58d4150710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33192/0 (socket says 172.21.15.90:33192) 2024-04-25T01:17:25.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.590+0000 7f58d9800640 1 -- 172.21.15.90:0/199171017 learned_addr learned my addr 172.21.15.90:0/199171017 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:25.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.590+0000 7f58d9800640 1 -- 172.21.15.90:0/199171017 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f58d41596c0 msgr2=0x7f58d415dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:25.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.590+0000 7f58d9800640 1 --2- 172.21.15.90:0/199171017 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f58d41596c0 0x7f58d415dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:25.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.590+0000 7f58d9800640 1 -- 172.21.15.90:0/199171017 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58d40b2880 msgr2=0x7f58d4157b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:25.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.590+0000 7f58d9800640 1 --2- 172.21.15.90:0/199171017 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58d40b2880 0x7f58d4157b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:25.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.590+0000 7f58d9800640 1 -- 172.21.15.90:0/199171017 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58d4158580 con 0x7f58d406e5a0 2024-04-25T01:17:25.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.590+0000 7f58d9800640 1 --2- 172.21.15.90:0/199171017 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58d406e5a0 0x7f58d4150710 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f58c000c390 tx=0x7f58c000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:25.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.590+0000 7f58d1600640 1 -- 172.21.15.90:0/199171017 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c0017070 con 0x7f58d406e5a0 2024-04-25T01:17:25.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.591+0000 7f58d1600640 1 -- 172.21.15.90:0/199171017 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58c0005ce0 con 0x7f58d406e5a0 2024-04-25T01:17:25.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.591+0000 7f58dac00640 1 -- 172.21.15.90:0/199171017 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58d4071620 con 0x7f58d406e5a0 2024-04-25T01:17:25.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.591+0000 7f58d1600640 1 -- 172.21.15.90:0/199171017 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c00072f0 con 0x7f58d406e5a0 2024-04-25T01:17:25.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.591+0000 7f58dac00640 1 -- 172.21.15.90:0/199171017 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58d4071be0 con 0x7f58d406e5a0 2024-04-25T01:17:25.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.592+0000 7f58dac00640 1 -- 172.21.15.90:0/199171017 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58d4071de0 con 0x7f58d406e5a0 2024-04-25T01:17:25.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.593+0000 7f58d1600640 1 -- 172.21.15.90:0/199171017 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f58c001a070 con 0x7f58d406e5a0 2024-04-25T01:17:25.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.594+0000 7f58d1600640 1 --2- 172.21.15.90:0/199171017 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f58b406be30 0x7f58b406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:25.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.594+0000 7f58d1600640 1 -- 172.21.15.90:0/199171017 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f58c008a650 con 0x7f58d406e5a0 2024-04-25T01:17:25.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.594+0000 7f58d8e00640 1 --2- 172.21.15.90:0/199171017 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f58b406be30 0x7f58b406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:25.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.595+0000 7f58d8e00640 1 --2- 172.21.15.90:0/199171017 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f58b406be30 0x7f58b406e2f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f58d415d790 tx=0x7f58c8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:25.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.599+0000 7f58d1600640 1 -- 172.21.15.90:0/199171017 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f58d4071de0 con 0x7f58d406e5a0 2024-04-25T01:17:25.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.797+0000 7f58dac00640 1 -- 172.21.15.90:0/199171017 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f58d4075350 con 0x7f58d406e5a0 2024-04-25T01:17:25.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.798+0000 7f58d1600640 1 -- 172.21.15.90:0/199171017 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) ==== 104+0+11 (secure 0 0 0) 0x7f58c0058bf0 con 0x7f58d406e5a0 2024-04-25T01:17:25.783 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:17:25.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.801+0000 7f58dac00640 1 -- 172.21.15.90:0/199171017 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f58b406be30 msgr2=0x7f58b406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:25.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.801+0000 7f58dac00640 1 --2- 172.21.15.90:0/199171017 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f58b406be30 0x7f58b406e2f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f58d415d790 tx=0x7f58c8008040 comp rx=0 tx=0).stop 2024-04-25T01:17:25.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.801+0000 7f58dac00640 1 -- 172.21.15.90:0/199171017 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58d406e5a0 msgr2=0x7f58d4150710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:25.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.801+0000 7f58dac00640 1 --2- 172.21.15.90:0/199171017 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58d406e5a0 0x7f58d4150710 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f58c000c390 tx=0x7f58c000c850 comp rx=0 tx=0).stop 2024-04-25T01:17:25.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.801+0000 7f58dac00640 1 -- 172.21.15.90:0/199171017 shutdown_connections 2024-04-25T01:17:25.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.801+0000 7f58dac00640 1 --2- 172.21.15.90:0/199171017 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f58b406be30 0x7f58b406e2f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:25.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.801+0000 7f58dac00640 1 --2- 172.21.15.90:0/199171017 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f58d41596c0 0x7f58d415dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:25.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.801+0000 7f58dac00640 1 --2- 172.21.15.90:0/199171017 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58d40b2880 0x7f58d4157b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:25.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.801+0000 7f58dac00640 1 --2- 172.21.15.90:0/199171017 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58d406e5a0 0x7f58d4150710 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:25.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.801+0000 7f58dac00640 1 -- 172.21.15.90:0/199171017 >> 172.21.15.90:0/199171017 conn(0x7f58d406b120 msgr2=0x7f58d4155460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:25.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.801+0000 7f58dac00640 1 -- 172.21.15.90:0/199171017 shutdown_connections 2024-04-25T01:17:25.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:25.801+0000 7f58dac00640 1 -- 172.21.15.90:0/199171017 wait complete. 2024-04-25T01:17:25.804 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:17:26.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.024+0000 7fe9c7200640 1 Processor -- start 2024-04-25T01:17:26.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.024+0000 7fe9c7200640 1 -- start start 2024-04-25T01:17:26.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.025+0000 7fe9c7200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe9c81596c0 0x7fe9c806de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:26.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.025+0000 7fe9c7200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9c80b2880 0x7fe9c806e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:26.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.025+0000 7fe9c7200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9c815c200 con 0x7fe9c80cf7f0 2024-04-25T01:17:26.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.025+0000 7fe9c7200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe9c8159aa0 con 0x7fe9c81596c0 2024-04-25T01:17:26.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.025+0000 7fe9c7200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe9c815c470 con 0x7fe9c80b2880 2024-04-25T01:17:26.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.026+0000 7fe9c6800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe9c80cf7f0 0x7fe9c80cfc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47278/0 (socket says 172.21.15.90:47278) 2024-04-25T01:17:26.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.026+0000 7fe9c6800640 1 -- 172.21.15.90:0/132747047 learned_addr learned my addr 172.21.15.90:0/132747047 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:26.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.026+0000 7fe9c5e00640 1 --2- 172.21.15.90:0/132747047 >> v2:172.21.15.103:3300/0 conn(0x7fe9c81596c0 0x7fe9c806de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:26.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.026+0000 7fe9bfe00640 1 --2- 172.21.15.90:0/132747047 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9c80b2880 0x7fe9c806e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:26.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.026+0000 7fe9c5400640 1 -- 172.21.15.90:0/132747047 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1078324295 0 0) 0x7fe9c815c200 con 0x7fe9c80cf7f0 2024-04-25T01:17:26.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.026+0000 7fe9c5400640 1 -- 172.21.15.90:0/132747047 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9ac003520 con 0x7fe9c80cf7f0 2024-04-25T01:17:26.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.026+0000 7fe9c5400640 1 -- 172.21.15.90:0/132747047 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1645203813 0 0) 0x7fe9ac003520 con 0x7fe9c80cf7f0 2024-04-25T01:17:26.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.027+0000 7fe9c5400640 1 -- 172.21.15.90:0/132747047 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9c80b2880 msgr2=0x7fe9c806e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:26.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.027+0000 7fe9c5400640 1 --2- 172.21.15.90:0/132747047 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9c80b2880 0x7fe9c806e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:26.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.027+0000 7fe9c5400640 1 -- 172.21.15.90:0/132747047 >> v2:172.21.15.103:3300/0 conn(0x7fe9c81596c0 msgr2=0x7fe9c806de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:26.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.027+0000 7fe9c5400640 1 --2- 172.21.15.90:0/132747047 >> v2:172.21.15.103:3300/0 conn(0x7fe9c81596c0 0x7fe9c806de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:26.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.027+0000 7fe9c5400640 1 -- 172.21.15.90:0/132747047 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9c815c750 con 0x7fe9c80cf7f0 2024-04-25T01:17:26.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.027+0000 7fe9c5400640 1 -- 172.21.15.90:0/132747047 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7fe9c00019f0 con 0x7fe9c80cf7f0 2024-04-25T01:17:26.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.027+0000 7fe9c5400640 1 -- 172.21.15.90:0/132747047 >> v1:172.21.15.90:6789/0 conn(0x7fe9c80cf7f0 legacy=0x7fe9c80cfc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:26.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.027+0000 7fe9c5400640 1 --2- 172.21.15.90:0/132747047 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9c80b2880 0x7fe9ac0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:26.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.027+0000 7fe9c5400640 1 --2- 172.21.15.90:0/132747047 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9ac003f30 0x7fe9ac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:26.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.027+0000 7fe9c5400640 1 --2- 172.21.15.90:0/132747047 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9ac004d40 0x7fe9ac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:26.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.027+0000 7fe9c5400640 1 -- 172.21.15.90:0/132747047 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c8159aa0 con 0x7fe9ac004d40 2024-04-25T01:17:26.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.027+0000 7fe9c5400640 1 -- 172.21.15.90:0/132747047 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c815c470 con 0x7fe9ac003f30 2024-04-25T01:17:26.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.027+0000 7fe9c5400640 1 -- 172.21.15.90:0/132747047 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe9ac00b9c0 con 0x7fe9c80b2880 2024-04-25T01:17:26.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.027+0000 7fe9c6800640 1 --2- 172.21.15.90:0/132747047 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9ac004d40 0x7fe9ac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:26.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.027+0000 7fe9c5e00640 1 --2- 172.21.15.90:0/132747047 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9c80b2880 0x7fe9ac0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:26.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.027+0000 7fe9c6800640 1 -- 172.21.15.90:0/132747047 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9c80b2880 msgr2=0x7fe9ac0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:26.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.027+0000 7fe9c6800640 1 --2- 172.21.15.90:0/132747047 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9c80b2880 0x7fe9ac0039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:26.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.027+0000 7fe9c6800640 1 -- 172.21.15.90:0/132747047 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9ac003f30 msgr2=0x7fe9ac004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:26.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.027+0000 7fe9c6800640 1 --2- 172.21.15.90:0/132747047 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9ac003f30 0x7fe9ac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:26.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.027+0000 7fe9c6800640 1 -- 172.21.15.90:0/132747047 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9ac003520 con 0x7fe9ac004d40 2024-04-25T01:17:26.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.028+0000 7fe9c6800640 1 --2- 172.21.15.90:0/132747047 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9ac004d40 0x7fe9ac0051c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe9c000a8a0 tx=0x7fe9ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=271102417f12d6ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:26.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.029+0000 7fe9c5400640 1 -- 172.21.15.90:0/132747047 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c000a4c0 con 0x7fe9ac004d40 2024-04-25T01:17:26.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.029+0000 7fe9c5400640 1 -- 172.21.15.90:0/132747047 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9c002f070 con 0x7fe9ac004d40 2024-04-25T01:17:26.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.029+0000 7fe9c5400640 1 -- 172.21.15.90:0/132747047 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c002bd50 con 0x7fe9ac004d40 2024-04-25T01:17:26.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.029+0000 7fe9c7200640 1 -- 172.21.15.90:0/132747047 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9ac004d40 msgr2=0x7fe9ac0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:26.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.029+0000 7fe9c7200640 1 --2- 172.21.15.90:0/132747047 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9ac004d40 0x7fe9ac0051c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe9c000a8a0 tx=0x7fe9ac005830 comp rx=0 tx=0).stop 2024-04-25T01:17:26.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.029+0000 7fe9c6800640 1 -- 172.21.15.90:0/132747047 reap_dead start 2024-04-25T01:17:26.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.029+0000 7fe9c7200640 1 -- 172.21.15.90:0/132747047 shutdown_connections 2024-04-25T01:17:26.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.029+0000 7fe9c7200640 1 -- 172.21.15.90:0/132747047 >> 172.21.15.90:0/132747047 conn(0x7fe9c806b120 msgr2=0x7fe9c806b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:26.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.029+0000 7fe9c7200640 1 -- 172.21.15.90:0/132747047 shutdown_connections 2024-04-25T01:17:26.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.029+0000 7fe9c7200640 1 -- 172.21.15.90:0/132747047 wait complete. 2024-04-25T01:17:26.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.031+0000 7fe9c7200640 1 Processor -- start 2024-04-25T01:17:26.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.031+0000 7fe9c7200640 1 -- start start 2024-04-25T01:17:26.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.032+0000 7fe9c7200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9ac004d40 0x7fe9c814a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:26.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.032+0000 7fe9c7200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9c8149c80 0x7fe9c81565e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:26.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.032+0000 7fe9c7200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9c80cba70 0x7fe9c80cbef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:26.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.032+0000 7fe9c7200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c806e3d0 con 0x7fe9c8149c80 2024-04-25T01:17:26.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.032+0000 7fe9c7200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c815cd10 con 0x7fe9ac004d40 2024-04-25T01:17:26.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.032+0000 7fe9c7200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe9c806e550 con 0x7fe9c80cba70 2024-04-25T01:17:26.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.032+0000 7fe9bfe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9c8149c80 0x7fe9c81565e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:26.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.032+0000 7fe9bfe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9c8149c80 0x7fe9c81565e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33202/0 (socket says 172.21.15.90:33202) 2024-04-25T01:17:26.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.033+0000 7fe9bfe00640 1 -- 172.21.15.90:0/898563681 learned_addr learned my addr 172.21.15.90:0/898563681 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:26.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.033+0000 7fe9c5e00640 1 --2- 172.21.15.90:0/898563681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9ac004d40 0x7fe9c814a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:26.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.033+0000 7fe9bfe00640 1 -- 172.21.15.90:0/898563681 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9c80cba70 msgr2=0x7fe9c80cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:26.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.033+0000 7fe9bfe00640 1 --2- 172.21.15.90:0/898563681 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9c80cba70 0x7fe9c80cbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:26.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.033+0000 7fe9bfe00640 1 -- 172.21.15.90:0/898563681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9ac004d40 msgr2=0x7fe9c814a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:26.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.033+0000 7fe9bfe00640 1 --2- 172.21.15.90:0/898563681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9ac004d40 0x7fe9c814a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:26.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.033+0000 7fe9bfe00640 1 -- 172.21.15.90:0/898563681 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9c815b4b0 con 0x7fe9c8149c80 2024-04-25T01:17:26.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.033+0000 7fe9bfe00640 1 --2- 172.21.15.90:0/898563681 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9c8149c80 0x7fe9c81565e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe9b000e820 tx=0x7fe9b000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:26.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.033+0000 7fe9be000640 1 -- 172.21.15.90:0/898563681 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b0017070 con 0x7fe9c8149c80 2024-04-25T01:17:26.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.033+0000 7fe9be000640 1 -- 172.21.15.90:0/898563681 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9b000c840 con 0x7fe9c8149c80 2024-04-25T01:17:26.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.033+0000 7fe9c7200640 1 -- 172.21.15.90:0/898563681 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9c815b790 con 0x7fe9c8149c80 2024-04-25T01:17:26.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.034+0000 7fe9be000640 1 -- 172.21.15.90:0/898563681 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b000cc30 con 0x7fe9c8149c80 2024-04-25T01:17:26.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.034+0000 7fe9c7200640 1 -- 172.21.15.90:0/898563681 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9c8158220 con 0x7fe9c8149c80 2024-04-25T01:17:26.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.035+0000 7fe9c7200640 1 -- 172.21.15.90:0/898563681 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9c8075350 con 0x7fe9c8149c80 2024-04-25T01:17:26.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.036+0000 7fe9be000640 1 -- 172.21.15.90:0/898563681 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe9b001a070 con 0x7fe9c8149c80 2024-04-25T01:17:26.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.037+0000 7fe9be000640 1 --2- 172.21.15.90:0/898563681 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe9a406be30 0x7fe9a406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:26.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.037+0000 7fe9be000640 1 -- 172.21.15.90:0/898563681 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7fe9b008b600 con 0x7fe9c8149c80 2024-04-25T01:17:26.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.037+0000 7fe9c5e00640 1 --2- 172.21.15.90:0/898563681 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe9a406be30 0x7fe9a406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:26.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.038+0000 7fe9c5e00640 1 --2- 172.21.15.90:0/898563681 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe9a406be30 0x7fe9a406e2f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe9c80af900 tx=0x7fe9b8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:26.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.041+0000 7fe9be000640 1 -- 172.21.15.90:0/898563681 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe9b0059cd0 con 0x7fe9c8149c80 2024-04-25T01:17:26.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.238+0000 7fe9c7200640 1 -- 172.21.15.90:0/898563681 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe9c80cc610 con 0x7fe9c8149c80 2024-04-25T01:17:26.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.238+0000 7fe9be000640 1 -- 172.21.15.90:0/898563681 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) ==== 100+0+11 (secure 0 0 0) 0x7fe9b00595d0 con 0x7fe9c8149c80 2024-04-25T01:17:26.225 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-25T01:17:26.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.241+0000 7fe9c7200640 1 -- 172.21.15.90:0/898563681 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe9a406be30 msgr2=0x7fe9a406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:26.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.241+0000 7fe9c7200640 1 --2- 172.21.15.90:0/898563681 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe9a406be30 0x7fe9a406e2f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe9c80af900 tx=0x7fe9b8008040 comp rx=0 tx=0).stop 2024-04-25T01:17:26.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.241+0000 7fe9c7200640 1 -- 172.21.15.90:0/898563681 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9c8149c80 msgr2=0x7fe9c81565e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:26.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.241+0000 7fe9c7200640 1 --2- 172.21.15.90:0/898563681 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9c8149c80 0x7fe9c81565e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe9b000e820 tx=0x7fe9b000ece0 comp rx=0 tx=0).stop 2024-04-25T01:17:26.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.241+0000 7fe9c7200640 1 -- 172.21.15.90:0/898563681 shutdown_connections 2024-04-25T01:17:26.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.241+0000 7fe9c7200640 1 --2- 172.21.15.90:0/898563681 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe9a406be30 0x7fe9a406e2f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:26.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.241+0000 7fe9c7200640 1 --2- 172.21.15.90:0/898563681 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9c80cba70 0x7fe9c80cbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:26.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.241+0000 7fe9c7200640 1 --2- 172.21.15.90:0/898563681 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9c8149c80 0x7fe9c81565e0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:26.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.241+0000 7fe9c7200640 1 --2- 172.21.15.90:0/898563681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9ac004d40 0x7fe9c814a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:26.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.241+0000 7fe9c7200640 1 -- 172.21.15.90:0/898563681 >> 172.21.15.90:0/898563681 conn(0x7fe9c806b120 msgr2=0x7fe9c80b3660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:26.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.241+0000 7fe9c7200640 1 -- 172.21.15.90:0/898563681 shutdown_connections 2024-04-25T01:17:26.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.241+0000 7fe9c7200640 1 -- 172.21.15.90:0/898563681 wait complete. 2024-04-25T01:17:26.246 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:17:26.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.466+0000 7fb41ac00640 1 Processor -- start 2024-04-25T01:17:26.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.466+0000 7fb41ac00640 1 -- start start 2024-04-25T01:17:26.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.468+0000 7fb41ac00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb4141596c0 0x7fb41406de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:26.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.468+0000 7fb41ac00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb41415c200 con 0x7fb4140b2880 2024-04-25T01:17:26.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.468+0000 7fb41ac00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb414159aa0 con 0x7fb4140cff00 2024-04-25T01:17:26.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.468+0000 7fb41ac00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fb4140d0330 con 0x7fb4141596c0 2024-04-25T01:17:26.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.468+0000 7fb418e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fb4140b2880 0x7fb4140cf7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47294/0 (socket says 172.21.15.90:47294) 2024-04-25T01:17:26.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.468+0000 7fb418e00640 1 -- 172.21.15.90:0/1524792850 learned_addr learned my addr 172.21.15.90:0/1524792850 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:26.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.468+0000 7fb419800640 1 --2- 172.21.15.90:0/1524792850 >> v2:172.21.15.90:3300/0 conn(0x7fb4141596c0 0x7fb41406de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:26.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.469+0000 7fb413e00640 1 -- 172.21.15.90:0/1524792850 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3501530357 0 0) 0x7fb41415c200 con 0x7fb4140b2880 2024-04-25T01:17:26.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.469+0000 7fb413e00640 1 -- 172.21.15.90:0/1524792850 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3fc003520 con 0x7fb4140b2880 2024-04-25T01:17:26.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.469+0000 7fb419800640 1 -- 172.21.15.90:0/1524792850 >> v1:172.21.15.103:6789/0 conn(0x7fb4140cff00 legacy=0x7fb41415f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:26.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.469+0000 7fb419800640 1 -- 172.21.15.90:0/1524792850 >> v1:172.21.15.90:6789/0 conn(0x7fb4140b2880 legacy=0x7fb4140cf7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:26.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.469+0000 7fb419800640 1 -- 172.21.15.90:0/1524792850 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb41415c470 con 0x7fb4141596c0 2024-04-25T01:17:26.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.470+0000 7fb419800640 1 --2- 172.21.15.90:0/1524792850 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb4141596c0 0x7fb41406de90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb4080097a0 tx=0x7fb40802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3864a96d4fccbf4c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:26.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.470+0000 7fb413e00640 1 -- 172.21.15.90:0/1524792850 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb408037070 con 0x7fb4141596c0 2024-04-25T01:17:26.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.470+0000 7fb413e00640 1 -- 172.21.15.90:0/1524792850 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb408032440 con 0x7fb4141596c0 2024-04-25T01:17:26.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.470+0000 7fb413e00640 1 -- 172.21.15.90:0/1524792850 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb408032810 con 0x7fb4141596c0 2024-04-25T01:17:26.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.471+0000 7fb41ac00640 1 -- 172.21.15.90:0/1524792850 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb4141596c0 msgr2=0x7fb41406de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:26.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.471+0000 7fb41ac00640 1 --2- 172.21.15.90:0/1524792850 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb4141596c0 0x7fb41406de90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb4080097a0 tx=0x7fb40802a230 comp rx=0 tx=0).stop 2024-04-25T01:17:26.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.471+0000 7fb41ac00640 1 -- 172.21.15.90:0/1524792850 shutdown_connections 2024-04-25T01:17:26.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.471+0000 7fb41ac00640 1 --2- 172.21.15.90:0/1524792850 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb4141596c0 0x7fb41406de90 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:26.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.471+0000 7fb41ac00640 1 -- 172.21.15.90:0/1524792850 >> 172.21.15.90:0/1524792850 conn(0x7fb41406b120 msgr2=0x7fb41406b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:26.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.471+0000 7fb41ac00640 1 -- 172.21.15.90:0/1524792850 shutdown_connections 2024-04-25T01:17:26.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.471+0000 7fb41ac00640 1 -- 172.21.15.90:0/1524792850 wait complete. 2024-04-25T01:17:26.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.473+0000 7fb41ac00640 1 Processor -- start 2024-04-25T01:17:26.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.473+0000 7fb41ac00640 1 -- start start 2024-04-25T01:17:26.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.474+0000 7fb41ac00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb4140b2880 0x7fb4141f44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:26.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.474+0000 7fb41ac00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4140cff00 0x7fb4140af760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:26.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.474+0000 7fb41ac00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4141596c0 0x7fb414157f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:26.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.474+0000 7fb41ac00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb41415cd10 con 0x7fb4140b2880 2024-04-25T01:17:26.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.474+0000 7fb41ac00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb41415ca30 con 0x7fb4141596c0 2024-04-25T01:17:26.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.474+0000 7fb41ac00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb41406e3d0 con 0x7fb4140cff00 2024-04-25T01:17:26.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.474+0000 7fb419800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb4140b2880 0x7fb4141f44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:26.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.474+0000 7fb419800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb4140b2880 0x7fb4141f44e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33212/0 (socket says 172.21.15.90:33212) 2024-04-25T01:17:26.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.474+0000 7fb419800640 1 -- 172.21.15.90:0/2419051131 learned_addr learned my addr 172.21.15.90:0/2419051131 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:26.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.475+0000 7fb41a200640 1 --2- 172.21.15.90:0/2419051131 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4141596c0 0x7fb414157f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:26.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.475+0000 7fb419800640 1 -- 172.21.15.90:0/2419051131 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4140cff00 msgr2=0x7fb4140af760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:26.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.475+0000 7fb418e00640 1 --2- 172.21.15.90:0/2419051131 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4140cff00 0x7fb4140af760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:26.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.475+0000 7fb419800640 1 --2- 172.21.15.90:0/2419051131 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4140cff00 0x7fb4140af760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:26.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.475+0000 7fb419800640 1 -- 172.21.15.90:0/2419051131 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4141596c0 msgr2=0x7fb414157f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:26.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.475+0000 7fb419800640 1 --2- 172.21.15.90:0/2419051131 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4141596c0 0x7fb414157f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:26.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.475+0000 7fb419800640 1 -- 172.21.15.90:0/2419051131 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4140ae560 con 0x7fb4140b2880 2024-04-25T01:17:26.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.475+0000 7fb41a200640 1 --2- 172.21.15.90:0/2419051131 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4141596c0 0x7fb414157f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:26.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.475+0000 7fb419800640 1 --2- 172.21.15.90:0/2419051131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb4140b2880 0x7fb4141f44e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb408035040 tx=0x7fb4080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:26.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.475+0000 7fb411600640 1 -- 172.21.15.90:0/2419051131 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb408037050 con 0x7fb4140b2880 2024-04-25T01:17:26.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.475+0000 7fb411600640 1 -- 172.21.15.90:0/2419051131 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb408038040 con 0x7fb4140b2880 2024-04-25T01:17:26.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.475+0000 7fb41ac00640 1 -- 172.21.15.90:0/2419051131 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb41415e7a0 con 0x7fb4140b2880 2024-04-25T01:17:26.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.475+0000 7fb411600640 1 -- 172.21.15.90:0/2419051131 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb408004490 con 0x7fb4140b2880 2024-04-25T01:17:26.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.476+0000 7fb41ac00640 1 -- 172.21.15.90:0/2419051131 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb41415ed60 con 0x7fb4140b2880 2024-04-25T01:17:26.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.477+0000 7fb41ac00640 1 -- 172.21.15.90:0/2419051131 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb41415bed0 con 0x7fb4140b2880 2024-04-25T01:17:26.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.477+0000 7fb411600640 1 -- 172.21.15.90:0/2419051131 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb40803c6d0 con 0x7fb4140b2880 2024-04-25T01:17:26.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.478+0000 7fb411600640 1 --2- 172.21.15.90:0/2419051131 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb3f406be30 0x7fb3f406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:26.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.479+0000 7fb411600640 1 -- 172.21.15.90:0/2419051131 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7fb4080abbf0 con 0x7fb4140b2880 2024-04-25T01:17:26.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.479+0000 7fb418e00640 1 --2- 172.21.15.90:0/2419051131 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb3f406be30 0x7fb3f406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:26.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.479+0000 7fb418e00640 1 --2- 172.21.15.90:0/2419051131 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb3f406be30 0x7fb3f406e2f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb41415c310 tx=0x7fb400009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:26.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.483+0000 7fb411600640 1 -- 172.21.15.90:0/2419051131 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb41415bed0 con 0x7fb4140b2880 2024-04-25T01:17:26.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.687+0000 7fb41ac00640 1 -- 172.21.15.90:0/2419051131 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb414075350 con 0x7fb4140b2880 2024-04-25T01:17:26.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.688+0000 7fb411600640 1 -- 172.21.15.90:0/2419051131 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) ==== 103+0+11 (secure 0 0 0) 0x7fb40807a230 con 0x7fb4140b2880 2024-04-25T01:17:26.675 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:17:26.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.690+0000 7fb41ac00640 1 -- 172.21.15.90:0/2419051131 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb3f406be30 msgr2=0x7fb3f406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:26.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.690+0000 7fb41ac00640 1 --2- 172.21.15.90:0/2419051131 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb3f406be30 0x7fb3f406e2f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb41415c310 tx=0x7fb400009040 comp rx=0 tx=0).stop 2024-04-25T01:17:26.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.691+0000 7fb41ac00640 1 -- 172.21.15.90:0/2419051131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb4140b2880 msgr2=0x7fb4141f44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:26.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.691+0000 7fb41ac00640 1 --2- 172.21.15.90:0/2419051131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb4140b2880 0x7fb4141f44e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb408035040 tx=0x7fb4080046c0 comp rx=0 tx=0).stop 2024-04-25T01:17:26.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.691+0000 7fb41ac00640 1 -- 172.21.15.90:0/2419051131 shutdown_connections 2024-04-25T01:17:26.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.691+0000 7fb41ac00640 1 --2- 172.21.15.90:0/2419051131 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb3f406be30 0x7fb3f406e2f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:26.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.691+0000 7fb41ac00640 1 --2- 172.21.15.90:0/2419051131 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4141596c0 0x7fb414157f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:26.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.691+0000 7fb41ac00640 1 --2- 172.21.15.90:0/2419051131 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb4140cff00 0x7fb4140af760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:26.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.691+0000 7fb41ac00640 1 --2- 172.21.15.90:0/2419051131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb4140b2880 0x7fb4141f44e0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:26.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.691+0000 7fb41ac00640 1 -- 172.21.15.90:0/2419051131 >> 172.21.15.90:0/2419051131 conn(0x7fb41406b120 msgr2=0x7fb4140b2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:26.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.691+0000 7fb41ac00640 1 -- 172.21.15.90:0/2419051131 shutdown_connections 2024-04-25T01:17:26.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.691+0000 7fb41ac00640 1 -- 172.21.15.90:0/2419051131 wait complete. 2024-04-25T01:17:26.697 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:17:26.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.916+0000 7f1256000640 1 Processor -- start 2024-04-25T01:17:26.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.916+0000 7f1256000640 1 -- start start 2024-04-25T01:17:26.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.918+0000 7f1256000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1250145590 0x7f12500b4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:26.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.918+0000 7f1256000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f125006e280 0x7f125015fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:26.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.918+0000 7f1256000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12500b4240 con 0x7f125015f9b0 2024-04-25T01:17:26.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.918+0000 7f1256000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f1250160830 con 0x7f1250145590 2024-04-25T01:17:26.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.918+0000 7f1256000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1250160350 con 0x7f125006e280 2024-04-25T01:17:26.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.918+0000 7f124f400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1250145590 0x7f12500b4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:26.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.918+0000 7f124f400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1250145590 0x7f12500b4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33220/0 (socket says 172.21.15.90:33220) 2024-04-25T01:17:26.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.918+0000 7f124f400640 1 -- 172.21.15.90:0/2914776087 learned_addr learned my addr 172.21.15.90:0/2914776087 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:26.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.919+0000 7f124f400640 1 -- 172.21.15.90:0/2914776087 >> v2:172.21.15.103:3300/0 conn(0x7f125006e280 msgr2=0x7f125015fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:26.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.919+0000 7f124f400640 1 --2- 172.21.15.90:0/2914776087 >> v2:172.21.15.103:3300/0 conn(0x7f125006e280 0x7f125015fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:26.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.919+0000 7f124f400640 1 -- 172.21.15.90:0/2914776087 >> v1:172.21.15.103:6789/0 conn(0x7f125015f9b0 legacy=0x7f12500ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:26.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.919+0000 7f124f400640 1 -- 172.21.15.90:0/2914776087 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12500b4f60 con 0x7f1250145590 2024-04-25T01:17:26.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.920+0000 7f124f400640 1 --2- 172.21.15.90:0/2914776087 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1250145590 0x7f12500b4a20 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1250068b60 tx=0x7f124002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cccef6811bfaa6dc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:26.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.920+0000 7f124e000640 1 -- 172.21.15.90:0/2914776087 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1240037070 con 0x7f1250145590 2024-04-25T01:17:26.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.920+0000 7f124e000640 1 -- 172.21.15.90:0/2914776087 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1240032440 con 0x7f1250145590 2024-04-25T01:17:26.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.920+0000 7f124e000640 1 -- 172.21.15.90:0/2914776087 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1240032810 con 0x7f1250145590 2024-04-25T01:17:26.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.920+0000 7f1256000640 1 -- 172.21.15.90:0/2914776087 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1250145590 msgr2=0x7f12500b4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:26.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.920+0000 7f1256000640 1 --2- 172.21.15.90:0/2914776087 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1250145590 0x7f12500b4a20 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1250068b60 tx=0x7f124002a230 comp rx=0 tx=0).stop 2024-04-25T01:17:26.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.921+0000 7f1256000640 1 -- 172.21.15.90:0/2914776087 shutdown_connections 2024-04-25T01:17:26.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.921+0000 7f1256000640 1 --2- 172.21.15.90:0/2914776087 >> v2:172.21.15.103:3300/0 conn(0x7f125006e280 0x7f125015fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:26.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.921+0000 7f1256000640 1 --2- 172.21.15.90:0/2914776087 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1250145590 0x7f12500b4a20 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:26.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.921+0000 7f1256000640 1 -- 172.21.15.90:0/2914776087 >> 172.21.15.90:0/2914776087 conn(0x7f12500b1820 msgr2=0x7f12500b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:26.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.921+0000 7f1256000640 1 -- 172.21.15.90:0/2914776087 shutdown_connections 2024-04-25T01:17:26.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.921+0000 7f1256000640 1 -- 172.21.15.90:0/2914776087 wait complete. 2024-04-25T01:17:26.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.922+0000 7f1256000640 1 Processor -- start 2024-04-25T01:17:26.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.923+0000 7f1256000640 1 -- start start 2024-04-25T01:17:26.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.923+0000 7f1256000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f125006e280 0x7f125015b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:26.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.923+0000 7f1256000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1250145590 0x7f1250142eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:26.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.923+0000 7f1256000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f125015f9b0 0x7f12501f48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:26.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.923+0000 7f1256000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f12500b3c80 con 0x7f125006e280 2024-04-25T01:17:26.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.923+0000 7f1256000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f125006cde0 con 0x7f1250145590 2024-04-25T01:17:26.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.923+0000 7f1256000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f12500b3f60 con 0x7f125015f9b0 2024-04-25T01:17:26.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.924+0000 7f124f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f125006e280 0x7f125015b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:26.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.924+0000 7f124f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f125006e280 0x7f125015b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33228/0 (socket says 172.21.15.90:33228) 2024-04-25T01:17:26.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.924+0000 7f124f400640 1 -- 172.21.15.90:0/770971519 learned_addr learned my addr 172.21.15.90:0/770971519 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:26.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.924+0000 7f124fe00640 1 --2- 172.21.15.90:0/770971519 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f125015f9b0 0x7f12501f48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:26.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.924+0000 7f124f400640 1 -- 172.21.15.90:0/770971519 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f125015f9b0 msgr2=0x7f12501f48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:26.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.924+0000 7f124f400640 1 --2- 172.21.15.90:0/770971519 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f125015f9b0 0x7f12501f48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:26.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.924+0000 7f124f400640 1 -- 172.21.15.90:0/770971519 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1250145590 msgr2=0x7f1250142eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:26.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.924+0000 7f124f400640 1 --2- 172.21.15.90:0/770971519 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1250145590 0x7f1250142eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:26.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.924+0000 7f124f400640 1 -- 172.21.15.90:0/770971519 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f125015bce0 con 0x7f125006e280 2024-04-25T01:17:26.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.924+0000 7f124fe00640 1 --2- 172.21.15.90:0/770971519 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f125015f9b0 0x7f12501f48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:26.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.925+0000 7f124f400640 1 --2- 172.21.15.90:0/770971519 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f125006e280 0x7f125015b7a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f124002a270 tx=0x7f12400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:26.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.925+0000 7f1247400640 1 -- 172.21.15.90:0/770971519 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1240041070 con 0x7f125006e280 2024-04-25T01:17:26.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.925+0000 7f1247400640 1 -- 172.21.15.90:0/770971519 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f124003c870 con 0x7f125006e280 2024-04-25T01:17:26.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.925+0000 7f1247400640 1 -- 172.21.15.90:0/770971519 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1240037040 con 0x7f125006e280 2024-04-25T01:17:26.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.925+0000 7f1256000640 1 -- 172.21.15.90:0/770971519 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1250143700 con 0x7f125006e280 2024-04-25T01:17:26.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.925+0000 7f1256000640 1 -- 172.21.15.90:0/770971519 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1250143cc0 con 0x7f125006e280 2024-04-25T01:17:26.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.926+0000 7f1256000640 1 -- 172.21.15.90:0/770971519 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1250068fb0 con 0x7f125006e280 2024-04-25T01:17:26.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.927+0000 7f1247400640 1 -- 172.21.15.90:0/770971519 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1240044070 con 0x7f125006e280 2024-04-25T01:17:26.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.928+0000 7f1247400640 1 --2- 172.21.15.90:0/770971519 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f123006be30 0x7f123006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:26.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.928+0000 7f1247400640 1 -- 172.21.15.90:0/770971519 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f12400abd50 con 0x7f125006e280 2024-04-25T01:17:26.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.928+0000 7f124ea00640 1 --2- 172.21.15.90:0/770971519 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f123006be30 0x7f123006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:26.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.929+0000 7f124ea00640 1 --2- 172.21.15.90:0/770971519 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f123006be30 0x7f123006e2f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1250144b40 tx=0x7f123c006ba0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:26.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:26.932+0000 7f1247400640 1 -- 172.21.15.90:0/770971519 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f125015bce0 con 0x7f125006e280 2024-04-25T01:17:27.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.133+0000 7f1256000640 1 -- 172.21.15.90:0/770971519 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f125006ea50 con 0x7f125006e280 2024-04-25T01:17:27.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.136+0000 7f1247400640 1 -- 172.21.15.90:0/770971519 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18576 (secure 0 0 0) 0x7f124007a380 con 0x7f125006e280 2024-04-25T01:17:27.121 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:17:27.121 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:17:09.346008+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:16:20.094512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:16:21.078268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:16:24.670642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.e","mappings":[{"from":0,"to":7}]},{"pgid":"4.f","mappings":[{"from":7,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:17:27.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.139+0000 7f1256000640 1 -- 172.21.15.90:0/770971519 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f123006be30 msgr2=0x7f123006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:27.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.139+0000 7f1256000640 1 --2- 172.21.15.90:0/770971519 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f123006be30 0x7f123006e2f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1250144b40 tx=0x7f123c006ba0 comp rx=0 tx=0).stop 2024-04-25T01:17:27.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.139+0000 7f1256000640 1 -- 172.21.15.90:0/770971519 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f125006e280 msgr2=0x7f125015b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:27.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.139+0000 7f1256000640 1 --2- 172.21.15.90:0/770971519 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f125006e280 0x7f125015b7a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f124002a270 tx=0x7f12400046c0 comp rx=0 tx=0).stop 2024-04-25T01:17:27.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.139+0000 7f1256000640 1 -- 172.21.15.90:0/770971519 shutdown_connections 2024-04-25T01:17:27.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.139+0000 7f1256000640 1 --2- 172.21.15.90:0/770971519 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f123006be30 0x7f123006e2f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:27.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.139+0000 7f1256000640 1 --2- 172.21.15.90:0/770971519 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f125015f9b0 0x7f12501f48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:27.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.139+0000 7f1256000640 1 --2- 172.21.15.90:0/770971519 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1250145590 0x7f1250142eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:27.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.139+0000 7f1256000640 1 --2- 172.21.15.90:0/770971519 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f125006e280 0x7f125015b7a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:27.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.139+0000 7f1256000640 1 -- 172.21.15.90:0/770971519 >> 172.21.15.90:0/770971519 conn(0x7f12500b1820 msgr2=0x7f12500b0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:27.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.139+0000 7f1256000640 1 -- 172.21.15.90:0/770971519 shutdown_connections 2024-04-25T01:17:27.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.140+0000 7f1256000640 1 -- 172.21.15.90:0/770971519 wait complete. 2024-04-25T01:17:27.142 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:16:15.894126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:16:20.094512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:16:21.078268+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:16:24.670642+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:17:27.143 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:17:27.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.360+0000 7f247ac00640 1 Processor -- start 2024-04-25T01:17:27.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.361+0000 7f247ac00640 1 -- start start 2024-04-25T01:17:27.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.362+0000 7f247ac00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f247415df00 0x7f24740cf360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:27.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.362+0000 7f247ac00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24740d4980 0x7f24740d0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:27.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.362+0000 7f247ac00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24740b09d0 con 0x7f24740d0cd0 2024-04-25T01:17:27.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.362+0000 7f247ac00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f24740d1120 con 0x7f247415df00 2024-04-25T01:17:27.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.362+0000 7f247ac00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f24740b12e0 con 0x7f24740d4980 2024-04-25T01:17:27.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.363+0000 7f247a200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f24740d0cd0 0x7f24740d1950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47308/0 (socket says 172.21.15.90:47308) 2024-04-25T01:17:27.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.363+0000 7f247a200640 1 -- 172.21.15.90:0/4120547133 learned_addr learned my addr 172.21.15.90:0/4120547133 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:27.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.363+0000 7f2478e00640 1 --2- 172.21.15.90:0/4120547133 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24740d4980 0x7f24740d0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:27.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.363+0000 7f2479800640 1 --2- 172.21.15.90:0/4120547133 >> v2:172.21.15.103:3300/0 conn(0x7f247415df00 0x7f24740cf360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:27.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.363+0000 7f2473e00640 1 -- 172.21.15.90:0/4120547133 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1349662281 0 0) 0x7f24740b09d0 con 0x7f24740d0cd0 2024-04-25T01:17:27.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.363+0000 7f2473e00640 1 -- 172.21.15.90:0/4120547133 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f245c003520 con 0x7f24740d0cd0 2024-04-25T01:17:27.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.363+0000 7f2473e00640 1 -- 172.21.15.90:0/4120547133 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1673416272 0 0) 0x7f245c003520 con 0x7f24740d0cd0 2024-04-25T01:17:27.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.363+0000 7f2473e00640 1 -- 172.21.15.90:0/4120547133 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24740d4980 msgr2=0x7f24740d0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:27.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.363+0000 7f2473e00640 1 --2- 172.21.15.90:0/4120547133 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24740d4980 0x7f24740d0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:27.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.363+0000 7f2473e00640 1 -- 172.21.15.90:0/4120547133 >> v2:172.21.15.103:3300/0 conn(0x7f247415df00 msgr2=0x7f24740cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:27.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.363+0000 7f2473e00640 1 --2- 172.21.15.90:0/4120547133 >> v2:172.21.15.103:3300/0 conn(0x7f247415df00 0x7f24740cf360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:27.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.363+0000 7f2473e00640 1 -- 172.21.15.90:0/4120547133 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24740cfbb0 con 0x7f24740d0cd0 2024-04-25T01:17:27.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.363+0000 7f2473e00640 1 -- 172.21.15.90:0/4120547133 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f246c0019f0 con 0x7f24740d0cd0 2024-04-25T01:17:27.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.363+0000 7f2473e00640 1 -- 172.21.15.90:0/4120547133 >> v1:172.21.15.90:6789/0 conn(0x7f24740d0cd0 legacy=0x7f24740d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:27.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.364+0000 7f2473e00640 1 --2- 172.21.15.90:0/4120547133 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f245c0039b0 0x7f245c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:27.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.364+0000 7f2473e00640 1 --2- 172.21.15.90:0/4120547133 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24740d4980 0x7f245c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:27.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.364+0000 7f2473e00640 1 --2- 172.21.15.90:0/4120547133 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f245c004d40 0x7f245c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:27.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.364+0000 7f2473e00640 1 -- 172.21.15.90:0/4120547133 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f24740d1120 con 0x7f245c004d40 2024-04-25T01:17:27.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.364+0000 7f2473e00640 1 -- 172.21.15.90:0/4120547133 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f24740b12e0 con 0x7f245c0039b0 2024-04-25T01:17:27.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.364+0000 7f2473e00640 1 -- 172.21.15.90:0/4120547133 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f245c00b9c0 con 0x7f24740d4980 2024-04-25T01:17:27.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.364+0000 7f247a200640 1 --2- 172.21.15.90:0/4120547133 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f245c004d40 0x7f245c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:27.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.364+0000 7f2478e00640 1 --2- 172.21.15.90:0/4120547133 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24740d4980 0x7f245c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:27.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.364+0000 7f2479800640 1 --2- 172.21.15.90:0/4120547133 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f245c0039b0 0x7f245c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:27.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.364+0000 7f247a200640 1 -- 172.21.15.90:0/4120547133 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24740d4980 msgr2=0x7f245c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:27.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.364+0000 7f247a200640 1 --2- 172.21.15.90:0/4120547133 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24740d4980 0x7f245c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:27.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.364+0000 7f247a200640 1 -- 172.21.15.90:0/4120547133 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f245c0039b0 msgr2=0x7f245c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:27.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.364+0000 7f247a200640 1 --2- 172.21.15.90:0/4120547133 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f245c0039b0 0x7f245c003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:27.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.364+0000 7f247a200640 1 -- 172.21.15.90:0/4120547133 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f245c003520 con 0x7f245c004d40 2024-04-25T01:17:27.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.364+0000 7f2478e00640 1 --2- 172.21.15.90:0/4120547133 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f24740d4980 0x7f245c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:27.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.365+0000 7f247a200640 1 --2- 172.21.15.90:0/4120547133 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f245c004d40 0x7f245c0051c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f246c00a870 tx=0x7f245c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4853d6f8cf3e9e1e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:27.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.365+0000 7f2473e00640 1 -- 172.21.15.90:0/4120547133 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f246c00a4c0 con 0x7f245c004d40 2024-04-25T01:17:27.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.366+0000 7f2473e00640 1 -- 172.21.15.90:0/4120547133 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f246c02f070 con 0x7f245c004d40 2024-04-25T01:17:27.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.366+0000 7f2473e00640 1 -- 172.21.15.90:0/4120547133 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f246c02bcc0 con 0x7f245c004d40 2024-04-25T01:17:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.366+0000 7f247ac00640 1 -- 172.21.15.90:0/4120547133 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f245c004d40 msgr2=0x7f245c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.366+0000 7f247ac00640 1 --2- 172.21.15.90:0/4120547133 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f245c004d40 0x7f245c0051c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f246c00a870 tx=0x7f245c005830 comp rx=0 tx=0).stop 2024-04-25T01:17:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.366+0000 7f247a200640 1 -- 172.21.15.90:0/4120547133 reap_dead start 2024-04-25T01:17:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.366+0000 7f247ac00640 1 -- 172.21.15.90:0/4120547133 shutdown_connections 2024-04-25T01:17:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.366+0000 7f247ac00640 1 -- 172.21.15.90:0/4120547133 >> 172.21.15.90:0/4120547133 conn(0x7f24740d3d80 msgr2=0x7f24740d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.366+0000 7f247ac00640 1 -- 172.21.15.90:0/4120547133 shutdown_connections 2024-04-25T01:17:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.366+0000 7f247ac00640 1 -- 172.21.15.90:0/4120547133 wait complete. 2024-04-25T01:17:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.368+0000 7f247ac00640 1 Processor -- start 2024-04-25T01:17:27.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.368+0000 7f247ac00640 1 -- start start 2024-04-25T01:17:27.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.369+0000 7f247ac00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f245c004d40 0x7f24740c8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:27.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.369+0000 7f247ac00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24740ca7d0 0x7f24741f3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:27.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.369+0000 7f247ac00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24741f6be0 0x7f247406b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:27.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.369+0000 7f247ac00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f24740cf8a0 con 0x7f24740ca7d0 2024-04-25T01:17:27.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.369+0000 7f247ac00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f24740b0f70 con 0x7f24741f6be0 2024-04-25T01:17:27.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.369+0000 7f247ac00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f247415e2e0 con 0x7f245c004d40 2024-04-25T01:17:27.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.369+0000 7f2478e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24740ca7d0 0x7f24741f3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:27.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.369+0000 7f2478e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24740ca7d0 0x7f24741f3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33250/0 (socket says 172.21.15.90:33250) 2024-04-25T01:17:27.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.369+0000 7f2478e00640 1 -- 172.21.15.90:0/1739614535 learned_addr learned my addr 172.21.15.90:0/1739614535 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:27.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.370+0000 7f247a200640 1 --2- 172.21.15.90:0/1739614535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24741f6be0 0x7f247406b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:27.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.370+0000 7f2479800640 1 --2- 172.21.15.90:0/1739614535 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f245c004d40 0x7f24740c8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:27.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.370+0000 7f2478e00640 1 -- 172.21.15.90:0/1739614535 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f245c004d40 msgr2=0x7f24740c8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:27.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.370+0000 7f2478e00640 1 --2- 172.21.15.90:0/1739614535 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f245c004d40 0x7f24740c8e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:27.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.370+0000 7f2478e00640 1 -- 172.21.15.90:0/1739614535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24741f6be0 msgr2=0x7f247406b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:27.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.370+0000 7f2478e00640 1 --2- 172.21.15.90:0/1739614535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24741f6be0 0x7f247406b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:27.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.370+0000 7f2478e00640 1 -- 172.21.15.90:0/1739614535 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24741f7550 con 0x7f24740ca7d0 2024-04-25T01:17:27.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.370+0000 7f247a200640 1 --2- 172.21.15.90:0/1739614535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24741f6be0 0x7f247406b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:27.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.370+0000 7f2478e00640 1 --2- 172.21.15.90:0/1739614535 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24740ca7d0 0x7f24741f3f90 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f24600108e0 tx=0x7f2460010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:27.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.370+0000 7f2471600640 1 -- 172.21.15.90:0/1739614535 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f246000f040 con 0x7f24740ca7d0 2024-04-25T01:17:27.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.370+0000 7f2471600640 1 -- 172.21.15.90:0/1739614535 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f246000ec20 con 0x7f24740ca7d0 2024-04-25T01:17:27.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.371+0000 7f2471600640 1 -- 172.21.15.90:0/1739614535 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f246001c400 con 0x7f24740ca7d0 2024-04-25T01:17:27.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.371+0000 7f247ac00640 1 -- 172.21.15.90:0/1739614535 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24741f7830 con 0x7f24740ca7d0 2024-04-25T01:17:27.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.371+0000 7f247ac00640 1 -- 172.21.15.90:0/1739614535 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24741f3a70 con 0x7f24740ca7d0 2024-04-25T01:17:27.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.372+0000 7f247ac00640 1 -- 172.21.15.90:0/1739614535 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2474067dd0 con 0x7f24740ca7d0 2024-04-25T01:17:27.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.373+0000 7f2471600640 1 -- 172.21.15.90:0/1739614535 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f246001c5a0 con 0x7f24740ca7d0 2024-04-25T01:17:27.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.374+0000 7f2471600640 1 --2- 172.21.15.90:0/1739614535 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f245406be60 0x7f245406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:27.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.374+0000 7f2471600640 1 -- 172.21.15.90:0/1739614535 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f246008b960 con 0x7f24740ca7d0 2024-04-25T01:17:27.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.374+0000 7f2479800640 1 --2- 172.21.15.90:0/1739614535 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f245406be60 0x7f245406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:27.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.375+0000 7f2479800640 1 --2- 172.21.15.90:0/1739614535 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f245406be60 0x7f245406e320 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f24740cc350 tx=0x7f2468008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:27.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.378+0000 7f2471600640 1 -- 172.21.15.90:0/1739614535 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2460018030 con 0x7f24740ca7d0 2024-04-25T01:17:27.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.574+0000 7f247ac00640 1 -- 172.21.15.90:0/1739614535 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2474075350 con 0x7f24740ca7d0 2024-04-25T01:17:27.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.575+0000 7f2471600640 1 -- 172.21.15.90:0/1739614535 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) ==== 93+0+10 (secure 0 0 0) 0x7f2460059fa0 con 0x7f24740ca7d0 2024-04-25T01:17:27.560 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T01:17:27.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.578+0000 7f247ac00640 1 -- 172.21.15.90:0/1739614535 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f245406be60 msgr2=0x7f245406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:27.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.578+0000 7f247ac00640 1 --2- 172.21.15.90:0/1739614535 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f245406be60 0x7f245406e320 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f24740cc350 tx=0x7f2468008040 comp rx=0 tx=0).stop 2024-04-25T01:17:27.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.578+0000 7f247ac00640 1 -- 172.21.15.90:0/1739614535 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24740ca7d0 msgr2=0x7f24741f3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:27.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.578+0000 7f247ac00640 1 --2- 172.21.15.90:0/1739614535 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24740ca7d0 0x7f24741f3f90 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f24600108e0 tx=0x7f2460010da0 comp rx=0 tx=0).stop 2024-04-25T01:17:27.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.578+0000 7f247ac00640 1 -- 172.21.15.90:0/1739614535 shutdown_connections 2024-04-25T01:17:27.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.578+0000 7f247ac00640 1 --2- 172.21.15.90:0/1739614535 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f245406be60 0x7f245406e320 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:27.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.578+0000 7f247ac00640 1 --2- 172.21.15.90:0/1739614535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24741f6be0 0x7f247406b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:27.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.578+0000 7f247ac00640 1 --2- 172.21.15.90:0/1739614535 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24740ca7d0 0x7f24741f3f90 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:27.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.578+0000 7f247ac00640 1 --2- 172.21.15.90:0/1739614535 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f245c004d40 0x7f24740c8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:27.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.578+0000 7f247ac00640 1 -- 172.21.15.90:0/1739614535 >> 172.21.15.90:0/1739614535 conn(0x7f24740d3d80 msgr2=0x7f247415ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:27.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.578+0000 7f247ac00640 1 -- 172.21.15.90:0/1739614535 shutdown_connections 2024-04-25T01:17:27.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.578+0000 7f247ac00640 1 -- 172.21.15.90:0/1739614535 wait complete. 2024-04-25T01:17:27.580 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:17:27.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.795+0000 7fbb54a00640 1 Processor -- start 2024-04-25T01:17:27.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.795+0000 7fbb54a00640 1 -- start start 2024-04-25T01:17:27.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.796+0000 7fbb54a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbb5015fc40 0x7fbb501600a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:27.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.797+0000 7fbb54a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb5006d110 con 0x7fbb500722e0 2024-04-25T01:17:27.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.797+0000 7fbb54a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb5006d3f0 con 0x7fbb500cedc0 2024-04-25T01:17:27.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.797+0000 7fbb54a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fbb50160aa0 con 0x7fbb5015fc40 2024-04-25T01:17:27.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.797+0000 7fbb4de00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fbb500722e0 0x7fbb5006ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47314/0 (socket says 172.21.15.90:47314) 2024-04-25T01:17:27.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.797+0000 7fbb4de00640 1 -- 172.21.15.90:0/736514822 learned_addr learned my addr 172.21.15.90:0/736514822 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:27.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.797+0000 7fbb4d400640 1 --2- 172.21.15.90:0/736514822 >> v2:172.21.15.103:3300/0 conn(0x7fbb5015fc40 0x7fbb501600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:27.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.797+0000 7fbb4ca00640 1 -- 172.21.15.90:0/736514822 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1000833184 0 0) 0x7fbb5006d110 con 0x7fbb500722e0 2024-04-25T01:17:27.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.797+0000 7fbb4ca00640 1 -- 172.21.15.90:0/736514822 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb34003520 con 0x7fbb500722e0 2024-04-25T01:17:27.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.798+0000 7fbb4ca00640 1 -- 172.21.15.90:0/736514822 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2595672260 0 0) 0x7fbb34003520 con 0x7fbb500722e0 2024-04-25T01:17:27.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.798+0000 7fbb4ca00640 1 -- 172.21.15.90:0/736514822 >> v2:172.21.15.103:3300/0 conn(0x7fbb5015fc40 msgr2=0x7fbb501600a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:27.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.798+0000 7fbb4ca00640 1 --2- 172.21.15.90:0/736514822 >> v2:172.21.15.103:3300/0 conn(0x7fbb5015fc40 0x7fbb501600a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:27.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.798+0000 7fbb4ca00640 1 -- 172.21.15.90:0/736514822 >> v1:172.21.15.103:6789/0 conn(0x7fbb500cedc0 legacy=0x7fbb500cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:27.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.798+0000 7fbb4ca00640 1 -- 172.21.15.90:0/736514822 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb5006d6d0 con 0x7fbb500722e0 2024-04-25T01:17:27.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.798+0000 7fbb4ca00640 1 -- 172.21.15.90:0/736514822 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7fbb400019f0 con 0x7fbb500722e0 2024-04-25T01:17:27.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.798+0000 7fbb4ca00640 1 -- 172.21.15.90:0/736514822 >> v1:172.21.15.90:6789/0 conn(0x7fbb500722e0 legacy=0x7fbb5006ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:27.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.798+0000 7fbb4ca00640 1 --2- 172.21.15.90:0/736514822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb34003a00 0x7fbb34003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:27.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.798+0000 7fbb4ca00640 1 --2- 172.21.15.90:0/736514822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb34004800 0x7fbb34004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:27.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.798+0000 7fbb4ca00640 1 --2- 172.21.15.90:0/736514822 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb340052e0 0x7fbb3400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:27.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.798+0000 7fbb4ca00640 1 -- 172.21.15.90:0/736514822 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbb50160aa0 con 0x7fbb34003a00 2024-04-25T01:17:27.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.798+0000 7fbb4ca00640 1 -- 172.21.15.90:0/736514822 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbb3400c200 con 0x7fbb34004800 2024-04-25T01:17:27.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.798+0000 7fbb4ca00640 1 -- 172.21.15.90:0/736514822 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbb3400c400 con 0x7fbb340052e0 2024-04-25T01:17:27.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.798+0000 7fbb4de00640 1 --2- 172.21.15.90:0/736514822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb34003a00 0x7fbb34003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:27.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.798+0000 7fbb4d400640 1 --2- 172.21.15.90:0/736514822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb34004800 0x7fbb34004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:27.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.798+0000 7fbb4de00640 1 -- 172.21.15.90:0/736514822 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb340052e0 msgr2=0x7fbb3400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:27.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.799+0000 7fbb4de00640 1 --2- 172.21.15.90:0/736514822 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb340052e0 0x7fbb3400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:27.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.799+0000 7fbb4de00640 1 -- 172.21.15.90:0/736514822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb34004800 msgr2=0x7fbb34004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:27.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.799+0000 7fbb4de00640 1 --2- 172.21.15.90:0/736514822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb34004800 0x7fbb34004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:27.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.799+0000 7fbb4de00640 1 -- 172.21.15.90:0/736514822 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb34003520 con 0x7fbb34003a00 2024-04-25T01:17:27.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.799+0000 7fbb4d400640 1 --2- 172.21.15.90:0/736514822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb34004800 0x7fbb34004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:27.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.799+0000 7fbb4e800640 1 -- 172.21.15.90:0/736514822 reap_dead start 2024-04-25T01:17:27.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.800+0000 7fbb4de00640 1 --2- 172.21.15.90:0/736514822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb34003a00 0x7fbb34003e20 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fbb4000a7b0 tx=0x7fbb34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d23cadf25290c7bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:27.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.800+0000 7fbb4ca00640 1 -- 172.21.15.90:0/736514822 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb4000a3d0 con 0x7fbb34003a00 2024-04-25T01:17:27.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.800+0000 7fbb4ca00640 1 -- 172.21.15.90:0/736514822 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb4002f070 con 0x7fbb34003a00 2024-04-25T01:17:27.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.800+0000 7fbb4ca00640 1 -- 172.21.15.90:0/736514822 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb4002bc80 con 0x7fbb34003a00 2024-04-25T01:17:27.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.800+0000 7fbb54a00640 1 -- 172.21.15.90:0/736514822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb34003a00 msgr2=0x7fbb34003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:27.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.800+0000 7fbb54a00640 1 --2- 172.21.15.90:0/736514822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb34003a00 0x7fbb34003e20 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fbb4000a7b0 tx=0x7fbb34005830 comp rx=0 tx=0).stop 2024-04-25T01:17:27.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.801+0000 7fbb54a00640 1 -- 172.21.15.90:0/736514822 shutdown_connections 2024-04-25T01:17:27.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.801+0000 7fbb54a00640 1 --2- 172.21.15.90:0/736514822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb34003a00 0x7fbb34003e20 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:27.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.801+0000 7fbb54a00640 1 -- 172.21.15.90:0/736514822 >> 172.21.15.90:0/736514822 conn(0x7fbb5006b720 msgr2=0x7fbb500b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:27.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.801+0000 7fbb54a00640 1 -- 172.21.15.90:0/736514822 shutdown_connections 2024-04-25T01:17:27.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.801+0000 7fbb54a00640 1 -- 172.21.15.90:0/736514822 wait complete. 2024-04-25T01:17:27.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.802+0000 7fbb54a00640 1 Processor -- start 2024-04-25T01:17:27.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.803+0000 7fbb54a00640 1 -- start start 2024-04-25T01:17:27.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.804+0000 7fbb54a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb34003a00 0x7fbb500b3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:27.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.804+0000 7fbb54a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb501599f0 0x7fbb501f47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:27.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.804+0000 7fbb54a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb501f5c90 0x7fbb501f6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:27.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.804+0000 7fbb54a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbb50160430 con 0x7fbb501f5c90 2024-04-25T01:17:27.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.804+0000 7fbb54a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbb501602b0 con 0x7fbb34003a00 2024-04-25T01:17:27.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.804+0000 7fbb54a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbb501605b0 con 0x7fbb501599f0 2024-04-25T01:17:27.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.804+0000 7fbb4e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb501f5c90 0x7fbb501f6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:27.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.804+0000 7fbb4e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb501f5c90 0x7fbb501f6110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33264/0 (socket says 172.21.15.90:33264) 2024-04-25T01:17:27.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.804+0000 7fbb4e800640 1 -- 172.21.15.90:0/3485628450 learned_addr learned my addr 172.21.15.90:0/3485628450 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:27.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.804+0000 7fbb4d400640 1 --2- 172.21.15.90:0/3485628450 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb501599f0 0x7fbb501f47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:27.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.804+0000 7fbb4de00640 1 --2- 172.21.15.90:0/3485628450 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb34003a00 0x7fbb500b3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:27.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.804+0000 7fbb4e800640 1 -- 172.21.15.90:0/3485628450 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb501599f0 msgr2=0x7fbb501f47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:27.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.804+0000 7fbb4e800640 1 --2- 172.21.15.90:0/3485628450 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb501599f0 0x7fbb501f47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:27.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.804+0000 7fbb4e800640 1 -- 172.21.15.90:0/3485628450 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb34003a00 msgr2=0x7fbb500b3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:27.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.804+0000 7fbb4e800640 1 --2- 172.21.15.90:0/3485628450 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb34003a00 0x7fbb500b3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:27.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.804+0000 7fbb4e800640 1 -- 172.21.15.90:0/3485628450 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb500b3b90 con 0x7fbb501f5c90 2024-04-25T01:17:27.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.804+0000 7fbb4d400640 1 --2- 172.21.15.90:0/3485628450 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb501599f0 0x7fbb501f47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:27.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.805+0000 7fbb4e800640 1 --2- 172.21.15.90:0/3485628450 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb501f5c90 0x7fbb501f6110 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fbb500b27c0 tx=0x7fbb4800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:27.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.805+0000 7fbb46000640 1 -- 172.21.15.90:0/3485628450 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb480074e0 con 0x7fbb501f5c90 2024-04-25T01:17:27.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.805+0000 7fbb46000640 1 -- 172.21.15.90:0/3485628450 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb48004020 con 0x7fbb501f5c90 2024-04-25T01:17:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.805+0000 7fbb54a00640 1 -- 172.21.15.90:0/3485628450 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb500b3e70 con 0x7fbb501f5c90 2024-04-25T01:17:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.805+0000 7fbb46000640 1 -- 172.21.15.90:0/3485628450 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb48004410 con 0x7fbb501f5c90 2024-04-25T01:17:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.805+0000 7fbb54a00640 1 -- 172.21.15.90:0/3485628450 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb501f5770 con 0x7fbb501f5c90 2024-04-25T01:17:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.806+0000 7fbb54a00640 1 -- 172.21.15.90:0/3485628450 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb5006cc00 con 0x7fbb501f5c90 2024-04-25T01:17:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.807+0000 7fbb46000640 1 -- 172.21.15.90:0/3485628450 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbb480057c0 con 0x7fbb501f5c90 2024-04-25T01:17:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.808+0000 7fbb46000640 1 --2- 172.21.15.90:0/3485628450 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbb2c06be30 0x7fbb2c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.808+0000 7fbb46000640 1 -- 172.21.15.90:0/3485628450 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7fbb4808a2f0 con 0x7fbb501f5c90 2024-04-25T01:17:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.808+0000 7fbb4de00640 1 --2- 172.21.15.90:0/3485628450 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbb2c06be30 0x7fbb2c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.809+0000 7fbb4de00640 1 --2- 172.21.15.90:0/3485628450 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbb2c06be30 0x7fbb2c06e2f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbb4000a780 tx=0x7fbb40009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:27.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:27.813+0000 7fbb46000640 1 -- 172.21.15.90:0/3485628450 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fbb5006cc00 con 0x7fbb501f5c90 2024-04-25T01:17:27.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.011+0000 7fbb54a00640 1 -- 172.21.15.90:0/3485628450 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbb50068fb0 con 0x7fbb501f5c90 2024-04-25T01:17:27.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.012+0000 7fbb46000640 1 -- 172.21.15.90:0/3485628450 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) ==== 104+0+11 (secure 0 0 0) 0x7fbb48058920 con 0x7fbb501f5c90 2024-04-25T01:17:27.998 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:17:28.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.015+0000 7fbb54a00640 1 -- 172.21.15.90:0/3485628450 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbb2c06be30 msgr2=0x7fbb2c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:28.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.015+0000 7fbb54a00640 1 --2- 172.21.15.90:0/3485628450 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbb2c06be30 0x7fbb2c06e2f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbb4000a780 tx=0x7fbb40009030 comp rx=0 tx=0).stop 2024-04-25T01:17:28.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.015+0000 7fbb54a00640 1 -- 172.21.15.90:0/3485628450 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb501f5c90 msgr2=0x7fbb501f6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:28.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.015+0000 7fbb54a00640 1 --2- 172.21.15.90:0/3485628450 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb501f5c90 0x7fbb501f6110 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fbb500b27c0 tx=0x7fbb4800c820 comp rx=0 tx=0).stop 2024-04-25T01:17:28.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.015+0000 7fbb54a00640 1 -- 172.21.15.90:0/3485628450 shutdown_connections 2024-04-25T01:17:28.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.015+0000 7fbb54a00640 1 --2- 172.21.15.90:0/3485628450 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbb2c06be30 0x7fbb2c06e2f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:28.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.015+0000 7fbb54a00640 1 --2- 172.21.15.90:0/3485628450 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb501f5c90 0x7fbb501f6110 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:28.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.015+0000 7fbb54a00640 1 --2- 172.21.15.90:0/3485628450 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb501599f0 0x7fbb501f47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:28.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.015+0000 7fbb54a00640 1 --2- 172.21.15.90:0/3485628450 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb34003a00 0x7fbb500b3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:28.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.015+0000 7fbb54a00640 1 -- 172.21.15.90:0/3485628450 >> 172.21.15.90:0/3485628450 conn(0x7fbb5006b720 msgr2=0x7fbb5015f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:28.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.015+0000 7fbb54a00640 1 -- 172.21.15.90:0/3485628450 shutdown_connections 2024-04-25T01:17:28.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.015+0000 7fbb54a00640 1 -- 172.21.15.90:0/3485628450 wait complete. 2024-04-25T01:17:28.019 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:17:28.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.231+0000 7f002fa00640 1 Processor -- start 2024-04-25T01:17:28.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.231+0000 7f002fa00640 1 -- start start 2024-04-25T01:17:28.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.232+0000 7f002fa00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f00280722e0 0x7f002806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:28.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.233+0000 7f002fa00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f00280cf4d0 0x7f00281600c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:28.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.233+0000 7f002fa00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f002806e410 con 0x7f002815fc40 2024-04-25T01:17:28.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.233+0000 7f002fa00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f0028160aa0 con 0x7f00280cf4d0 2024-04-25T01:17:28.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.233+0000 7f002fa00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f002806da40 con 0x7f00280722e0 2024-04-25T01:17:28.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.233+0000 7f002da00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f00280cf4d0 0x7f00281600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:28.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.233+0000 7f002da00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f00280cf4d0 0x7f00281600c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33266/0 (socket says 172.21.15.90:33266) 2024-04-25T01:17:28.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.233+0000 7f002da00640 1 -- 172.21.15.90:0/289345514 learned_addr learned my addr 172.21.15.90:0/289345514 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:28.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.233+0000 7f0027e00640 1 --1- 172.21.15.90:0/289345514 >> v1:172.21.15.90:6789/0 conn(0x7f002815fc40 0x7f00280cedc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47320/0 (socket says 172.21.15.90:47320) 2024-04-25T01:17:28.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.233+0000 7f002d000640 1 --2- 172.21.15.90:0/289345514 >> v2:172.21.15.103:3300/0 conn(0x7f00280722e0 0x7f002806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:28.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.233+0000 7f002da00640 1 -- 172.21.15.90:0/289345514 >> v2:172.21.15.103:3300/0 conn(0x7f00280722e0 msgr2=0x7f002806ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:28.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.233+0000 7f002da00640 1 --2- 172.21.15.90:0/289345514 >> v2:172.21.15.103:3300/0 conn(0x7f00280722e0 0x7f002806ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:28.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.234+0000 7f002da00640 1 -- 172.21.15.90:0/289345514 >> v1:172.21.15.90:6789/0 conn(0x7f002815fc40 legacy=0x7f00280cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:28.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.234+0000 7f002da00640 1 -- 172.21.15.90:0/289345514 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f002806d110 con 0x7f00280cf4d0 2024-04-25T01:17:28.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.234+0000 7f0027400640 1 -- 172.21.15.90:0/289345514 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3886441785 0 0) 0x7f002806e410 con 0x7f002815fc40 2024-04-25T01:17:28.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.235+0000 7f002da00640 1 --2- 172.21.15.90:0/289345514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00280cf4d0 0x7f00281600c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f00280727b0 tx=0x7f002002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=283187b5f6c40923 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:28.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.235+0000 7f0027400640 1 -- 172.21.15.90:0/289345514 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0020036070 con 0x7f00280cf4d0 2024-04-25T01:17:28.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.235+0000 7f0027400640 1 -- 172.21.15.90:0/289345514 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0020031600 con 0x7f00280cf4d0 2024-04-25T01:17:28.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.235+0000 7f0027400640 1 -- 172.21.15.90:0/289345514 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00200319d0 con 0x7f00280cf4d0 2024-04-25T01:17:28.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.236+0000 7f002fa00640 1 -- 172.21.15.90:0/289345514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00280cf4d0 msgr2=0x7f00281600c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:28.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.236+0000 7f002fa00640 1 --2- 172.21.15.90:0/289345514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00280cf4d0 0x7f00281600c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f00280727b0 tx=0x7f002002a6a0 comp rx=0 tx=0).stop 2024-04-25T01:17:28.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.236+0000 7f002fa00640 1 -- 172.21.15.90:0/289345514 shutdown_connections 2024-04-25T01:17:28.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.236+0000 7f002fa00640 1 --2- 172.21.15.90:0/289345514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00280cf4d0 0x7f00281600c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:28.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.236+0000 7f002fa00640 1 --2- 172.21.15.90:0/289345514 >> v2:172.21.15.103:3300/0 conn(0x7f00280722e0 0x7f002806ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:28.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.236+0000 7f002fa00640 1 -- 172.21.15.90:0/289345514 >> 172.21.15.90:0/289345514 conn(0x7f002806b720 msgr2=0x7f00280b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:28.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.236+0000 7f002fa00640 1 -- 172.21.15.90:0/289345514 shutdown_connections 2024-04-25T01:17:28.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.236+0000 7f002fa00640 1 -- 172.21.15.90:0/289345514 wait complete. 2024-04-25T01:17:28.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.238+0000 7f002fa00640 1 Processor -- start 2024-04-25T01:17:28.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.238+0000 7f002fa00640 1 -- start start 2024-04-25T01:17:28.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.239+0000 7f002fa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f00280722e0 0x7f00281f2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:28.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.239+0000 7f002fa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00280cf4d0 0x7f002815df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:28.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.239+0000 7f002fa00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f002815fc40 0x7f002815e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:28.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.239+0000 7f002fa00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f002806d6d0 con 0x7f00280cf4d0 2024-04-25T01:17:28.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.239+0000 7f002fa00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f002806cec0 con 0x7f002815fc40 2024-04-25T01:17:28.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.239+0000 7f002fa00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0028070b50 con 0x7f00280722e0 2024-04-25T01:17:28.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.239+0000 7f0027e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00280cf4d0 0x7f002815df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:28.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.239+0000 7f0027e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00280cf4d0 0x7f002815df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33268/0 (socket says 172.21.15.90:33268) 2024-04-25T01:17:28.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.239+0000 7f0027e00640 1 -- 172.21.15.90:0/484064398 learned_addr learned my addr 172.21.15.90:0/484064398 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:28.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.239+0000 7f002d000640 1 --2- 172.21.15.90:0/484064398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f00280722e0 0x7f00281f2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:28.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.239+0000 7f0027e00640 1 -- 172.21.15.90:0/484064398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f00280722e0 msgr2=0x7f00281f2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:28.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.239+0000 7f0027e00640 1 --2- 172.21.15.90:0/484064398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f00280722e0 0x7f00281f2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:28.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.239+0000 7f0027e00640 1 -- 172.21.15.90:0/484064398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f002815fc40 msgr2=0x7f002815e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:28.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.239+0000 7f0027e00640 1 --2- 172.21.15.90:0/484064398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f002815fc40 0x7f002815e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:28.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.239+0000 7f0027e00640 1 -- 172.21.15.90:0/484064398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0028133800 con 0x7f00280cf4d0 2024-04-25T01:17:28.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.240+0000 7f0027e00640 1 --2- 172.21.15.90:0/484064398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00280cf4d0 0x7f002815df90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f001400c430 tx=0x7f001400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:28.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.240+0000 7f0024c00640 1 -- 172.21.15.90:0/484064398 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00140074e0 con 0x7f00280cf4d0 2024-04-25T01:17:28.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.240+0000 7f0024c00640 1 -- 172.21.15.90:0/484064398 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00140047e0 con 0x7f00280cf4d0 2024-04-25T01:17:28.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.240+0000 7f0024c00640 1 -- 172.21.15.90:0/484064398 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0014004bd0 con 0x7f00280cf4d0 2024-04-25T01:17:28.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.240+0000 7f002fa00640 1 -- 172.21.15.90:0/484064398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00281430d0 con 0x7f00280cf4d0 2024-04-25T01:17:28.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.240+0000 7f002fa00640 1 -- 172.21.15.90:0/484064398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0028156db0 con 0x7f00280cf4d0 2024-04-25T01:17:28.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.241+0000 7f002fa00640 1 -- 172.21.15.90:0/484064398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0028068fb0 con 0x7f00280cf4d0 2024-04-25T01:17:28.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.242+0000 7f0024c00640 1 -- 172.21.15.90:0/484064398 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0014011070 con 0x7f00280cf4d0 2024-04-25T01:17:28.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.243+0000 7f0024c00640 1 --2- 172.21.15.90:0/484064398 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f000806be30 0x7f000806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:28.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.243+0000 7f0024c00640 1 -- 172.21.15.90:0/484064398 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f001408ada0 con 0x7f00280cf4d0 2024-04-25T01:17:28.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.244+0000 7f002d000640 1 --2- 172.21.15.90:0/484064398 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f000806be30 0x7f000806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:28.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.244+0000 7f002d000640 1 --2- 172.21.15.90:0/484064398 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f000806be30 0x7f000806e2f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f001c006fd0 tx=0x7f001c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:28.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.248+0000 7f0024c00640 1 -- 172.21.15.90:0/484064398 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0028133800 con 0x7f00280cf4d0 2024-04-25T01:17:28.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.446+0000 7f002fa00640 1 -- 172.21.15.90:0/484064398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f00281f3140 con 0x7f00280cf4d0 2024-04-25T01:17:28.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.447+0000 7f0024c00640 1 -- 172.21.15.90:0/484064398 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) ==== 100+0+11 (secure 0 0 0) 0x7f001400e280 con 0x7f00280cf4d0 2024-04-25T01:17:28.432 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-25T01:17:28.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.450+0000 7f002fa00640 1 -- 172.21.15.90:0/484064398 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f000806be30 msgr2=0x7f000806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:28.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.450+0000 7f002fa00640 1 --2- 172.21.15.90:0/484064398 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f000806be30 0x7f000806e2f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f001c006fd0 tx=0x7f001c008040 comp rx=0 tx=0).stop 2024-04-25T01:17:28.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.450+0000 7f002fa00640 1 -- 172.21.15.90:0/484064398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00280cf4d0 msgr2=0x7f002815df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:28.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.450+0000 7f002fa00640 1 --2- 172.21.15.90:0/484064398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00280cf4d0 0x7f002815df90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f001400c430 tx=0x7f001400c8f0 comp rx=0 tx=0).stop 2024-04-25T01:17:28.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.450+0000 7f002fa00640 1 -- 172.21.15.90:0/484064398 shutdown_connections 2024-04-25T01:17:28.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.450+0000 7f002fa00640 1 --2- 172.21.15.90:0/484064398 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f000806be30 0x7f000806e2f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:28.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.450+0000 7f002fa00640 1 --2- 172.21.15.90:0/484064398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f002815fc40 0x7f002815e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:28.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.450+0000 7f002fa00640 1 --2- 172.21.15.90:0/484064398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00280cf4d0 0x7f002815df90 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:28.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.450+0000 7f002fa00640 1 --2- 172.21.15.90:0/484064398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f00280722e0 0x7f00281f2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:28.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.450+0000 7f002fa00640 1 -- 172.21.15.90:0/484064398 >> 172.21.15.90:0/484064398 conn(0x7f002806b720 msgr2=0x7f0028070e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:28.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.450+0000 7f002fa00640 1 -- 172.21.15.90:0/484064398 shutdown_connections 2024-04-25T01:17:28.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.450+0000 7f002fa00640 1 -- 172.21.15.90:0/484064398 wait complete. 2024-04-25T01:17:28.453 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:17:28.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.670+0000 7fca6be00640 1 Processor -- start 2024-04-25T01:17:28.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.670+0000 7fca6be00640 1 -- start start 2024-04-25T01:17:28.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.672+0000 7fca6be00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fca6414c0f0 0x7fca640b3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:28.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.672+0000 7fca6be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca64162d10 0x7fca6406c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:28.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.672+0000 7fca6be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fca6406cf30 0x7fca64166200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:28.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.672+0000 7fca6be00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fca6406d3a0 con 0x7fca6406cf30 2024-04-25T01:17:28.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.672+0000 7fca6be00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fca640b3c70 con 0x7fca6414c0f0 2024-04-25T01:17:28.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.672+0000 7fca6be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fca640b3f50 con 0x7fca64162d10 2024-04-25T01:17:28.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.673+0000 7fca69e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fca6406cf30 0x7fca64166200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:28.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.673+0000 7fca69e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fca6406cf30 0x7fca64166200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33276/0 (socket says 172.21.15.90:33276) 2024-04-25T01:17:28.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.673+0000 7fca69e00640 1 -- 172.21.15.90:0/4094106646 learned_addr learned my addr 172.21.15.90:0/4094106646 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:28.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.673+0000 7fca68a00640 1 --2- 172.21.15.90:0/4094106646 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca64162d10 0x7fca6406c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:28.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.673+0000 7fca69400640 1 --2- 172.21.15.90:0/4094106646 >> v2:172.21.15.103:3300/0 conn(0x7fca6414c0f0 0x7fca640b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:28.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.673+0000 7fca69e00640 1 -- 172.21.15.90:0/4094106646 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca64162d10 msgr2=0x7fca6406c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:28.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.673+0000 7fca69e00640 1 --2- 172.21.15.90:0/4094106646 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca64162d10 0x7fca6406c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:28.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.673+0000 7fca69e00640 1 -- 172.21.15.90:0/4094106646 >> v2:172.21.15.103:3300/0 conn(0x7fca6414c0f0 msgr2=0x7fca640b3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:28.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.673+0000 7fca69e00640 1 --2- 172.21.15.90:0/4094106646 >> v2:172.21.15.103:3300/0 conn(0x7fca6414c0f0 0x7fca640b3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:28.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.673+0000 7fca69e00640 1 -- 172.21.15.90:0/4094106646 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca640b4230 con 0x7fca6406cf30 2024-04-25T01:17:28.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.674+0000 7fca69e00640 1 --2- 172.21.15.90:0/4094106646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca6406cf30 0x7fca64166200 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fca640b3e80 tx=0x7fca5c02a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=440dc07d357ae789 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:28.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.675+0000 7fca63e00640 1 -- 172.21.15.90:0/4094106646 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca5c036070 con 0x7fca6406cf30 2024-04-25T01:17:28.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.675+0000 7fca63e00640 1 -- 172.21.15.90:0/4094106646 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca5c031600 con 0x7fca6406cf30 2024-04-25T01:17:28.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.675+0000 7fca63e00640 1 -- 172.21.15.90:0/4094106646 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca5c0319d0 con 0x7fca6406cf30 2024-04-25T01:17:28.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.675+0000 7fca6be00640 1 -- 172.21.15.90:0/4094106646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca6406cf30 msgr2=0x7fca64166200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:28.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.675+0000 7fca6be00640 1 --2- 172.21.15.90:0/4094106646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca6406cf30 0x7fca64166200 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fca640b3e80 tx=0x7fca5c02a5f0 comp rx=0 tx=0).stop 2024-04-25T01:17:28.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.675+0000 7fca6be00640 1 -- 172.21.15.90:0/4094106646 shutdown_connections 2024-04-25T01:17:28.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.675+0000 7fca6be00640 1 --2- 172.21.15.90:0/4094106646 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca64162d10 0x7fca6406c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:28.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.675+0000 7fca6be00640 1 --2- 172.21.15.90:0/4094106646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca6406cf30 0x7fca64166200 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:28.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.675+0000 7fca6be00640 1 --2- 172.21.15.90:0/4094106646 >> v2:172.21.15.103:3300/0 conn(0x7fca6414c0f0 0x7fca640b3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:28.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.675+0000 7fca6be00640 1 -- 172.21.15.90:0/4094106646 >> 172.21.15.90:0/4094106646 conn(0x7fca6406b720 msgr2=0x7fca640b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:28.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.676+0000 7fca6be00640 1 -- 172.21.15.90:0/4094106646 shutdown_connections 2024-04-25T01:17:28.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.676+0000 7fca6be00640 1 -- 172.21.15.90:0/4094106646 wait complete. 2024-04-25T01:17:28.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.677+0000 7fca6be00640 1 Processor -- start 2024-04-25T01:17:28.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.677+0000 7fca6be00640 1 -- start start 2024-04-25T01:17:28.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.678+0000 7fca6be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca6406cf30 0x7fca64162080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:28.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.678+0000 7fca6be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca6414c0f0 0x7fca640ad940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:28.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.678+0000 7fca6be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca64162d10 0x7fca64071b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:28.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.678+0000 7fca6be00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fca640b3860 con 0x7fca64162d10 2024-04-25T01:17:28.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.678+0000 7fca6be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fca640b47f0 con 0x7fca6406cf30 2024-04-25T01:17:28.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.678+0000 7fca6be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fca640b39e0 con 0x7fca6414c0f0 2024-04-25T01:17:28.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.679+0000 7fca69e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca64162d10 0x7fca64071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:28.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.679+0000 7fca69e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca64162d10 0x7fca64071b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33278/0 (socket says 172.21.15.90:33278) 2024-04-25T01:17:28.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.679+0000 7fca69e00640 1 -- 172.21.15.90:0/1006554119 learned_addr learned my addr 172.21.15.90:0/1006554119 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:28.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.679+0000 7fca69400640 1 --2- 172.21.15.90:0/1006554119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca6406cf30 0x7fca64162080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:28.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.679+0000 7fca68a00640 1 --2- 172.21.15.90:0/1006554119 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca6414c0f0 0x7fca640ad940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:28.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.679+0000 7fca69e00640 1 -- 172.21.15.90:0/1006554119 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca6414c0f0 msgr2=0x7fca640ad940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:28.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.679+0000 7fca69e00640 1 --2- 172.21.15.90:0/1006554119 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca6414c0f0 0x7fca640ad940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:28.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.679+0000 7fca69e00640 1 -- 172.21.15.90:0/1006554119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca6406cf30 msgr2=0x7fca64162080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:28.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.679+0000 7fca69e00640 1 --2- 172.21.15.90:0/1006554119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca6406cf30 0x7fca64162080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:28.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.679+0000 7fca69e00640 1 -- 172.21.15.90:0/1006554119 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca641f8fd0 con 0x7fca64162d10 2024-04-25T01:17:28.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.679+0000 7fca69e00640 1 --2- 172.21.15.90:0/1006554119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca64162d10 0x7fca64071b90 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fca5c009f10 tx=0x7fca5c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:28.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.680+0000 7fca61600640 1 -- 172.21.15.90:0/1006554119 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca5c036050 con 0x7fca64162d10 2024-04-25T01:17:28.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.680+0000 7fca61600640 1 -- 172.21.15.90:0/1006554119 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca5c042070 con 0x7fca64162d10 2024-04-25T01:17:28.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.680+0000 7fca61600640 1 -- 172.21.15.90:0/1006554119 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca5c009510 con 0x7fca64162d10 2024-04-25T01:17:28.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.680+0000 7fca6be00640 1 -- 172.21.15.90:0/1006554119 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca640a9670 con 0x7fca64162d10 2024-04-25T01:17:28.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.680+0000 7fca6be00640 1 -- 172.21.15.90:0/1006554119 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca640a9c30 con 0x7fca64162d10 2024-04-25T01:17:28.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.681+0000 7fca6be00640 1 -- 172.21.15.90:0/1006554119 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca64068fb0 con 0x7fca64162d10 2024-04-25T01:17:28.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.682+0000 7fca61600640 1 -- 172.21.15.90:0/1006554119 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fca5c007bf0 con 0x7fca64162d10 2024-04-25T01:17:28.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.683+0000 7fca61600640 1 --2- 172.21.15.90:0/1006554119 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fca4406be30 0x7fca4406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:28.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.683+0000 7fca61600640 1 -- 172.21.15.90:0/1006554119 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7fca5c0ac0b0 con 0x7fca64162d10 2024-04-25T01:17:28.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.683+0000 7fca69400640 1 --2- 172.21.15.90:0/1006554119 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fca4406be30 0x7fca4406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:28.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.684+0000 7fca69400640 1 --2- 172.21.15.90:0/1006554119 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fca4406be30 0x7fca4406e2f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fca58006fd0 tx=0x7fca58008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:28.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.687+0000 7fca61600640 1 -- 172.21.15.90:0/1006554119 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fca5c07a650 con 0x7fca64162d10 2024-04-25T01:17:28.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.886+0000 7fca6be00640 1 -- 172.21.15.90:0/1006554119 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fca64068180 con 0x7fca64162d10 2024-04-25T01:17:28.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.886+0000 7fca61600640 1 -- 172.21.15.90:0/1006554119 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) ==== 103+0+11 (secure 0 0 0) 0x7fca5c079f50 con 0x7fca64162d10 2024-04-25T01:17:28.872 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:17:28.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.889+0000 7fca6be00640 1 -- 172.21.15.90:0/1006554119 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fca4406be30 msgr2=0x7fca4406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:28.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.889+0000 7fca6be00640 1 --2- 172.21.15.90:0/1006554119 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fca4406be30 0x7fca4406e2f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fca58006fd0 tx=0x7fca58008040 comp rx=0 tx=0).stop 2024-04-25T01:17:28.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.889+0000 7fca6be00640 1 -- 172.21.15.90:0/1006554119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca64162d10 msgr2=0x7fca64071b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:28.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.889+0000 7fca6be00640 1 --2- 172.21.15.90:0/1006554119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca64162d10 0x7fca64071b90 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fca5c009f10 tx=0x7fca5c004a40 comp rx=0 tx=0).stop 2024-04-25T01:17:28.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.889+0000 7fca6be00640 1 -- 172.21.15.90:0/1006554119 shutdown_connections 2024-04-25T01:17:28.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.890+0000 7fca6be00640 1 --2- 172.21.15.90:0/1006554119 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fca4406be30 0x7fca4406e2f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:28.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.890+0000 7fca6be00640 1 --2- 172.21.15.90:0/1006554119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca64162d10 0x7fca64071b90 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:28.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.890+0000 7fca6be00640 1 --2- 172.21.15.90:0/1006554119 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca6414c0f0 0x7fca640ad940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:28.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.890+0000 7fca6be00640 1 --2- 172.21.15.90:0/1006554119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca6406cf30 0x7fca64162080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:28.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.890+0000 7fca6be00640 1 -- 172.21.15.90:0/1006554119 >> 172.21.15.90:0/1006554119 conn(0x7fca6406b720 msgr2=0x7fca64070df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:28.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.890+0000 7fca6be00640 1 -- 172.21.15.90:0/1006554119 shutdown_connections 2024-04-25T01:17:28.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:28.890+0000 7fca6be00640 1 -- 172.21.15.90:0/1006554119 wait complete. 2024-04-25T01:17:28.908 INFO:tasks.cephfs_test_runner:Starting test: test_rebuild_simple (tasks.cephfs.test_recovery_pool.TestRecoveryPool) 2024-04-25T01:17:28.908 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:17:29.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.130+0000 7fea95e00640 1 Processor -- start 2024-04-25T01:17:29.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.130+0000 7fea95e00640 1 -- start start 2024-04-25T01:17:29.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.131+0000 7fea95e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fea900cf520 0x7fea900ae730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:29.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.131+0000 7fea95e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea9006d910 0x7fea900aec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:29.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.131+0000 7fea95e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea9006ba90 con 0x7fea9015ad40 2024-04-25T01:17:29.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.131+0000 7fea95e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fea9006b830 con 0x7fea900cf520 2024-04-25T01:17:29.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.131+0000 7fea95e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fea9006cb50 con 0x7fea9006d910 2024-04-25T01:17:29.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.132+0000 7fea95400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea9006d910 0x7fea900aec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:29.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.132+0000 7fea95400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea9006d910 0x7fea900aec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.90:41496/0 (socket says 172.21.15.90:41496) 2024-04-25T01:17:29.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.132+0000 7fea95400640 1 -- 172.21.15.90:0/302680615 learned_addr learned my addr 172.21.15.90:0/302680615 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:29.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.132+0000 7fea87e00640 1 --2- 172.21.15.90:0/302680615 >> v2:172.21.15.103:3300/0 conn(0x7fea900cf520 0x7fea900ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:29.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.133+0000 7fea95400640 1 -- 172.21.15.90:0/302680615 >> v2:172.21.15.103:3300/0 conn(0x7fea900cf520 msgr2=0x7fea900ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:29.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.133+0000 7fea95400640 1 --2- 172.21.15.90:0/302680615 >> v2:172.21.15.103:3300/0 conn(0x7fea900cf520 0x7fea900ae730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:29.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.133+0000 7fea95400640 1 -- 172.21.15.90:0/302680615 >> v1:172.21.15.103:6789/0 conn(0x7fea9015ad40 legacy=0x7fea9006b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:29.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.133+0000 7fea95400640 1 -- 172.21.15.90:0/302680615 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea9006ce30 con 0x7fea9006d910 2024-04-25T01:17:29.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.134+0000 7fea95400640 1 --2- 172.21.15.90:0/302680615 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea9006d910 0x7fea900aec70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fea88009e50 tx=0x7fea8802a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55976d10ba96984f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:29.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.134+0000 7fea8fe00640 1 -- 172.21.15.90:0/302680615 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea88036070 con 0x7fea9006d910 2024-04-25T01:17:29.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.134+0000 7fea8fe00640 1 -- 172.21.15.90:0/302680615 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea88031600 con 0x7fea9006d910 2024-04-25T01:17:29.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.134+0000 7fea8fe00640 1 -- 172.21.15.90:0/302680615 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea880319d0 con 0x7fea9006d910 2024-04-25T01:17:29.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.135+0000 7fea95e00640 1 -- 172.21.15.90:0/302680615 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea9006d910 msgr2=0x7fea900aec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:29.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.135+0000 7fea95e00640 1 --2- 172.21.15.90:0/302680615 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea9006d910 0x7fea900aec70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fea88009e50 tx=0x7fea8802a710 comp rx=0 tx=0).stop 2024-04-25T01:17:29.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.135+0000 7fea95e00640 1 -- 172.21.15.90:0/302680615 shutdown_connections 2024-04-25T01:17:29.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.135+0000 7fea95e00640 1 --2- 172.21.15.90:0/302680615 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea9006d910 0x7fea900aec70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:29.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.135+0000 7fea95e00640 1 --2- 172.21.15.90:0/302680615 >> v2:172.21.15.103:3300/0 conn(0x7fea900cf520 0x7fea900ae730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:29.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.135+0000 7fea95e00640 1 -- 172.21.15.90:0/302680615 >> 172.21.15.90:0/302680615 conn(0x7fea900b1660 msgr2=0x7fea900b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:29.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.135+0000 7fea95e00640 1 -- 172.21.15.90:0/302680615 shutdown_connections 2024-04-25T01:17:29.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.135+0000 7fea95e00640 1 -- 172.21.15.90:0/302680615 wait complete. 2024-04-25T01:17:29.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.137+0000 7fea95e00640 1 Processor -- start 2024-04-25T01:17:29.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.137+0000 7fea95e00640 1 -- start start 2024-04-25T01:17:29.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.138+0000 7fea95e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea9006d910 0x7fea9014fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:29.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.138+0000 7fea95e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea900cf520 0x7fea90153d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:29.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.138+0000 7fea95e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea9015ad40 0x7fea90152d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:29.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.138+0000 7fea95e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fea900cd870 con 0x7fea900cf520 2024-04-25T01:17:29.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.138+0000 7fea95e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fea9006d3f0 con 0x7fea9015ad40 2024-04-25T01:17:29.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.138+0000 7fea95e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fea900cd9f0 con 0x7fea9006d910 2024-04-25T01:17:29.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.138+0000 7fea87e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea900cf520 0x7fea90153d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:29.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.138+0000 7fea87e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea900cf520 0x7fea90153d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33284/0 (socket says 172.21.15.90:33284) 2024-04-25T01:17:29.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.138+0000 7fea87e00640 1 -- 172.21.15.90:0/2331380998 learned_addr learned my addr 172.21.15.90:0/2331380998 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:29.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.139+0000 7fea94a00640 1 --2- 172.21.15.90:0/2331380998 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea9006d910 0x7fea9014fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:29.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.139+0000 7fea95400640 1 --2- 172.21.15.90:0/2331380998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea9015ad40 0x7fea90152d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:29.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.139+0000 7fea87e00640 1 -- 172.21.15.90:0/2331380998 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea9006d910 msgr2=0x7fea9014fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:29.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.139+0000 7fea87e00640 1 --2- 172.21.15.90:0/2331380998 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea9006d910 0x7fea9014fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:29.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.139+0000 7fea87e00640 1 -- 172.21.15.90:0/2331380998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea9015ad40 msgr2=0x7fea90152d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:29.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.139+0000 7fea87e00640 1 --2- 172.21.15.90:0/2331380998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea9015ad40 0x7fea90152d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:29.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.139+0000 7fea87e00640 1 -- 172.21.15.90:0/2331380998 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea9015d710 con 0x7fea900cf520 2024-04-25T01:17:29.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.139+0000 7fea95400640 1 --2- 172.21.15.90:0/2331380998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea9015ad40 0x7fea90152d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:29.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.139+0000 7fea87e00640 1 --2- 172.21.15.90:0/2331380998 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea900cf520 0x7fea90153d40 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fea7800e9e0 tx=0x7fea7800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:29.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.139+0000 7fea8d600640 1 -- 172.21.15.90:0/2331380998 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea78017070 con 0x7fea900cf520 2024-04-25T01:17:29.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.139+0000 7fea8d600640 1 -- 172.21.15.90:0/2331380998 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea7800c9d0 con 0x7fea900cf520 2024-04-25T01:17:29.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.139+0000 7fea8d600640 1 -- 172.21.15.90:0/2331380998 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea7800cdc0 con 0x7fea900cf520 2024-04-25T01:17:29.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.140+0000 7fea95e00640 1 -- 172.21.15.90:0/2331380998 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea9015d9f0 con 0x7fea900cf520 2024-04-25T01:17:29.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.140+0000 7fea95e00640 1 -- 172.21.15.90:0/2331380998 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea90152840 con 0x7fea900cf520 2024-04-25T01:17:29.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.141+0000 7fea95e00640 1 -- 172.21.15.90:0/2331380998 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea90075350 con 0x7fea900cf520 2024-04-25T01:17:29.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.142+0000 7fea8d600640 1 -- 172.21.15.90:0/2331380998 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fea7801a070 con 0x7fea900cf520 2024-04-25T01:17:29.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.143+0000 7fea8d600640 1 --2- 172.21.15.90:0/2331380998 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fea6c06be30 0x7fea6c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:29.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.143+0000 7fea8d600640 1 -- 172.21.15.90:0/2331380998 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7fea7808b5b0 con 0x7fea900cf520 2024-04-25T01:17:29.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.143+0000 7fea94a00640 1 --2- 172.21.15.90:0/2331380998 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fea6c06be30 0x7fea6c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:29.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.144+0000 7fea94a00640 1 --2- 172.21.15.90:0/2331380998 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fea6c06be30 0x7fea6c06e2f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fea80007fd0 tx=0x7fea80009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:29.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.147+0000 7fea8d600640 1 -- 172.21.15.90:0/2331380998 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fea78059be0 con 0x7fea900cf520 2024-04-25T01:17:29.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.345+0000 7fea95e00640 1 -- 172.21.15.90:0/2331380998 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fea900af2b0 con 0x7fea900cf520 2024-04-25T01:17:29.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.348+0000 7fea8d600640 1 -- 172.21.15.90:0/2331380998 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18576 (secure 0 0 0) 0x7fea780594e0 con 0x7fea900cf520 2024-04-25T01:17:29.334 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:17:29.334 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:17:09.346008+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:16:20.094512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:16:21.078268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:16:24.670642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.e","mappings":[{"from":0,"to":7}]},{"pgid":"4.f","mappings":[{"from":7,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:17:29.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.351+0000 7fea95e00640 1 -- 172.21.15.90:0/2331380998 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fea6c06be30 msgr2=0x7fea6c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:29.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.351+0000 7fea95e00640 1 --2- 172.21.15.90:0/2331380998 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fea6c06be30 0x7fea6c06e2f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fea80007fd0 tx=0x7fea80009040 comp rx=0 tx=0).stop 2024-04-25T01:17:29.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.351+0000 7fea95e00640 1 -- 172.21.15.90:0/2331380998 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea900cf520 msgr2=0x7fea90153d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:29.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.351+0000 7fea95e00640 1 --2- 172.21.15.90:0/2331380998 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea900cf520 0x7fea90153d40 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fea7800e9e0 tx=0x7fea7800eea0 comp rx=0 tx=0).stop 2024-04-25T01:17:29.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.351+0000 7fea95e00640 1 -- 172.21.15.90:0/2331380998 shutdown_connections 2024-04-25T01:17:29.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.351+0000 7fea95e00640 1 --2- 172.21.15.90:0/2331380998 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fea6c06be30 0x7fea6c06e2f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:29.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.351+0000 7fea95e00640 1 --2- 172.21.15.90:0/2331380998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea9015ad40 0x7fea90152d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:29.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.351+0000 7fea95e00640 1 --2- 172.21.15.90:0/2331380998 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea900cf520 0x7fea90153d40 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:29.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.351+0000 7fea95e00640 1 --2- 172.21.15.90:0/2331380998 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fea9006d910 0x7fea9014fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:29.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.351+0000 7fea95e00640 1 -- 172.21.15.90:0/2331380998 >> 172.21.15.90:0/2331380998 conn(0x7fea900b1660 msgr2=0x7fea9006b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:29.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.351+0000 7fea95e00640 1 -- 172.21.15.90:0/2331380998 shutdown_connections 2024-04-25T01:17:29.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.351+0000 7fea95e00640 1 -- 172.21.15.90:0/2331380998 wait complete. 2024-04-25T01:17:29.355 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:16:15.894126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:16:20.094512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:16:21.078268+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:16:24.670642+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:17:29.356 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:17:29.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.574+0000 7f5717800640 1 Processor -- start 2024-04-25T01:17:29.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.575+0000 7f5717800640 1 -- start start 2024-04-25T01:17:29.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.576+0000 7f5717800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f571015ce70 0x7f5710071c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:29.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.576+0000 7f5717800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f571015d250 0x7f5710160530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:29.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.576+0000 7f5717800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5710160a70 con 0x7f571006b120 2024-04-25T01:17:29.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.576+0000 7f5717800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f5710072230 con 0x7f571015d250 2024-04-25T01:17:29.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.576+0000 7f5717800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f5710072510 con 0x7f571015ce70 2024-04-25T01:17:29.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.577+0000 7f570fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f571015d250 0x7f5710160530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:29.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.577+0000 7f570fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f571015d250 0x7f5710160530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33294/0 (socket says 172.21.15.90:33294) 2024-04-25T01:17:29.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.577+0000 7f570fe00640 1 -- 172.21.15.90:0/1317670210 learned_addr learned my addr 172.21.15.90:0/1317670210 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:29.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.577+0000 7f5714e00640 1 --2- 172.21.15.90:0/1317670210 >> v2:172.21.15.103:3300/0 conn(0x7f571015ce70 0x7f5710071c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:29.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.577+0000 7f570fe00640 1 -- 172.21.15.90:0/1317670210 >> v2:172.21.15.103:3300/0 conn(0x7f571015ce70 msgr2=0x7f5710071c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:29.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.577+0000 7f570fe00640 1 --2- 172.21.15.90:0/1317670210 >> v2:172.21.15.103:3300/0 conn(0x7f571015ce70 0x7f5710071c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:29.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.577+0000 7f570fe00640 1 -- 172.21.15.90:0/1317670210 >> v1:172.21.15.103:6789/0 conn(0x7f571006b120 legacy=0x7f57100cfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:29.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.577+0000 7f570fe00640 1 -- 172.21.15.90:0/1317670210 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57100727f0 con 0x7f571015d250 2024-04-25T01:17:29.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.578+0000 7f570fe00640 1 --2- 172.21.15.90:0/1317670210 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f571015d250 0x7f5710160530 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f57100717e0 tx=0x7f56fc02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30dc6935e98dc107 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:29.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.578+0000 7f570f400640 1 -- 172.21.15.90:0/1317670210 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56fc037070 con 0x7f571015d250 2024-04-25T01:17:29.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.579+0000 7f570f400640 1 -- 172.21.15.90:0/1317670210 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56fc032440 con 0x7f571015d250 2024-04-25T01:17:29.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.579+0000 7f570f400640 1 -- 172.21.15.90:0/1317670210 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56fc032810 con 0x7f571015d250 2024-04-25T01:17:29.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.579+0000 7f5717800640 1 -- 172.21.15.90:0/1317670210 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f571015d250 msgr2=0x7f5710160530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:29.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.579+0000 7f5717800640 1 --2- 172.21.15.90:0/1317670210 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f571015d250 0x7f5710160530 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f57100717e0 tx=0x7f56fc02a230 comp rx=0 tx=0).stop 2024-04-25T01:17:29.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.579+0000 7f5717800640 1 -- 172.21.15.90:0/1317670210 shutdown_connections 2024-04-25T01:17:29.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.579+0000 7f5717800640 1 --2- 172.21.15.90:0/1317670210 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f571015d250 0x7f5710160530 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:29.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.579+0000 7f5717800640 1 --2- 172.21.15.90:0/1317670210 >> v2:172.21.15.103:3300/0 conn(0x7f571015ce70 0x7f5710071c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:29.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.579+0000 7f5717800640 1 -- 172.21.15.90:0/1317670210 >> 172.21.15.90:0/1317670210 conn(0x7f57100d0b70 msgr2=0x7f57100d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:29.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.579+0000 7f5717800640 1 -- 172.21.15.90:0/1317670210 shutdown_connections 2024-04-25T01:17:29.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.579+0000 7f5717800640 1 -- 172.21.15.90:0/1317670210 wait complete. 2024-04-25T01:17:29.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.581+0000 7f5717800640 1 Processor -- start 2024-04-25T01:17:29.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.581+0000 7f5717800640 1 -- start start 2024-04-25T01:17:29.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.582+0000 7f5717800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f571006b120 0x7f5710158710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:29.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.582+0000 7f5717800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f571015ce70 0x7f57101500c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:29.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.582+0000 7f5717800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f571015d250 0x7f57101f5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:29.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.582+0000 7f5717800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f57100cf5e0 con 0x7f571006b120 2024-04-25T01:17:29.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.582+0000 7f5717800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f57100cf460 con 0x7f571015d250 2024-04-25T01:17:29.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.582+0000 7f5717800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f571013f5c0 con 0x7f571015ce70 2024-04-25T01:17:29.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.582+0000 7f5714e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f571006b120 0x7f5710158710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:29.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.582+0000 7f5714e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f571006b120 0x7f5710158710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33298/0 (socket says 172.21.15.90:33298) 2024-04-25T01:17:29.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.582+0000 7f5714e00640 1 -- 172.21.15.90:0/2710529280 learned_addr learned my addr 172.21.15.90:0/2710529280 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:29.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.583+0000 7f5714e00640 1 -- 172.21.15.90:0/2710529280 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f571015ce70 msgr2=0x7f57101500c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:29.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.583+0000 7f5714e00640 1 --2- 172.21.15.90:0/2710529280 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f571015ce70 0x7f57101500c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:29.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.583+0000 7f5714e00640 1 -- 172.21.15.90:0/2710529280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f571015d250 msgr2=0x7f57101f5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:29.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.583+0000 7f5714e00640 1 --2- 172.21.15.90:0/2710529280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f571015d250 0x7f57101f5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:29.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.583+0000 7f5714e00640 1 -- 172.21.15.90:0/2710529280 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5710153230 con 0x7f571006b120 2024-04-25T01:17:29.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.583+0000 7f5714e00640 1 --2- 172.21.15.90:0/2710529280 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f571006b120 0x7f5710158710 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f570000d800 tx=0x7f570000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:29.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.583+0000 7f570cc00640 1 -- 172.21.15.90:0/2710529280 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570000b6c0 con 0x7f571006b120 2024-04-25T01:17:29.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.583+0000 7f570cc00640 1 -- 172.21.15.90:0/2710529280 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5700019070 con 0x7f571006b120 2024-04-25T01:17:29.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.583+0000 7f5717800640 1 -- 172.21.15.90:0/2710529280 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57101e6940 con 0x7f571006b120 2024-04-25T01:17:29.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.583+0000 7f570cc00640 1 -- 172.21.15.90:0/2710529280 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570000ba80 con 0x7f571006b120 2024-04-25T01:17:29.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.583+0000 7f5717800640 1 -- 172.21.15.90:0/2710529280 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57101e6f00 con 0x7f571006b120 2024-04-25T01:17:29.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.584+0000 7f5717800640 1 -- 172.21.15.90:0/2710529280 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57100b36d0 con 0x7f571006b120 2024-04-25T01:17:29.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.585+0000 7f570cc00640 1 -- 172.21.15.90:0/2710529280 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f570000bc20 con 0x7f571006b120 2024-04-25T01:17:29.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.586+0000 7f570cc00640 1 --2- 172.21.15.90:0/2710529280 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f56f006be30 0x7f56f006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:29.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.587+0000 7f570cc00640 1 -- 172.21.15.90:0/2710529280 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f570008b6a0 con 0x7f571006b120 2024-04-25T01:17:29.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.587+0000 7f570fe00640 1 --2- 172.21.15.90:0/2710529280 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f56f006be30 0x7f56f006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:29.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.587+0000 7f570fe00640 1 --2- 172.21.15.90:0/2710529280 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f56f006be30 0x7f56f006e2f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f56fc034000 tx=0x7f56fc02c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:29.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.591+0000 7f570cc00640 1 -- 172.21.15.90:0/2710529280 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f57100b36d0 con 0x7f571006b120 2024-04-25T01:17:29.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.786+0000 7f5717800640 1 -- 172.21.15.90:0/2710529280 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5710068180 con 0x7f571006b120 2024-04-25T01:17:29.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.787+0000 7f570cc00640 1 -- 172.21.15.90:0/2710529280 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) ==== 93+0+10 (secure 0 0 0) 0x7f5700059ce0 con 0x7f571006b120 2024-04-25T01:17:29.773 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T01:17:29.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.790+0000 7f5717800640 1 -- 172.21.15.90:0/2710529280 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f56f006be30 msgr2=0x7f56f006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:29.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.790+0000 7f5717800640 1 --2- 172.21.15.90:0/2710529280 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f56f006be30 0x7f56f006e2f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f56fc034000 tx=0x7f56fc02c000 comp rx=0 tx=0).stop 2024-04-25T01:17:29.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.790+0000 7f5717800640 1 -- 172.21.15.90:0/2710529280 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f571006b120 msgr2=0x7f5710158710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:29.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.790+0000 7f5717800640 1 --2- 172.21.15.90:0/2710529280 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f571006b120 0x7f5710158710 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f570000d800 tx=0x7f570000dcc0 comp rx=0 tx=0).stop 2024-04-25T01:17:29.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.790+0000 7f5717800640 1 -- 172.21.15.90:0/2710529280 shutdown_connections 2024-04-25T01:17:29.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.790+0000 7f5717800640 1 --2- 172.21.15.90:0/2710529280 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f56f006be30 0x7f56f006e2f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:29.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.790+0000 7f5717800640 1 --2- 172.21.15.90:0/2710529280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f571015d250 0x7f57101f5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:29.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.790+0000 7f5717800640 1 --2- 172.21.15.90:0/2710529280 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f571015ce70 0x7f57101500c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:29.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.790+0000 7f5717800640 1 --2- 172.21.15.90:0/2710529280 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f571006b120 0x7f5710158710 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:29.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.790+0000 7f5717800640 1 -- 172.21.15.90:0/2710529280 >> 172.21.15.90:0/2710529280 conn(0x7f57100d0b70 msgr2=0x7f571015de70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:29.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.790+0000 7f5717800640 1 -- 172.21.15.90:0/2710529280 shutdown_connections 2024-04-25T01:17:29.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:29.790+0000 7f5717800640 1 -- 172.21.15.90:0/2710529280 wait complete. 2024-04-25T01:17:29.795 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:17:29.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.006+0000 7f223c600640 1 Processor -- start 2024-04-25T01:17:29.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.006+0000 7f223c600640 1 -- start start 2024-04-25T01:17:29.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.007+0000 7f223c600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f22340722e0 0x7f223406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:29.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.008+0000 7f223c600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22340cf4d0 0x7f22341600c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:29.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.008+0000 7f223c600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f223406e410 con 0x7f223415fc40 2024-04-25T01:17:29.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.008+0000 7f223c600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f2234160aa0 con 0x7f22340722e0 2024-04-25T01:17:29.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.008+0000 7f223c600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f223406da40 con 0x7f22340cf4d0 2024-04-25T01:17:29.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.008+0000 7f2239c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f22340722e0 0x7f223406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:29.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.008+0000 7f2239200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f223415fc40 0x7f22340cedc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47334/0 (socket says 172.21.15.90:47334) 2024-04-25T01:17:29.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.008+0000 7f2239200640 1 -- 172.21.15.90:0/3177113315 learned_addr learned my addr 172.21.15.90:0/3177113315 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:29.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.009+0000 7f2239c00640 1 -- 172.21.15.90:0/3177113315 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22340cf4d0 msgr2=0x7f22341600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:29.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.008+0000 7f223a600640 1 --2- 172.21.15.90:0/3177113315 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22340cf4d0 0x7f22341600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:29.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.009+0000 7f2233e00640 1 -- 172.21.15.90:0/3177113315 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 579715133 0 0) 0x7f223406e410 con 0x7f223415fc40 2024-04-25T01:17:29.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.009+0000 7f2239c00640 1 --2- 172.21.15.90:0/3177113315 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22340cf4d0 0x7f22341600c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:29.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.009+0000 7f2239c00640 1 -- 172.21.15.90:0/3177113315 >> v1:172.21.15.90:6789/0 conn(0x7f223415fc40 legacy=0x7f22340cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:29.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.009+0000 7f2239c00640 1 -- 172.21.15.90:0/3177113315 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f223406d110 con 0x7f22340722e0 2024-04-25T01:17:29.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.009+0000 7f223a600640 1 --2- 172.21.15.90:0/3177113315 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22340cf4d0 0x7f22341600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:29.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.010+0000 7f2239c00640 1 --2- 172.21.15.90:0/3177113315 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22340722e0 0x7f223406ded0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f22280097a0 tx=0x7f222802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e584e2c9587a26aa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:29.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.010+0000 7f2233e00640 1 -- 172.21.15.90:0/3177113315 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2228037070 con 0x7f22340722e0 2024-04-25T01:17:29.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.010+0000 7f2233e00640 1 -- 172.21.15.90:0/3177113315 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f222802ae50 con 0x7f22340722e0 2024-04-25T01:17:29.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.010+0000 7f2233e00640 1 -- 172.21.15.90:0/3177113315 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2228032660 con 0x7f22340722e0 2024-04-25T01:17:29.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.011+0000 7f223c600640 1 -- 172.21.15.90:0/3177113315 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22340722e0 msgr2=0x7f223406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:29.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.011+0000 7f223c600640 1 --2- 172.21.15.90:0/3177113315 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22340722e0 0x7f223406ded0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f22280097a0 tx=0x7f222802a100 comp rx=0 tx=0).stop 2024-04-25T01:17:29.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.011+0000 7f223c600640 1 -- 172.21.15.90:0/3177113315 shutdown_connections 2024-04-25T01:17:29.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.011+0000 7f223c600640 1 --2- 172.21.15.90:0/3177113315 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22340cf4d0 0x7f22341600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:29.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.011+0000 7f223c600640 1 --2- 172.21.15.90:0/3177113315 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22340722e0 0x7f223406ded0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:29.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.011+0000 7f223c600640 1 -- 172.21.15.90:0/3177113315 >> 172.21.15.90:0/3177113315 conn(0x7f223406b720 msgr2=0x7f22340b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:29.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.011+0000 7f223c600640 1 -- 172.21.15.90:0/3177113315 shutdown_connections 2024-04-25T01:17:29.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.011+0000 7f223c600640 1 -- 172.21.15.90:0/3177113315 wait complete. 2024-04-25T01:17:29.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.013+0000 7f223c600640 1 Processor -- start 2024-04-25T01:17:29.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.013+0000 7f223c600640 1 -- start start 2024-04-25T01:17:29.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.014+0000 7f223c600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22340722e0 0x7f22341f3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:29.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.014+0000 7f223c600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22340cf4d0 0x7f2234143b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:29.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.014+0000 7f223c600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f223415fc40 0x7f2234146cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:29.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.014+0000 7f223c600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f223406d6d0 con 0x7f22340722e0 2024-04-25T01:17:29.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.014+0000 7f223c600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f223406cec0 con 0x7f223415fc40 2024-04-25T01:17:29.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.014+0000 7f223c600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2234070b50 con 0x7f22340cf4d0 2024-04-25T01:17:30.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.014+0000 7f2239c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22340722e0 0x7f22341f3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:30.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.014+0000 7f2239c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22340722e0 0x7f22341f3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33308/0 (socket says 172.21.15.90:33308) 2024-04-25T01:17:30.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.014+0000 7f2239c00640 1 -- 172.21.15.90:0/2020824942 learned_addr learned my addr 172.21.15.90:0/2020824942 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:30.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.015+0000 7f2239c00640 1 -- 172.21.15.90:0/2020824942 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22340cf4d0 msgr2=0x7f2234143b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:30.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.015+0000 7f2239c00640 1 --2- 172.21.15.90:0/2020824942 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22340cf4d0 0x7f2234143b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:30.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.015+0000 7f2239c00640 1 -- 172.21.15.90:0/2020824942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f223415fc40 msgr2=0x7f2234146cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:30.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.015+0000 7f2239c00640 1 --2- 172.21.15.90:0/2020824942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f223415fc40 0x7f2234146cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:30.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.015+0000 7f2239c00640 1 -- 172.21.15.90:0/2020824942 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22341471f0 con 0x7f22340722e0 2024-04-25T01:17:30.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.015+0000 7f2239c00640 1 --2- 172.21.15.90:0/2020824942 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22340722e0 0x7f22341f3c30 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2228035040 tx=0x7f22280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:30.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.015+0000 7f2231600640 1 -- 172.21.15.90:0/2020824942 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2228041070 con 0x7f22340722e0 2024-04-25T01:17:30.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.015+0000 7f2231600640 1 -- 172.21.15.90:0/2020824942 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f222803c670 con 0x7f22340722e0 2024-04-25T01:17:30.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.015+0000 7f223c600640 1 -- 172.21.15.90:0/2020824942 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22341f4240 con 0x7f22340722e0 2024-04-25T01:17:30.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.015+0000 7f2231600640 1 -- 172.21.15.90:0/2020824942 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2228037040 con 0x7f22340722e0 2024-04-25T01:17:30.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.016+0000 7f223c600640 1 -- 172.21.15.90:0/2020824942 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22341f4800 con 0x7f22340722e0 2024-04-25T01:17:30.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.017+0000 7f223c600640 1 -- 172.21.15.90:0/2020824942 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2234068fb0 con 0x7f22340722e0 2024-04-25T01:17:30.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.017+0000 7f2231600640 1 -- 172.21.15.90:0/2020824942 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f222803a070 con 0x7f22340722e0 2024-04-25T01:17:30.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.018+0000 7f2231600640 1 --2- 172.21.15.90:0/2020824942 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f221406be30 0x7f221406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:30.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.018+0000 7f2231600640 1 -- 172.21.15.90:0/2020824942 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f22280ab6c0 con 0x7f22340722e0 2024-04-25T01:17:30.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.019+0000 7f2239200640 1 --2- 172.21.15.90:0/2020824942 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f221406be30 0x7f221406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:30.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.019+0000 7f2239200640 1 --2- 172.21.15.90:0/2020824942 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f221406be30 0x7f221406e2f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2220007fd0 tx=0x7f2220007960 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:30.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.023+0000 7f2231600640 1 -- 172.21.15.90:0/2020824942 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f22341471f0 con 0x7f22340722e0 2024-04-25T01:17:30.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.220+0000 7f223c600640 1 -- 172.21.15.90:0/2020824942 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f22340746e0 con 0x7f22340722e0 2024-04-25T01:17:30.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.221+0000 7f2231600640 1 -- 172.21.15.90:0/2020824942 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) ==== 104+0+11 (secure 0 0 0) 0x7f2228079dc0 con 0x7f22340722e0 2024-04-25T01:17:30.206 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:17:30.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.224+0000 7f223c600640 1 -- 172.21.15.90:0/2020824942 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f221406be30 msgr2=0x7f221406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:30.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.224+0000 7f223c600640 1 --2- 172.21.15.90:0/2020824942 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f221406be30 0x7f221406e2f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2220007fd0 tx=0x7f2220007960 comp rx=0 tx=0).stop 2024-04-25T01:17:30.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.224+0000 7f223c600640 1 -- 172.21.15.90:0/2020824942 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22340722e0 msgr2=0x7f22341f3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:30.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.224+0000 7f223c600640 1 --2- 172.21.15.90:0/2020824942 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22340722e0 0x7f22341f3c30 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2228035040 tx=0x7f22280046c0 comp rx=0 tx=0).stop 2024-04-25T01:17:30.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.224+0000 7f223c600640 1 -- 172.21.15.90:0/2020824942 shutdown_connections 2024-04-25T01:17:30.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.224+0000 7f223c600640 1 --2- 172.21.15.90:0/2020824942 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f221406be30 0x7f221406e2f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:30.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.224+0000 7f223c600640 1 --2- 172.21.15.90:0/2020824942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f223415fc40 0x7f2234146cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:30.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.224+0000 7f223c600640 1 --2- 172.21.15.90:0/2020824942 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f22340cf4d0 0x7f2234143b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:30.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.224+0000 7f223c600640 1 --2- 172.21.15.90:0/2020824942 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22340722e0 0x7f22341f3c30 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:30.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.224+0000 7f223c600640 1 -- 172.21.15.90:0/2020824942 >> 172.21.15.90:0/2020824942 conn(0x7f223406b720 msgr2=0x7f22340ce2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:30.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.224+0000 7f223c600640 1 -- 172.21.15.90:0/2020824942 shutdown_connections 2024-04-25T01:17:30.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.224+0000 7f223c600640 1 -- 172.21.15.90:0/2020824942 wait complete. 2024-04-25T01:17:30.227 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:17:30.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.433+0000 7ff62aa00640 1 Processor -- start 2024-04-25T01:17:30.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.434+0000 7ff62aa00640 1 -- start start 2024-04-25T01:17:30.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.435+0000 7ff62aa00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff62415df00 0x7ff6240cf360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:30.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.435+0000 7ff62aa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6240d0ea0 0x7ff6240b1770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:30.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.435+0000 7ff62aa00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6240b09d0 con 0x7ff6240d4980 2024-04-25T01:17:30.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.435+0000 7ff62aa00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7ff6240b12e0 con 0x7ff62415df00 2024-04-25T01:17:30.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.435+0000 7ff62aa00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff6240cfbb0 con 0x7ff6240d0ea0 2024-04-25T01:17:30.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.435+0000 7ff629600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff62415df00 0x7ff6240cf360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:30.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.435+0000 7ff629600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff62415df00 0x7ff6240cf360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33320/0 (socket says 172.21.15.90:33320) 2024-04-25T01:17:30.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.435+0000 7ff629600640 1 -- 172.21.15.90:0/1156210644 learned_addr learned my addr 172.21.15.90:0/1156210644 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:30.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.436+0000 7ff62a000640 1 --2- 172.21.15.90:0/1156210644 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6240d0ea0 0x7ff6240b1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:30.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.436+0000 7ff629600640 1 -- 172.21.15.90:0/1156210644 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6240d0ea0 msgr2=0x7ff6240b1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:30.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.436+0000 7ff629600640 1 --2- 172.21.15.90:0/1156210644 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6240d0ea0 0x7ff6240b1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:30.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.436+0000 7ff629600640 1 -- 172.21.15.90:0/1156210644 >> v1:172.21.15.90:6789/0 conn(0x7ff6240d4980 legacy=0x7ff6240d0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:30.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.436+0000 7ff629600640 1 -- 172.21.15.90:0/1156210644 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6240b0c90 con 0x7ff62415df00 2024-04-25T01:17:30.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.437+0000 7ff629600640 1 --2- 172.21.15.90:0/1156210644 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff62415df00 0x7ff6240cf360 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff618009870 tx=0x7ff61802a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c3f8a056dff7730 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:30.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.437+0000 7ff623e00640 1 -- 172.21.15.90:0/1156210644 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff618037070 con 0x7ff62415df00 2024-04-25T01:17:30.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.437+0000 7ff623e00640 1 -- 172.21.15.90:0/1156210644 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff618032440 con 0x7ff62415df00 2024-04-25T01:17:30.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.437+0000 7ff623e00640 1 -- 172.21.15.90:0/1156210644 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff618032810 con 0x7ff62415df00 2024-04-25T01:17:30.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.438+0000 7ff62aa00640 1 -- 172.21.15.90:0/1156210644 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff62415df00 msgr2=0x7ff6240cf360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:30.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.438+0000 7ff62aa00640 1 --2- 172.21.15.90:0/1156210644 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff62415df00 0x7ff6240cf360 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff618009870 tx=0x7ff61802a250 comp rx=0 tx=0).stop 2024-04-25T01:17:30.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.438+0000 7ff62aa00640 1 -- 172.21.15.90:0/1156210644 shutdown_connections 2024-04-25T01:17:30.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.438+0000 7ff62aa00640 1 --2- 172.21.15.90:0/1156210644 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff6240d0ea0 0x7ff6240b1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:30.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.438+0000 7ff62aa00640 1 --2- 172.21.15.90:0/1156210644 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff62415df00 0x7ff6240cf360 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:30.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.438+0000 7ff62aa00640 1 -- 172.21.15.90:0/1156210644 >> 172.21.15.90:0/1156210644 conn(0x7ff6240d3d80 msgr2=0x7ff6240d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:30.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.438+0000 7ff62aa00640 1 -- 172.21.15.90:0/1156210644 shutdown_connections 2024-04-25T01:17:30.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.438+0000 7ff62aa00640 1 -- 172.21.15.90:0/1156210644 wait complete. 2024-04-25T01:17:30.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.440+0000 7ff62aa00640 1 Processor -- start 2024-04-25T01:17:30.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.440+0000 7ff62aa00640 1 -- start start 2024-04-25T01:17:30.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.441+0000 7ff62aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff6240d0ea0 0x7ff624160a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:30.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.441+0000 7ff62aa00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6240d4980 0x7ff6240cae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:30.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.441+0000 7ff62aa00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff62415df00 0x7ff6240cb390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:30.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.441+0000 7ff62aa00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff62415e2e0 con 0x7ff6240d0ea0 2024-04-25T01:17:30.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.441+0000 7ff62aa00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff6240cf8a0 con 0x7ff6240d4980 2024-04-25T01:17:30.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.441+0000 7ff62aa00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff62415e460 con 0x7ff62415df00 2024-04-25T01:17:30.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.441+0000 7ff629600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff6240d0ea0 0x7ff624160a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:30.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.441+0000 7ff629600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff6240d0ea0 0x7ff624160a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33334/0 (socket says 172.21.15.90:33334) 2024-04-25T01:17:30.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.441+0000 7ff629600640 1 -- 172.21.15.90:0/1158863114 learned_addr learned my addr 172.21.15.90:0/1158863114 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:30.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.441+0000 7ff629600640 1 -- 172.21.15.90:0/1158863114 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff62415df00 msgr2=0x7ff6240cb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:30.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.441+0000 7ff629600640 1 --2- 172.21.15.90:0/1158863114 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff62415df00 0x7ff6240cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:30.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.441+0000 7ff629600640 1 -- 172.21.15.90:0/1158863114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6240d4980 msgr2=0x7ff6240cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:30.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.441+0000 7ff629600640 1 --2- 172.21.15.90:0/1158863114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6240d4980 0x7ff6240cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:30.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.441+0000 7ff629600640 1 -- 172.21.15.90:0/1158863114 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6240cba10 con 0x7ff6240d0ea0 2024-04-25T01:17:30.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.442+0000 7ff629600640 1 --2- 172.21.15.90:0/1158863114 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff6240d0ea0 0x7ff624160a20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff618000c00 tx=0x7ff6180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:30.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.442+0000 7ff621600640 1 -- 172.21.15.90:0/1158863114 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff618041020 con 0x7ff6240d0ea0 2024-04-25T01:17:30.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.442+0000 7ff621600640 1 -- 172.21.15.90:0/1158863114 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6180306c0 con 0x7ff6240d0ea0 2024-04-25T01:17:30.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.442+0000 7ff621600640 1 -- 172.21.15.90:0/1158863114 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff618037040 con 0x7ff6240d0ea0 2024-04-25T01:17:30.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.442+0000 7ff62aa00640 1 -- 172.21.15.90:0/1158863114 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6240cbcf0 con 0x7ff6240d0ea0 2024-04-25T01:17:30.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.442+0000 7ff62aa00640 1 -- 172.21.15.90:0/1158863114 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6240cc2b0 con 0x7ff6240d0ea0 2024-04-25T01:17:30.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.443+0000 7ff62aa00640 1 -- 172.21.15.90:0/1158863114 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff624067dd0 con 0x7ff6240d0ea0 2024-04-25T01:17:30.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.444+0000 7ff621600640 1 -- 172.21.15.90:0/1158863114 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff618044020 con 0x7ff6240d0ea0 2024-04-25T01:17:30.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.445+0000 7ff621600640 1 --2- 172.21.15.90:0/1158863114 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff60806beb0 0x7ff60806e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:30.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.445+0000 7ff621600640 1 -- 172.21.15.90:0/1158863114 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7ff6180ab9c0 con 0x7ff6240d0ea0 2024-04-25T01:17:30.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.446+0000 7ff628c00640 1 --2- 172.21.15.90:0/1158863114 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff60806beb0 0x7ff60806e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:30.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.446+0000 7ff628c00640 1 --2- 172.21.15.90:0/1158863114 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff60806beb0 0x7ff60806e370 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff624153de0 tx=0x7ff614007cf0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:30.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.450+0000 7ff621600640 1 -- 172.21.15.90:0/1158863114 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff61804b030 con 0x7ff6240d0ea0 2024-04-25T01:17:30.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.647+0000 7ff62aa00640 1 -- 172.21.15.90:0/1158863114 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff62408d3d0 con 0x7ff6240d0ea0 2024-04-25T01:17:30.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.648+0000 7ff621600640 1 -- 172.21.15.90:0/1158863114 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) ==== 100+0+11 (secure 0 0 0) 0x7ff61807ee20 con 0x7ff6240d0ea0 2024-04-25T01:17:30.632 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-25T01:17:30.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.650+0000 7ff62aa00640 1 -- 172.21.15.90:0/1158863114 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff60806beb0 msgr2=0x7ff60806e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:30.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.650+0000 7ff62aa00640 1 --2- 172.21.15.90:0/1158863114 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff60806beb0 0x7ff60806e370 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff624153de0 tx=0x7ff614007cf0 comp rx=0 tx=0).stop 2024-04-25T01:17:30.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.650+0000 7ff62aa00640 1 -- 172.21.15.90:0/1158863114 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff6240d0ea0 msgr2=0x7ff624160a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:30.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.650+0000 7ff62aa00640 1 --2- 172.21.15.90:0/1158863114 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff6240d0ea0 0x7ff624160a20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff618000c00 tx=0x7ff6180046c0 comp rx=0 tx=0).stop 2024-04-25T01:17:30.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.651+0000 7ff62aa00640 1 -- 172.21.15.90:0/1158863114 shutdown_connections 2024-04-25T01:17:30.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.651+0000 7ff62aa00640 1 --2- 172.21.15.90:0/1158863114 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff60806beb0 0x7ff60806e370 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:30.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.651+0000 7ff62aa00640 1 --2- 172.21.15.90:0/1158863114 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff62415df00 0x7ff6240cb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:30.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.651+0000 7ff62aa00640 1 --2- 172.21.15.90:0/1158863114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6240d4980 0x7ff6240cae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:30.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.651+0000 7ff62aa00640 1 --2- 172.21.15.90:0/1158863114 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff6240d0ea0 0x7ff624160a20 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:30.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.651+0000 7ff62aa00640 1 -- 172.21.15.90:0/1158863114 >> 172.21.15.90:0/1158863114 conn(0x7ff6240d3d80 msgr2=0x7ff62415eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:30.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.651+0000 7ff62aa00640 1 -- 172.21.15.90:0/1158863114 shutdown_connections 2024-04-25T01:17:30.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.651+0000 7ff62aa00640 1 -- 172.21.15.90:0/1158863114 wait complete. 2024-04-25T01:17:30.657 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:17:30.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.865+0000 7f4f76000640 1 Processor -- start 2024-04-25T01:17:30.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.866+0000 7f4f76000640 1 -- start start 2024-04-25T01:17:30.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.867+0000 7f4f76000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f701596c0 0x7f4f7006de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:30.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.867+0000 7f4f76000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4f700cff00 0x7f4f7006e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:30.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.867+0000 7f4f76000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f7015c200 con 0x7f4f700b2880 2024-04-25T01:17:30.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.867+0000 7f4f76000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4f70159aa0 con 0x7f4f700cff00 2024-04-25T01:17:30.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.867+0000 7f4f76000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4f7015c470 con 0x7f4f701596c0 2024-04-25T01:17:30.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.867+0000 7f4f67e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4f700b2880 0x7f4f700cf7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47360/0 (socket says 172.21.15.90:47360) 2024-04-25T01:17:30.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.868+0000 7f4f67e00640 1 -- 172.21.15.90:0/205556964 learned_addr learned my addr 172.21.15.90:0/205556964 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:30.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.868+0000 7f4f75600640 1 --2- 172.21.15.90:0/205556964 >> v2:172.21.15.103:3300/0 conn(0x7f4f700cff00 0x7f4f7006e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:30.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.868+0000 7f4f74c00640 1 --2- 172.21.15.90:0/205556964 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f701596c0 0x7f4f7006de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:30.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.868+0000 7f4f6fe00640 1 -- 172.21.15.90:0/205556964 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 567095551 0 0) 0x7f4f7015c200 con 0x7f4f700b2880 2024-04-25T01:17:30.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.868+0000 7f4f6fe00640 1 -- 172.21.15.90:0/205556964 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f54003520 con 0x7f4f700b2880 2024-04-25T01:17:30.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.868+0000 7f4f6fe00640 1 -- 172.21.15.90:0/205556964 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1797389570 0 0) 0x7f4f54003520 con 0x7f4f700b2880 2024-04-25T01:17:30.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.868+0000 7f4f6fe00640 1 -- 172.21.15.90:0/205556964 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f701596c0 msgr2=0x7f4f7006de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:30.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.868+0000 7f4f6fe00640 1 --2- 172.21.15.90:0/205556964 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f701596c0 0x7f4f7006de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:30.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.868+0000 7f4f6fe00640 1 -- 172.21.15.90:0/205556964 >> v2:172.21.15.103:3300/0 conn(0x7f4f700cff00 msgr2=0x7f4f7006e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:30.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.868+0000 7f4f6fe00640 1 --2- 172.21.15.90:0/205556964 >> v2:172.21.15.103:3300/0 conn(0x7f4f700cff00 0x7f4f7006e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:30.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.868+0000 7f4f6fe00640 1 -- 172.21.15.90:0/205556964 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f7015c750 con 0x7f4f700b2880 2024-04-25T01:17:30.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.868+0000 7f4f6fe00640 1 -- 172.21.15.90:0/205556964 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f4f58001670 con 0x7f4f700b2880 2024-04-25T01:17:30.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.868+0000 7f4f6fe00640 1 -- 172.21.15.90:0/205556964 >> v1:172.21.15.90:6789/0 conn(0x7f4f700b2880 legacy=0x7f4f700cf7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:30.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.869+0000 7f4f6fe00640 1 --2- 172.21.15.90:0/205556964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f540039b0 0x7f4f54003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:30.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.869+0000 7f4f6fe00640 1 --2- 172.21.15.90:0/205556964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f54004800 0x7f4f54004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:30.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.869+0000 7f4f6fe00640 1 --2- 172.21.15.90:0/205556964 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f701596c0 0x7f4f540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:30.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.869+0000 7f4f6fe00640 1 -- 172.21.15.90:0/205556964 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4f70159aa0 con 0x7f4f540039b0 2024-04-25T01:17:30.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.869+0000 7f4f6fe00640 1 -- 172.21.15.90:0/205556964 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7015c470 con 0x7f4f54004800 2024-04-25T01:17:30.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.869+0000 7f4f6fe00640 1 -- 172.21.15.90:0/205556964 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4f5400b9c0 con 0x7f4f701596c0 2024-04-25T01:17:30.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.869+0000 7f4f74c00640 1 --2- 172.21.15.90:0/205556964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f540039b0 0x7f4f54003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:30.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.869+0000 7f4f74c00640 1 -- 172.21.15.90:0/205556964 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f701596c0 msgr2=0x7f4f540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:30.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.869+0000 7f4f74c00640 1 --2- 172.21.15.90:0/205556964 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f701596c0 0x7f4f540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:30.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.869+0000 7f4f67e00640 1 --2- 172.21.15.90:0/205556964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f54004800 0x7f4f54004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:30.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.869+0000 7f4f74c00640 1 -- 172.21.15.90:0/205556964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f54004800 msgr2=0x7f4f54004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:30.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.869+0000 7f4f74c00640 1 --2- 172.21.15.90:0/205556964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f54004800 0x7f4f54004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:30.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.869+0000 7f4f74c00640 1 -- 172.21.15.90:0/205556964 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f54003520 con 0x7f4f540039b0 2024-04-25T01:17:30.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.869+0000 7f4f67e00640 1 --2- 172.21.15.90:0/205556964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f54004800 0x7f4f54004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:30.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.870+0000 7f4f74c00640 1 --2- 172.21.15.90:0/205556964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f540039b0 0x7f4f54003dd0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4f60009ac0 tx=0x7f4f54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c95e4b3f75c9701 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:30.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.871+0000 7f4f6fe00640 1 -- 172.21.15.90:0/205556964 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f60036070 con 0x7f4f540039b0 2024-04-25T01:17:30.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.871+0000 7f4f6fe00640 1 -- 172.21.15.90:0/205556964 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f60031400 con 0x7f4f540039b0 2024-04-25T01:17:30.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.871+0000 7f4f6fe00640 1 -- 172.21.15.90:0/205556964 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f60009690 con 0x7f4f540039b0 2024-04-25T01:17:30.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.871+0000 7f4f76000640 1 -- 172.21.15.90:0/205556964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f540039b0 msgr2=0x7f4f54003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:30.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.871+0000 7f4f76000640 1 --2- 172.21.15.90:0/205556964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f540039b0 0x7f4f54003dd0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4f60009ac0 tx=0x7f4f54005830 comp rx=0 tx=0).stop 2024-04-25T01:17:30.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.871+0000 7f4f75600640 1 -- 172.21.15.90:0/205556964 reap_dead start 2024-04-25T01:17:30.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.871+0000 7f4f76000640 1 -- 172.21.15.90:0/205556964 shutdown_connections 2024-04-25T01:17:30.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.871+0000 7f4f76000640 1 -- 172.21.15.90:0/205556964 >> 172.21.15.90:0/205556964 conn(0x7f4f7006b120 msgr2=0x7f4f7006b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:30.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.871+0000 7f4f76000640 1 -- 172.21.15.90:0/205556964 shutdown_connections 2024-04-25T01:17:30.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.871+0000 7f4f76000640 1 -- 172.21.15.90:0/205556964 wait complete. 2024-04-25T01:17:30.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.873+0000 7f4f76000640 1 Processor -- start 2024-04-25T01:17:30.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.873+0000 7f4f76000640 1 -- start start 2024-04-25T01:17:30.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.874+0000 7f4f76000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f540039b0 0x7f4f7014c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:30.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.874+0000 7f4f76000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f70157d60 0x7f4f70155410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:30.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.874+0000 7f4f76000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f70155df0 0x7f4f70156280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:30.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.874+0000 7f4f76000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7006de90 con 0x7f4f540039b0 2024-04-25T01:17:30.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.874+0000 7f4f76000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7015cd10 con 0x7f4f70157d60 2024-04-25T01:17:30.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.874+0000 7f4f76000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4f7006e010 con 0x7f4f70155df0 2024-04-25T01:17:30.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.874+0000 7f4f74c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f540039b0 0x7f4f7014c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:30.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.874+0000 7f4f74c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f540039b0 0x7f4f7014c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33350/0 (socket says 172.21.15.90:33350) 2024-04-25T01:17:30.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.874+0000 7f4f74c00640 1 -- 172.21.15.90:0/3128023883 learned_addr learned my addr 172.21.15.90:0/3128023883 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:30.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.875+0000 7f4f67e00640 1 --2- 172.21.15.90:0/3128023883 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f70157d60 0x7f4f70155410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:30.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.875+0000 7f4f75600640 1 --2- 172.21.15.90:0/3128023883 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f70155df0 0x7f4f70156280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:30.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.875+0000 7f4f74c00640 1 -- 172.21.15.90:0/3128023883 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f70155df0 msgr2=0x7f4f70156280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:30.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.875+0000 7f4f74c00640 1 --2- 172.21.15.90:0/3128023883 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f70155df0 0x7f4f70156280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:30.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.875+0000 7f4f74c00640 1 -- 172.21.15.90:0/3128023883 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f70157d60 msgr2=0x7f4f70155410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:30.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.875+0000 7f4f74c00640 1 --2- 172.21.15.90:0/3128023883 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f70157d60 0x7f4f70155410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:30.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.875+0000 7f4f74c00640 1 -- 172.21.15.90:0/3128023883 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f70158200 con 0x7f4f540039b0 2024-04-25T01:17:30.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.875+0000 7f4f74c00640 1 --2- 172.21.15.90:0/3128023883 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f540039b0 0x7f4f7014c180 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4f60009be0 tx=0x7f4f60004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:30.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.876+0000 7f4f6d600640 1 -- 172.21.15.90:0/3128023883 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f60040020 con 0x7f4f540039b0 2024-04-25T01:17:30.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.876+0000 7f4f6d600640 1 -- 172.21.15.90:0/3128023883 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f600038d0 con 0x7f4f540039b0 2024-04-25T01:17:30.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.876+0000 7f4f6d600640 1 -- 172.21.15.90:0/3128023883 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f60036040 con 0x7f4f540039b0 2024-04-25T01:17:30.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.876+0000 7f4f76000640 1 -- 172.21.15.90:0/3128023883 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f701584e0 con 0x7f4f540039b0 2024-04-25T01:17:30.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.876+0000 7f4f76000640 1 -- 172.21.15.90:0/3128023883 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f70070e20 con 0x7f4f540039b0 2024-04-25T01:17:30.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.877+0000 7f4f76000640 1 -- 172.21.15.90:0/3128023883 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f70075350 con 0x7f4f540039b0 2024-04-25T01:17:30.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.878+0000 7f4f6d600640 1 -- 172.21.15.90:0/3128023883 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4f60043020 con 0x7f4f540039b0 2024-04-25T01:17:30.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.879+0000 7f4f6d600640 1 --2- 172.21.15.90:0/3128023883 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4f4c06be30 0x7f4f4c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:30.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.879+0000 7f4f6d600640 1 -- 172.21.15.90:0/3128023883 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f4f600ab360 con 0x7f4f540039b0 2024-04-25T01:17:30.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.879+0000 7f4f67e00640 1 --2- 172.21.15.90:0/3128023883 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4f4c06be30 0x7f4f4c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:30.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.880+0000 7f4f67e00640 1 --2- 172.21.15.90:0/3128023883 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4f4c06be30 0x7f4f4c06e2f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4f58000f60 tx=0x7f4f58009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:30.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:30.885+0000 7f4f6d600640 1 -- 172.21.15.90:0/3128023883 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4f6004a070 con 0x7f4f540039b0 2024-04-25T01:17:31.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.082+0000 7f4f76000640 1 -- 172.21.15.90:0/3128023883 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f4f700ad8e0 con 0x7f4f540039b0 2024-04-25T01:17:31.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.083+0000 7f4f6d600640 1 -- 172.21.15.90:0/3128023883 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) ==== 103+0+11 (secure 0 0 0) 0x7f4f60079990 con 0x7f4f540039b0 2024-04-25T01:17:31.067 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:17:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.085+0000 7f4f76000640 1 -- 172.21.15.90:0/3128023883 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4f4c06be30 msgr2=0x7f4f4c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.085+0000 7f4f76000640 1 --2- 172.21.15.90:0/3128023883 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4f4c06be30 0x7f4f4c06e2f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4f58000f60 tx=0x7f4f58009040 comp rx=0 tx=0).stop 2024-04-25T01:17:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.085+0000 7f4f76000640 1 -- 172.21.15.90:0/3128023883 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f540039b0 msgr2=0x7f4f7014c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.085+0000 7f4f76000640 1 --2- 172.21.15.90:0/3128023883 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f540039b0 0x7f4f7014c180 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4f60009be0 tx=0x7f4f60004730 comp rx=0 tx=0).stop 2024-04-25T01:17:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.085+0000 7f4f76000640 1 -- 172.21.15.90:0/3128023883 shutdown_connections 2024-04-25T01:17:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.085+0000 7f4f76000640 1 --2- 172.21.15.90:0/3128023883 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4f4c06be30 0x7f4f4c06e2f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.085+0000 7f4f76000640 1 --2- 172.21.15.90:0/3128023883 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4f70155df0 0x7f4f70156280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:31.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.085+0000 7f4f76000640 1 --2- 172.21.15.90:0/3128023883 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f70157d60 0x7f4f70155410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:31.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.085+0000 7f4f76000640 1 --2- 172.21.15.90:0/3128023883 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f540039b0 0x7f4f7014c180 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:31.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.085+0000 7f4f76000640 1 -- 172.21.15.90:0/3128023883 >> 172.21.15.90:0/3128023883 conn(0x7f4f7006b120 msgr2=0x7f4f7006b500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:31.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.085+0000 7f4f76000640 1 -- 172.21.15.90:0/3128023883 shutdown_connections 2024-04-25T01:17:31.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.085+0000 7f4f76000640 1 -- 172.21.15.90:0/3128023883 wait complete. 2024-04-25T01:17:31.090 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_recovery_pool.TestRecoveryPool.test_rebuild_simple' 2024-04-25T01:17:31.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.303+0000 7ffad5000640 1 Processor -- start 2024-04-25T01:17:31.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.303+0000 7ffad5000640 1 -- start start 2024-04-25T01:17:31.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.305+0000 7ffad5000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffad0143940 0x7ffad006e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:31.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.305+0000 7ffad5000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ffad015f400 0x7ffad006eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:31.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.305+0000 7ffad5000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffad0070e60 con 0x7ffad0143d20 2024-04-25T01:17:31.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.305+0000 7ffad5000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ffad0071140 con 0x7ffad015f400 2024-04-25T01:17:31.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.305+0000 7ffad5000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ffad0071420 con 0x7ffad0143940 2024-04-25T01:17:31.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.305+0000 7ffac5400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ffad0143d20 0x7ffad015ecf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47368/0 (socket says 172.21.15.90:47368) 2024-04-25T01:17:31.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.305+0000 7ffac5400640 1 -- 172.21.15.90:0/2884020619 learned_addr learned my addr 172.21.15.90:0/2884020619 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:31.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.306+0000 7ffacde00640 1 --2- 172.21.15.90:0/2884020619 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffad0143940 0x7ffad006e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:31.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.306+0000 7ffacd400640 1 -- 172.21.15.90:0/2884020619 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2129871142 0 0) 0x7ffad0070e60 con 0x7ffad0143d20 2024-04-25T01:17:31.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.306+0000 7ffacd400640 1 -- 172.21.15.90:0/2884020619 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffab4003520 con 0x7ffad0143d20 2024-04-25T01:17:31.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.306+0000 7fface800640 1 --2- 172.21.15.90:0/2884020619 >> v2:172.21.15.103:3300/0 conn(0x7ffad015f400 0x7ffad006eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:31.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.306+0000 7ffacd400640 1 -- 172.21.15.90:0/2884020619 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3592041596 0 0) 0x7ffab4003520 con 0x7ffad0143d20 2024-04-25T01:17:31.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.306+0000 7ffacd400640 1 -- 172.21.15.90:0/2884020619 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffad0143940 msgr2=0x7ffad006e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:31.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.306+0000 7ffacd400640 1 --2- 172.21.15.90:0/2884020619 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffad0143940 0x7ffad006e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:31.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.306+0000 7ffacd400640 1 -- 172.21.15.90:0/2884020619 >> v2:172.21.15.103:3300/0 conn(0x7ffad015f400 msgr2=0x7ffad006eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:31.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.306+0000 7ffacd400640 1 --2- 172.21.15.90:0/2884020619 >> v2:172.21.15.103:3300/0 conn(0x7ffad015f400 0x7ffad006eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:31.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.306+0000 7ffacd400640 1 -- 172.21.15.90:0/2884020619 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffad0071700 con 0x7ffad0143d20 2024-04-25T01:17:31.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.306+0000 7ffacd400640 1 -- 172.21.15.90:0/2884020619 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7ffab8001670 con 0x7ffad0143d20 2024-04-25T01:17:31.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.306+0000 7ffacd400640 1 -- 172.21.15.90:0/2884020619 >> v1:172.21.15.90:6789/0 conn(0x7ffad0143d20 legacy=0x7ffad015ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:31.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.306+0000 7ffacd400640 1 --2- 172.21.15.90:0/2884020619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffab40039b0 0x7ffab4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:31.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.306+0000 7ffacd400640 1 --2- 172.21.15.90:0/2884020619 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffad0143940 0x7ffab4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:31.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.306+0000 7ffacd400640 1 --2- 172.21.15.90:0/2884020619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffab4004d40 0x7ffab40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:31.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.306+0000 7ffacd400640 1 -- 172.21.15.90:0/2884020619 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ffad0071140 con 0x7ffab40039b0 2024-04-25T01:17:31.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.306+0000 7ffacd400640 1 -- 172.21.15.90:0/2884020619 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ffad0071420 con 0x7ffab4004d40 2024-04-25T01:17:31.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.306+0000 7ffacd400640 1 -- 172.21.15.90:0/2884020619 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ffab400bba0 con 0x7ffad0143940 2024-04-25T01:17:31.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.307+0000 7ffacde00640 1 --2- 172.21.15.90:0/2884020619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffab40039b0 0x7ffab4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:31.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.307+0000 7ffacde00640 1 -- 172.21.15.90:0/2884020619 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffad0143940 msgr2=0x7ffab4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:31.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.307+0000 7ffacde00640 1 --2- 172.21.15.90:0/2884020619 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffad0143940 0x7ffab4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:31.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.307+0000 7ffacde00640 1 -- 172.21.15.90:0/2884020619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffab4004d40 msgr2=0x7ffab40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:31.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.307+0000 7ffacde00640 1 --2- 172.21.15.90:0/2884020619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffab4004d40 0x7ffab40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:31.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.307+0000 7ffacde00640 1 -- 172.21.15.90:0/2884020619 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffab4003520 con 0x7ffab40039b0 2024-04-25T01:17:31.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.308+0000 7ffacde00640 1 --2- 172.21.15.90:0/2884020619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffab40039b0 0x7ffab4003dd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ffac000ba60 tx=0x7ffab4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c3a1a128423b183 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:31.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.308+0000 7ffacd400640 1 -- 172.21.15.90:0/2884020619 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac0030070 con 0x7ffab40039b0 2024-04-25T01:17:31.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.308+0000 7ffacd400640 1 -- 172.21.15.90:0/2884020619 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffac002ae40 con 0x7ffab40039b0 2024-04-25T01:17:31.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.308+0000 7ffacd400640 1 -- 172.21.15.90:0/2884020619 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac000b600 con 0x7ffab40039b0 2024-04-25T01:17:31.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.309+0000 7ffad5000640 1 -- 172.21.15.90:0/2884020619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffab40039b0 msgr2=0x7ffab4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:31.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.309+0000 7ffad5000640 1 --2- 172.21.15.90:0/2884020619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffab40039b0 0x7ffab4003dd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ffac000ba60 tx=0x7ffab4005830 comp rx=0 tx=0).stop 2024-04-25T01:17:31.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.309+0000 7fface800640 1 -- 172.21.15.90:0/2884020619 reap_dead start 2024-04-25T01:17:31.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.309+0000 7ffad5000640 1 -- 172.21.15.90:0/2884020619 shutdown_connections 2024-04-25T01:17:31.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.309+0000 7ffad5000640 1 -- 172.21.15.90:0/2884020619 >> 172.21.15.90:0/2884020619 conn(0x7ffad00b1820 msgr2=0x7ffad00b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:31.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.309+0000 7ffad5000640 1 -- 172.21.15.90:0/2884020619 shutdown_connections 2024-04-25T01:17:31.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.309+0000 7ffad5000640 1 -- 172.21.15.90:0/2884020619 wait complete. 2024-04-25T01:17:31.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.311+0000 7ffad5000640 1 Processor -- start 2024-04-25T01:17:31.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.311+0000 7ffad5000640 1 -- start start 2024-04-25T01:17:31.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.312+0000 7ffad5000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffab40039b0 0x7ffad00c8560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:31.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.312+0000 7ffad5000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffad015d9e0 0x7ffad00c5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:31.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.312+0000 7ffad5000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffad00cac70 0x7ffad00cb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:31.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.312+0000 7ffad5000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ffad006e9e0 con 0x7ffad015d9e0 2024-04-25T01:17:31.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.312+0000 7ffad5000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ffad006e860 con 0x7ffad00cac70 2024-04-25T01:17:31.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.312+0000 7ffad5000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ffad0071c20 con 0x7ffab40039b0 2024-04-25T01:17:31.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.312+0000 7ffac5400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffad015d9e0 0x7ffad00c5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:31.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.312+0000 7ffac5400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffad015d9e0 0x7ffad00c5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33376/0 (socket says 172.21.15.90:33376) 2024-04-25T01:17:31.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.312+0000 7ffac5400640 1 -- 172.21.15.90:0/30478983 learned_addr learned my addr 172.21.15.90:0/30478983 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:31.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.313+0000 7ffac5400640 1 -- 172.21.15.90:0/30478983 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffab40039b0 msgr2=0x7ffad00c8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:31.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.313+0000 7ffac5400640 1 --2- 172.21.15.90:0/30478983 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffab40039b0 0x7ffad00c8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:31.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.313+0000 7ffac5400640 1 -- 172.21.15.90:0/30478983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffad00cac70 msgr2=0x7ffad00cb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:31.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.313+0000 7ffac5400640 1 --2- 172.21.15.90:0/30478983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffad00cac70 0x7ffad00cb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:31.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.313+0000 7ffac5400640 1 -- 172.21.15.90:0/30478983 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffad00c6280 con 0x7ffad015d9e0 2024-04-25T01:17:31.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.313+0000 7ffac5400640 1 --2- 172.21.15.90:0/30478983 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffad015d9e0 0x7ffad00c5ae0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ffab800c770 tx=0x7ffab800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:31.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.313+0000 7ffac6a00640 1 -- 172.21.15.90:0/30478983 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab8018070 con 0x7ffad015d9e0 2024-04-25T01:17:31.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.313+0000 7ffac6a00640 1 -- 172.21.15.90:0/30478983 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffab80074e0 con 0x7ffad015d9e0 2024-04-25T01:17:31.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.313+0000 7ffac6a00640 1 -- 172.21.15.90:0/30478983 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab8004a20 con 0x7ffad015d9e0 2024-04-25T01:17:31.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.313+0000 7ffad5000640 1 -- 172.21.15.90:0/30478983 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffad00c6560 con 0x7ffad015d9e0 2024-04-25T01:17:31.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.313+0000 7ffad5000640 1 -- 172.21.15.90:0/30478983 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffad00c6b20 con 0x7ffad015d9e0 2024-04-25T01:17:31.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.315+0000 7ffad5000640 1 -- 172.21.15.90:0/30478983 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffad0068fb0 con 0x7ffad015d9e0 2024-04-25T01:17:31.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.316+0000 7ffac6a00640 1 -- 172.21.15.90:0/30478983 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ffab8004bc0 con 0x7ffad015d9e0 2024-04-25T01:17:31.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.317+0000 7ffac6a00640 1 --2- 172.21.15.90:0/30478983 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ffaac06be60 0x7ffaac06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:31.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.317+0000 7ffac6a00640 1 -- 172.21.15.90:0/30478983 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7ffab808a710 con 0x7ffad015d9e0 2024-04-25T01:17:31.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.317+0000 7ffacde00640 1 --2- 172.21.15.90:0/30478983 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ffaac06be60 0x7ffaac06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:31.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.318+0000 7ffacde00640 1 --2- 172.21.15.90:0/30478983 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ffaac06be60 0x7ffaac06e320 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ffac000ba30 tx=0x7ffac002c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:31.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.322+0000 7ffac6a00640 1 -- 172.21.15.90:0/30478983 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ffab800ce20 con 0x7ffad015d9e0 2024-04-25T01:17:31.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.592+0000 7ffad5000640 1 -- 172.21.15.90:0/30478983 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_recovery_pool.TestRecoveryPool.test_rebuild_simple"]} v 0) -- 0x7ffad006d150 con 0x7ffad015d9e0 2024-04-25T01:17:31.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.991+0000 7ffac6a00640 1 -- 172.21.15.90:0/30478983 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_recovery_pool.TestRecoveryPool.test_rebuild_simple"]}]=0 v84) ==== 150+0+0 (secure 0 0 0) 0x7ffab8058cb0 con 0x7ffad015d9e0 2024-04-25T01:17:31.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.995+0000 7ffad5000640 1 -- 172.21.15.90:0/30478983 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ffaac06be60 msgr2=0x7ffaac06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:31.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.995+0000 7ffad5000640 1 --2- 172.21.15.90:0/30478983 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ffaac06be60 0x7ffaac06e320 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ffac000ba30 tx=0x7ffac002c000 comp rx=0 tx=0).stop 2024-04-25T01:17:31.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.995+0000 7ffad5000640 1 -- 172.21.15.90:0/30478983 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffad015d9e0 msgr2=0x7ffad00c5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:31.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.995+0000 7ffad5000640 1 --2- 172.21.15.90:0/30478983 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffad015d9e0 0x7ffad00c5ae0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ffab800c770 tx=0x7ffab800cc30 comp rx=0 tx=0).stop 2024-04-25T01:17:31.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.995+0000 7ffad5000640 1 -- 172.21.15.90:0/30478983 shutdown_connections 2024-04-25T01:17:31.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.995+0000 7ffad5000640 1 --2- 172.21.15.90:0/30478983 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ffaac06be60 0x7ffaac06e320 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:31.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.995+0000 7ffad5000640 1 --2- 172.21.15.90:0/30478983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffad00cac70 0x7ffad00cb0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:31.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.995+0000 7ffad5000640 1 --2- 172.21.15.90:0/30478983 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffad015d9e0 0x7ffad00c5ae0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:31.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.995+0000 7ffad5000640 1 --2- 172.21.15.90:0/30478983 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffab40039b0 0x7ffad00c8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:31.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.995+0000 7ffad5000640 1 -- 172.21.15.90:0/30478983 >> 172.21.15.90:0/30478983 conn(0x7ffad00b1820 msgr2=0x7ffad00d03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:31.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.995+0000 7ffad5000640 1 -- 172.21.15.90:0/30478983 shutdown_connections 2024-04-25T01:17:31.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:31.995+0000 7ffad5000640 1 -- 172.21.15.90:0/30478983 wait complete. 2024-04-25T01:17:32.001 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-25T01:17:32.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.215+0000 7f5fad400640 1 Processor -- start 2024-04-25T01:17:32.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.215+0000 7f5fad400640 1 -- start start 2024-04-25T01:17:32.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.217+0000 7f5fad400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5fa8141fc0 0x7f5fa815fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:32.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.217+0000 7f5fad400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5fa815ff70 0x7f5fa806cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:32.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.217+0000 7f5fad400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fa80ce870 con 0x7f5fa813ebe0 2024-04-25T01:17:32.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.217+0000 7f5fad400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f5fa80cf910 con 0x7f5fa815ff70 2024-04-25T01:17:32.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.217+0000 7f5fad400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f5fa8142430 con 0x7f5fa8141fc0 2024-04-25T01:17:32.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.217+0000 7f5fa7200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5fa815ff70 0x7f5fa806cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:32.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.217+0000 7f5fa7200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5fa815ff70 0x7f5fa806cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33386/0 (socket says 172.21.15.90:33386) 2024-04-25T01:17:32.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.217+0000 7f5fa7200640 1 -- 172.21.15.90:0/1927314683 learned_addr learned my addr 172.21.15.90:0/1927314683 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:32.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.217+0000 7f5fa6800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f5fa813ebe0 0x7f5fa80ce090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47376/0 (socket says 172.21.15.90:47376) 2024-04-25T01:17:32.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.218+0000 7f5fa5e00640 1 --2- 172.21.15.90:0/1927314683 >> v2:172.21.15.103:3300/0 conn(0x7f5fa8141fc0 0x7f5fa815fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:32.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.218+0000 7f5fa7200640 1 -- 172.21.15.90:0/1927314683 >> v2:172.21.15.103:3300/0 conn(0x7f5fa8141fc0 msgr2=0x7f5fa815fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:32.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.218+0000 7f5fa7200640 1 --2- 172.21.15.90:0/1927314683 >> v2:172.21.15.103:3300/0 conn(0x7f5fa8141fc0 0x7f5fa815fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:32.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.218+0000 7f5fa7200640 1 -- 172.21.15.90:0/1927314683 >> v1:172.21.15.90:6789/0 conn(0x7f5fa813ebe0 legacy=0x7f5fa80ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:32.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.218+0000 7f5fa7200640 1 -- 172.21.15.90:0/1927314683 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fa80ceb50 con 0x7f5fa815ff70 2024-04-25T01:17:32.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.218+0000 7f5fa5400640 1 -- 172.21.15.90:0/1927314683 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 786572910 0 0) 0x7f5fa80ce870 con 0x7f5fa813ebe0 2024-04-25T01:17:32.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.219+0000 7f5fa7200640 1 --2- 172.21.15.90:0/1927314683 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5fa815ff70 0x7f5fa806cc10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8160a00 tx=0x7f5fa002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2286d918fc3d415 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:32.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.219+0000 7f5fa5400640 1 -- 172.21.15.90:0/1927314683 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0037070 con 0x7f5fa815ff70 2024-04-25T01:17:32.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.219+0000 7f5fa5400640 1 -- 172.21.15.90:0/1927314683 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fa0032440 con 0x7f5fa815ff70 2024-04-25T01:17:32.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.219+0000 7f5fa5400640 1 -- 172.21.15.90:0/1927314683 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0032810 con 0x7f5fa815ff70 2024-04-25T01:17:32.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.220+0000 7f5fad400640 1 -- 172.21.15.90:0/1927314683 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5fa815ff70 msgr2=0x7f5fa806cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:32.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.220+0000 7f5fad400640 1 --2- 172.21.15.90:0/1927314683 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5fa815ff70 0x7f5fa806cc10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8160a00 tx=0x7f5fa002a270 comp rx=0 tx=0).stop 2024-04-25T01:17:32.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.220+0000 7f5fad400640 1 -- 172.21.15.90:0/1927314683 shutdown_connections 2024-04-25T01:17:32.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.220+0000 7f5fad400640 1 --2- 172.21.15.90:0/1927314683 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5fa815ff70 0x7f5fa806cc10 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:32.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.220+0000 7f5fad400640 1 --2- 172.21.15.90:0/1927314683 >> v2:172.21.15.103:3300/0 conn(0x7f5fa8141fc0 0x7f5fa815fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:32.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.220+0000 7f5fad400640 1 -- 172.21.15.90:0/1927314683 >> 172.21.15.90:0/1927314683 conn(0x7f5fa80b1740 msgr2=0x7f5fa80b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:32.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.220+0000 7f5fad400640 1 -- 172.21.15.90:0/1927314683 shutdown_connections 2024-04-25T01:17:32.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.220+0000 7f5fad400640 1 -- 172.21.15.90:0/1927314683 wait complete. 2024-04-25T01:17:32.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.222+0000 7f5fad400640 1 Processor -- start 2024-04-25T01:17:32.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.222+0000 7f5fad400640 1 -- start start 2024-04-25T01:17:32.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.223+0000 7f5fad400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5fa813ebe0 0x7f5fa8152110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:32.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.223+0000 7f5fad400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5fa8141fc0 0x7f5fa813feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:32.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.223+0000 7f5fad400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5fa815ff70 0x7f5fa81403f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:32.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.223+0000 7f5fad400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa80cf480 con 0x7f5fa8141fc0 2024-04-25T01:17:32.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.223+0000 7f5fad400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa81603c0 con 0x7f5fa813ebe0 2024-04-25T01:17:32.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.223+0000 7f5fad400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5fa80cf110 con 0x7f5fa815ff70 2024-04-25T01:17:32.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.223+0000 7f5fa5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5fa8141fc0 0x7f5fa813feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:32.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.223+0000 7f5fa5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5fa8141fc0 0x7f5fa813feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33388/0 (socket says 172.21.15.90:33388) 2024-04-25T01:17:32.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.223+0000 7f5fa5e00640 1 -- 172.21.15.90:0/1091399910 learned_addr learned my addr 172.21.15.90:0/1091399910 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:32.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.224+0000 7f5fa6800640 1 --2- 172.21.15.90:0/1091399910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5fa813ebe0 0x7f5fa8152110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:32.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.224+0000 7f5fa7200640 1 --2- 172.21.15.90:0/1091399910 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5fa815ff70 0x7f5fa81403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:32.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.224+0000 7f5fa5e00640 1 -- 172.21.15.90:0/1091399910 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5fa815ff70 msgr2=0x7f5fa81403f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:32.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.224+0000 7f5fa5e00640 1 --2- 172.21.15.90:0/1091399910 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5fa815ff70 0x7f5fa81403f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:32.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.224+0000 7f5fa5e00640 1 -- 172.21.15.90:0/1091399910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5fa813ebe0 msgr2=0x7f5fa8152110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:32.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.224+0000 7f5fa5e00640 1 --2- 172.21.15.90:0/1091399910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5fa813ebe0 0x7f5fa8152110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:32.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.224+0000 7f5fa5e00640 1 -- 172.21.15.90:0/1091399910 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fa8152650 con 0x7f5fa8141fc0 2024-04-25T01:17:32.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.224+0000 7f5fa7200640 1 --2- 172.21.15.90:0/1091399910 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5fa815ff70 0x7f5fa81403f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:32.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.224+0000 7f5fa5e00640 1 --2- 172.21.15.90:0/1091399910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5fa8141fc0 0x7f5fa813feb0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5f98007ef0 tx=0x7f5f9800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:32.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.224+0000 7f5f9ea00640 1 -- 172.21.15.90:0/1091399910 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f98014620 con 0x7f5fa8141fc0 2024-04-25T01:17:32.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.224+0000 7f5f9ea00640 1 -- 172.21.15.90:0/1091399910 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f98019070 con 0x7f5fa8141fc0 2024-04-25T01:17:32.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.224+0000 7f5f9ea00640 1 -- 172.21.15.90:0/1091399910 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f98014a00 con 0x7f5fa8141fc0 2024-04-25T01:17:32.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.224+0000 7f5fad400640 1 -- 172.21.15.90:0/1091399910 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fa80721e0 con 0x7f5fa8141fc0 2024-04-25T01:17:32.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.224+0000 7f5fad400640 1 -- 172.21.15.90:0/1091399910 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fa80727a0 con 0x7f5fa8141fc0 2024-04-25T01:17:32.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.226+0000 7f5fad400640 1 -- 172.21.15.90:0/1091399910 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fa8068fb0 con 0x7f5fa8141fc0 2024-04-25T01:17:32.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.227+0000 7f5f9ea00640 1 -- 172.21.15.90:0/1091399910 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5f98014ba0 con 0x7f5fa8141fc0 2024-04-25T01:17:32.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.228+0000 7f5f9ea00640 1 --2- 172.21.15.90:0/1091399910 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f5f8406be30 0x7f5f8406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:32.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.228+0000 7f5f9ea00640 1 -- 172.21.15.90:0/1091399910 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f5f9808bb20 con 0x7f5fa8141fc0 2024-04-25T01:17:32.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.228+0000 7f5fa6800640 1 --2- 172.21.15.90:0/1091399910 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f5f8406be30 0x7f5f8406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:32.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.229+0000 7f5fa6800640 1 --2- 172.21.15.90:0/1091399910 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f5f8406be30 0x7f5f8406e2f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5f90000f60 tx=0x7f5f90007b50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:32.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.233+0000 7f5f9ea00640 1 -- 172.21.15.90:0/1091399910 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f5f9805a150 con 0x7f5fa8141fc0 2024-04-25T01:17:32.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.429+0000 7f5fad400640 1 -- 172.21.15.90:0/1091399910 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f5fa815d000 con 0x7f5fa8141fc0 2024-04-25T01:17:32.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.436+0000 7f5f9ea00640 1 -- 172.21.15.90:0/1091399910 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7f5f98059a50 con 0x7f5fa8141fc0 2024-04-25T01:17:32.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.438+0000 7f5fad400640 1 -- 172.21.15.90:0/1091399910 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f5f8406be30 msgr2=0x7f5f8406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:32.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.438+0000 7f5fad400640 1 --2- 172.21.15.90:0/1091399910 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f5f8406be30 0x7f5f8406e2f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5f90000f60 tx=0x7f5f90007b50 comp rx=0 tx=0).stop 2024-04-25T01:17:32.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.438+0000 7f5fad400640 1 -- 172.21.15.90:0/1091399910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5fa8141fc0 msgr2=0x7f5fa813feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:32.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.438+0000 7f5fad400640 1 --2- 172.21.15.90:0/1091399910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5fa8141fc0 0x7f5fa813feb0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5f98007ef0 tx=0x7f5f9800e4b0 comp rx=0 tx=0).stop 2024-04-25T01:17:32.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.439+0000 7f5fad400640 1 -- 172.21.15.90:0/1091399910 shutdown_connections 2024-04-25T01:17:32.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.439+0000 7f5fad400640 1 --2- 172.21.15.90:0/1091399910 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f5f8406be30 0x7f5f8406e2f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:32.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.439+0000 7f5fad400640 1 --2- 172.21.15.90:0/1091399910 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5fa815ff70 0x7f5fa81403f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:32.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.439+0000 7f5fad400640 1 --2- 172.21.15.90:0/1091399910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5fa8141fc0 0x7f5fa813feb0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:32.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.439+0000 7f5fad400640 1 --2- 172.21.15.90:0/1091399910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5fa813ebe0 0x7f5fa8152110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:32.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.439+0000 7f5fad400640 1 -- 172.21.15.90:0/1091399910 >> 172.21.15.90:0/1091399910 conn(0x7f5fa80b1740 msgr2=0x7f5fa815eae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:32.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.439+0000 7f5fad400640 1 -- 172.21.15.90:0/1091399910 shutdown_connections 2024-04-25T01:17:32.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:32.439+0000 7f5fad400640 1 -- 172.21.15.90:0/1091399910 wait complete. 2024-04-25T01:17:32.441 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-25T01:17:32.441 DEBUG:teuthology.orchestra.run.smithi090:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T01:17:32.539 DEBUG:teuthology.orchestra.run.smithi103:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T01:17:32.648 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-25T01:17:32.648 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:17:32.648 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:17:32.705 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:17:32.705 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:17:32.757 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:17:32.757 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:17:32.811 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:17:33.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.026+0000 7fe7c2800640 1 Processor -- start 2024-04-25T01:17:33.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.027+0000 7fe7c2800640 1 -- start start 2024-04-25T01:17:33.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.028+0000 7fe7c2800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7bc145590 0x7fe7bc0b4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:33.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.028+0000 7fe7c2800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe7bc15f9b0 0x7fe7bc15fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:33.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.028+0000 7fe7c2800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7bc0b4240 con 0x7fe7bc0ce110 2024-04-25T01:17:33.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.028+0000 7fe7c2800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe7bc160830 con 0x7fe7bc15f9b0 2024-04-25T01:17:33.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.028+0000 7fe7c2800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe7bc160350 con 0x7fe7bc145590 2024-04-25T01:17:33.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.029+0000 7fe7b2a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe7bc15f9b0 0x7fe7bc15fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:33.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.029+0000 7fe7bb400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7bc145590 0x7fe7bc0b4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:33.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.029+0000 7fe7bb400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7bc145590 0x7fe7bc0b4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.90:41594/0 (socket says 172.21.15.90:41594) 2024-04-25T01:17:33.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.029+0000 7fe7bb400640 1 -- 172.21.15.90:0/451426595 learned_addr learned my addr 172.21.15.90:0/451426595 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:33.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.030+0000 7fe7bb400640 1 -- 172.21.15.90:0/451426595 >> v2:172.21.15.103:3300/0 conn(0x7fe7bc15f9b0 msgr2=0x7fe7bc15fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:33.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.030+0000 7fe7bb400640 1 --2- 172.21.15.90:0/451426595 >> v2:172.21.15.103:3300/0 conn(0x7fe7bc15f9b0 0x7fe7bc15fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.030+0000 7fe7bb400640 1 -- 172.21.15.90:0/451426595 >> v1:172.21.15.103:6789/0 conn(0x7fe7bc0ce110 legacy=0x7fe7bc0ce5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:33.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.030+0000 7fe7bb400640 1 -- 172.21.15.90:0/451426595 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7bc0b4f60 con 0x7fe7bc145590 2024-04-25T01:17:33.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.031+0000 7fe7bb400640 1 --2- 172.21.15.90:0/451426595 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7bc145590 0x7fe7bc0b4a20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac009e50 tx=0x7fe7ac02a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67a1b42933f0c165 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:33.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.032+0000 7fe7baa00640 1 -- 172.21.15.90:0/451426595 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7ac036070 con 0x7fe7bc145590 2024-04-25T01:17:33.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.032+0000 7fe7baa00640 1 -- 172.21.15.90:0/451426595 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7ac031600 con 0x7fe7bc145590 2024-04-25T01:17:33.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.032+0000 7fe7baa00640 1 -- 172.21.15.90:0/451426595 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7ac0319d0 con 0x7fe7bc145590 2024-04-25T01:17:33.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.032+0000 7fe7c2800640 1 -- 172.21.15.90:0/451426595 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7bc145590 msgr2=0x7fe7bc0b4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:33.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.032+0000 7fe7c2800640 1 --2- 172.21.15.90:0/451426595 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7bc145590 0x7fe7bc0b4a20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac009e50 tx=0x7fe7ac02a710 comp rx=0 tx=0).stop 2024-04-25T01:17:33.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.033+0000 7fe7c2800640 1 -- 172.21.15.90:0/451426595 shutdown_connections 2024-04-25T01:17:33.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.033+0000 7fe7c2800640 1 --2- 172.21.15.90:0/451426595 >> v2:172.21.15.103:3300/0 conn(0x7fe7bc15f9b0 0x7fe7bc15fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.033+0000 7fe7c2800640 1 --2- 172.21.15.90:0/451426595 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7bc145590 0x7fe7bc0b4a20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.033+0000 7fe7c2800640 1 -- 172.21.15.90:0/451426595 >> 172.21.15.90:0/451426595 conn(0x7fe7bc0b1820 msgr2=0x7fe7bc0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:33.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.033+0000 7fe7c2800640 1 -- 172.21.15.90:0/451426595 shutdown_connections 2024-04-25T01:17:33.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.033+0000 7fe7c2800640 1 -- 172.21.15.90:0/451426595 wait complete. 2024-04-25T01:17:33.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.034+0000 7fe7c2800640 1 Processor -- start 2024-04-25T01:17:33.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.034+0000 7fe7c2800640 1 -- start start 2024-04-25T01:17:33.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.035+0000 7fe7c2800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7bc0ce110 0x7fe7bc12f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:33.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.035+0000 7fe7c2800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7bc145590 0x7fe7bc134bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:33.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.035+0000 7fe7c2800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7bc15f9b0 0x7fe7bc135100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:33.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.035+0000 7fe7c2800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe7bc0b3c80 con 0x7fe7bc0ce110 2024-04-25T01:17:33.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.035+0000 7fe7c2800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe7bc06cde0 con 0x7fe7bc145590 2024-04-25T01:17:33.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.035+0000 7fe7c2800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe7bc0b3f60 con 0x7fe7bc15f9b0 2024-04-25T01:17:33.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.036+0000 7fe7bb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7bc0ce110 0x7fe7bc12f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:33.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.036+0000 7fe7bb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7bc0ce110 0x7fe7bc12f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33400/0 (socket says 172.21.15.90:33400) 2024-04-25T01:17:33.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.036+0000 7fe7bb400640 1 -- 172.21.15.90:0/491236555 learned_addr learned my addr 172.21.15.90:0/491236555 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:33.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.036+0000 7fe7bb400640 1 -- 172.21.15.90:0/491236555 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7bc15f9b0 msgr2=0x7fe7bc135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:33.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.036+0000 7fe7bb400640 1 --2- 172.21.15.90:0/491236555 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7bc15f9b0 0x7fe7bc135100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.036+0000 7fe7bb400640 1 -- 172.21.15.90:0/491236555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7bc145590 msgr2=0x7fe7bc134bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:33.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.036+0000 7fe7bb400640 1 --2- 172.21.15.90:0/491236555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7bc145590 0x7fe7bc134bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.036+0000 7fe7bb400640 1 -- 172.21.15.90:0/491236555 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7bc12f3e0 con 0x7fe7bc0ce110 2024-04-25T01:17:33.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.036+0000 7fe7bb400640 1 --2- 172.21.15.90:0/491236555 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7bc0ce110 0x7fe7bc12f6b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac034040 tx=0x7fe7ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:33.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.037+0000 7fe7b3e00640 1 -- 172.21.15.90:0/491236555 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7ac036050 con 0x7fe7bc0ce110 2024-04-25T01:17:33.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.037+0000 7fe7b3e00640 1 -- 172.21.15.90:0/491236555 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7ac042070 con 0x7fe7bc0ce110 2024-04-25T01:17:33.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.037+0000 7fe7b3e00640 1 -- 172.21.15.90:0/491236555 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7ac009510 con 0x7fe7bc0ce110 2024-04-25T01:17:33.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.037+0000 7fe7c2800640 1 -- 172.21.15.90:0/491236555 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7bc158b70 con 0x7fe7bc0ce110 2024-04-25T01:17:33.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.037+0000 7fe7c2800640 1 -- 172.21.15.90:0/491236555 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7bc15b850 con 0x7fe7bc0ce110 2024-04-25T01:17:33.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.038+0000 7fe7c2800640 1 -- 172.21.15.90:0/491236555 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7bc068fb0 con 0x7fe7bc0ce110 2024-04-25T01:17:33.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.039+0000 7fe7b3e00640 1 -- 172.21.15.90:0/491236555 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe7ac007c70 con 0x7fe7bc0ce110 2024-04-25T01:17:33.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.040+0000 7fe7b3e00640 1 --2- 172.21.15.90:0/491236555 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe79806be60 0x7fe79806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:33.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.040+0000 7fe7b3e00640 1 -- 172.21.15.90:0/491236555 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7fe7ac0abfd0 con 0x7fe7bc0ce110 2024-04-25T01:17:33.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.040+0000 7fe7b2a00640 1 --2- 172.21.15.90:0/491236555 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe79806be60 0x7fe79806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:33.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.040+0000 7fe7b2a00640 1 --2- 172.21.15.90:0/491236555 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe79806be60 0x7fe79806e320 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc135c50 tx=0x7fe7a4009500 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:33.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.044+0000 7fe7b3e00640 1 -- 172.21.15.90:0/491236555 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe7ac045360 con 0x7fe7bc0ce110 2024-04-25T01:17:33.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.228+0000 7fe7c2800640 1 -- 172.21.15.90:0/491236555 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe7bc06ea50 con 0x7fe7bc0ce110 2024-04-25T01:17:33.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.231+0000 7fe7b3e00640 1 -- 172.21.15.90:0/491236555 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18576 (secure 0 0 0) 0x7fe7ac07a600 con 0x7fe7bc0ce110 2024-04-25T01:17:33.216 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:17:33.216 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:17:09.346008+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:16:20.094512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:16:21.078268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:16:24.670642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.e","mappings":[{"from":0,"to":7}]},{"pgid":"4.f","mappings":[{"from":7,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:17:33.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.233+0000 7fe7c2800640 1 -- 172.21.15.90:0/491236555 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe79806be60 msgr2=0x7fe79806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:33.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.233+0000 7fe7c2800640 1 --2- 172.21.15.90:0/491236555 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe79806be60 0x7fe79806e320 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc135c50 tx=0x7fe7a4009500 comp rx=0 tx=0).stop 2024-04-25T01:17:33.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.233+0000 7fe7c2800640 1 -- 172.21.15.90:0/491236555 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7bc0ce110 msgr2=0x7fe7bc12f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:33.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.233+0000 7fe7c2800640 1 --2- 172.21.15.90:0/491236555 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7bc0ce110 0x7fe7bc12f6b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac034040 tx=0x7fe7ac004a40 comp rx=0 tx=0).stop 2024-04-25T01:17:33.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.234+0000 7fe7c2800640 1 -- 172.21.15.90:0/491236555 shutdown_connections 2024-04-25T01:17:33.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.234+0000 7fe7c2800640 1 --2- 172.21.15.90:0/491236555 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe79806be60 0x7fe79806e320 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.234+0000 7fe7c2800640 1 --2- 172.21.15.90:0/491236555 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe7bc15f9b0 0x7fe7bc135100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.234+0000 7fe7c2800640 1 --2- 172.21.15.90:0/491236555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7bc145590 0x7fe7bc134bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.234+0000 7fe7c2800640 1 --2- 172.21.15.90:0/491236555 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7bc0ce110 0x7fe7bc12f6b0 secure :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac034040 tx=0x7fe7ac004a40 comp rx=0 tx=0).stop 2024-04-25T01:17:33.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.234+0000 7fe7c2800640 1 -- 172.21.15.90:0/491236555 >> 172.21.15.90:0/491236555 conn(0x7fe7bc0b1820 msgr2=0x7fe7bc0b0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:33.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.234+0000 7fe7c2800640 1 -- 172.21.15.90:0/491236555 shutdown_connections 2024-04-25T01:17:33.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.234+0000 7fe7c2800640 1 -- 172.21.15.90:0/491236555 wait complete. 2024-04-25T01:17:33.236 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:16:15.894126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:16:20.094512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:16:21.078268+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:16:24.670642+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:17:33.237 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:17:33.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.447+0000 7f4fe2a00640 1 Processor -- start 2024-04-25T01:17:33.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.447+0000 7f4fe2a00640 1 -- start start 2024-04-25T01:17:33.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.449+0000 7f4fe2a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4fdc0713e0 0x7f4fdc06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:33.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.449+0000 7f4fe2a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fdc0b4dc0 con 0x7f4fdc0b44b0 2024-04-25T01:17:33.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.449+0000 7f4fe2a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fdc071ac0 con 0x7f4fdc0cf510 2024-04-25T01:17:33.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.449+0000 7f4fe2a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4fdc0b2a90 con 0x7f4fdc0713e0 2024-04-25T01:17:33.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.449+0000 7f4fe0c00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4fdc0b44b0 0x7f4fdc06d190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47384/0 (socket says 172.21.15.90:47384) 2024-04-25T01:17:33.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.449+0000 7f4fe0c00640 1 -- 172.21.15.90:0/1409542120 learned_addr learned my addr 172.21.15.90:0/1409542120 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:33.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.450+0000 7f4fdbe00640 1 -- 172.21.15.90:0/1409542120 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1594797363 0 0) 0x7f4fc8001110 con 0x7f4fdc0b44b0 2024-04-25T01:17:33.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.450+0000 7f4fdbe00640 1 -- 172.21.15.90:0/1409542120 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4fc4003520 con 0x7f4fdc0b44b0 2024-04-25T01:17:33.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.450+0000 7f4fe2000640 1 --2- 172.21.15.90:0/1409542120 >> v2:172.21.15.103:3300/0 conn(0x7f4fdc0713e0 0x7f4fdc06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:33.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.450+0000 7f4fdbe00640 1 -- 172.21.15.90:0/1409542120 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3503420321 0 0) 0x7f4fc4003520 con 0x7f4fdc0b44b0 2024-04-25T01:17:33.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.450+0000 7f4fdbe00640 1 -- 172.21.15.90:0/1409542120 >> v2:172.21.15.103:3300/0 conn(0x7f4fdc0713e0 msgr2=0x7f4fdc06d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:33.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.450+0000 7f4fdbe00640 1 --2- 172.21.15.90:0/1409542120 >> v2:172.21.15.103:3300/0 conn(0x7f4fdc0713e0 0x7f4fdc06d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.450+0000 7f4fdbe00640 1 -- 172.21.15.90:0/1409542120 >> v1:172.21.15.103:6789/0 conn(0x7f4fdc0cf510 legacy=0x7f4fdc0cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:33.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.450+0000 7f4fdbe00640 1 -- 172.21.15.90:0/1409542120 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fdc0b3630 con 0x7f4fdc0b44b0 2024-04-25T01:17:33.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.450+0000 7f4fdbe00640 1 -- 172.21.15.90:0/1409542120 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f4fc8001840 con 0x7f4fdc0b44b0 2024-04-25T01:17:33.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.450+0000 7f4fdbe00640 1 -- 172.21.15.90:0/1409542120 >> v1:172.21.15.90:6789/0 conn(0x7f4fdc0b44b0 legacy=0x7f4fdc06d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:33.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.450+0000 7f4fdbe00640 1 --2- 172.21.15.90:0/1409542120 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fc4003a00 0x7f4fc4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:33.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.450+0000 7f4fdbe00640 1 --2- 172.21.15.90:0/1409542120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fc4004800 0x7f4fc4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:33.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.450+0000 7f4fdbe00640 1 --2- 172.21.15.90:0/1409542120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fc4005210 0x7f4fc400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:33.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.450+0000 7f4fdbe00640 1 -- 172.21.15.90:0/1409542120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc400c120 con 0x7f4fc4004800 2024-04-25T01:17:33.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.450+0000 7f4fdbe00640 1 -- 172.21.15.90:0/1409542120 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc400c2d0 con 0x7f4fc4005210 2024-04-25T01:17:33.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.450+0000 7f4fdbe00640 1 -- 172.21.15.90:0/1409542120 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4fc400c4d0 con 0x7f4fc4003a00 2024-04-25T01:17:33.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.451+0000 7f4fe0c00640 1 --2- 172.21.15.90:0/1409542120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fc4004800 0x7f4fc4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:33.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.451+0000 7f4fe0c00640 1 -- 172.21.15.90:0/1409542120 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fc4003a00 msgr2=0x7f4fc4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:33.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.451+0000 7f4fe1600640 1 --2- 172.21.15.90:0/1409542120 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fc4003a00 0x7f4fc4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:33.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.451+0000 7f4fe2000640 1 --2- 172.21.15.90:0/1409542120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fc4005210 0x7f4fc400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:33.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.451+0000 7f4fe0c00640 1 --2- 172.21.15.90:0/1409542120 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fc4003a00 0x7f4fc4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.451+0000 7f4fe0c00640 1 -- 172.21.15.90:0/1409542120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fc4005210 msgr2=0x7f4fc400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:33.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.451+0000 7f4fe0c00640 1 --2- 172.21.15.90:0/1409542120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fc4005210 0x7f4fc400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.451+0000 7f4fe0c00640 1 -- 172.21.15.90:0/1409542120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fc4003520 con 0x7f4fc4004800 2024-04-25T01:17:33.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.451+0000 7f4fe2000640 1 -- 172.21.15.90:0/1409542120 reap_dead start 2024-04-25T01:17:33.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.451+0000 7f4fe1600640 1 --2- 172.21.15.90:0/1409542120 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fc4003a00 0x7f4fc4003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:33.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.452+0000 7f4fe0c00640 1 --2- 172.21.15.90:0/1409542120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fc4004800 0x7f4fc4004c80 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8007fd0 tx=0x7f4fc4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f83e363e97c17a56 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:33.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.452+0000 7f4fdbe00640 1 -- 172.21.15.90:0/1409542120 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc8007e30 con 0x7f4fc4004800 2024-04-25T01:17:33.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.452+0000 7f4fdbe00640 1 -- 172.21.15.90:0/1409542120 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fc8005af0 con 0x7f4fc4004800 2024-04-25T01:17:33.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.452+0000 7f4fdbe00640 1 -- 172.21.15.90:0/1409542120 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc8007070 con 0x7f4fc4004800 2024-04-25T01:17:33.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.453+0000 7f4fe2a00640 1 -- 172.21.15.90:0/1409542120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fc4004800 msgr2=0x7f4fc4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:33.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.453+0000 7f4fe2a00640 1 --2- 172.21.15.90:0/1409542120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fc4004800 0x7f4fc4004c80 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8007fd0 tx=0x7f4fc4005830 comp rx=0 tx=0).stop 2024-04-25T01:17:33.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.453+0000 7f4fe2a00640 1 -- 172.21.15.90:0/1409542120 shutdown_connections 2024-04-25T01:17:33.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.453+0000 7f4fe2a00640 1 --2- 172.21.15.90:0/1409542120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fc4004800 0x7f4fc4004c80 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.453+0000 7f4fe2a00640 1 -- 172.21.15.90:0/1409542120 >> 172.21.15.90:0/1409542120 conn(0x7f4fdc06b120 msgr2=0x7f4fdc06b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:33.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.453+0000 7f4fe2a00640 1 -- 172.21.15.90:0/1409542120 shutdown_connections 2024-04-25T01:17:33.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.453+0000 7f4fe2a00640 1 -- 172.21.15.90:0/1409542120 wait complete. 2024-04-25T01:17:33.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.455+0000 7f4fe2a00640 1 Processor -- start 2024-04-25T01:17:33.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.455+0000 7f4fe2a00640 1 -- start start 2024-04-25T01:17:33.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.456+0000 7f4fe2a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fc4004800 0x7f4fdc142ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:33.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.456+0000 7f4fe2a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fdc1425c0 0x7f4fdc144650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:33.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.456+0000 7f4fe2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fdc158b60 0x7f4fdc158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:33.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.456+0000 7f4fe2a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4fdc0b3bf0 con 0x7f4fdc158b60 2024-04-25T01:17:33.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.456+0000 7f4fe2a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4fdc071870 con 0x7f4fc4004800 2024-04-25T01:17:33.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.456+0000 7f4fe2a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4fdc0b3ed0 con 0x7f4fdc1425c0 2024-04-25T01:17:33.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.456+0000 7f4fe2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fdc158b60 0x7f4fdc158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:33.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.456+0000 7f4fe2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fdc158b60 0x7f4fdc158fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33430/0 (socket says 172.21.15.90:33430) 2024-04-25T01:17:33.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.456+0000 7f4fe2000640 1 -- 172.21.15.90:0/3046015826 learned_addr learned my addr 172.21.15.90:0/3046015826 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:33.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.457+0000 7f4fe1600640 1 --2- 172.21.15.90:0/3046015826 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fc4004800 0x7f4fdc142ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:33.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.457+0000 7f4fe0c00640 1 --2- 172.21.15.90:0/3046015826 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fdc1425c0 0x7f4fdc144650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:33.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.457+0000 7f4fe2000640 1 -- 172.21.15.90:0/3046015826 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fdc1425c0 msgr2=0x7f4fdc144650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:33.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.457+0000 7f4fe2000640 1 --2- 172.21.15.90:0/3046015826 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fdc1425c0 0x7f4fdc144650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.457+0000 7f4fe2000640 1 -- 172.21.15.90:0/3046015826 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fc4004800 msgr2=0x7f4fdc142ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:33.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.457+0000 7f4fe2000640 1 --2- 172.21.15.90:0/3046015826 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fc4004800 0x7f4fdc142ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.457+0000 7f4fe2000640 1 -- 172.21.15.90:0/3046015826 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fdc1435b0 con 0x7f4fdc158b60 2024-04-25T01:17:33.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.457+0000 7f4fe2000640 1 --2- 172.21.15.90:0/3046015826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fdc158b60 0x7f4fdc158fe0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc161900 tx=0x7f4fd4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:33.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.457+0000 7f4fd9600640 1 -- 172.21.15.90:0/3046015826 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd4019070 con 0x7f4fdc158b60 2024-04-25T01:17:33.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.457+0000 7f4fd9600640 1 -- 172.21.15.90:0/3046015826 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fd4007d50 con 0x7f4fdc158b60 2024-04-25T01:17:33.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.457+0000 7f4fd9600640 1 -- 172.21.15.90:0/3046015826 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd4014640 con 0x7f4fdc158b60 2024-04-25T01:17:33.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.457+0000 7f4fe2a00640 1 -- 172.21.15.90:0/3046015826 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fdc143890 con 0x7f4fdc158b60 2024-04-25T01:17:33.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.458+0000 7f4fe2a00640 1 -- 172.21.15.90:0/3046015826 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fdc143e50 con 0x7f4fdc158b60 2024-04-25T01:17:33.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.459+0000 7f4fe2a00640 1 -- 172.21.15.90:0/3046015826 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fdc075350 con 0x7f4fdc158b60 2024-04-25T01:17:33.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.459+0000 7f4fd9600640 1 -- 172.21.15.90:0/3046015826 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4fd401c070 con 0x7f4fdc158b60 2024-04-25T01:17:33.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.460+0000 7f4fd9600640 1 --2- 172.21.15.90:0/3046015826 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4fbc06be30 0x7f4fbc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:33.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.460+0000 7f4fd9600640 1 -- 172.21.15.90:0/3046015826 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f4fd408aff0 con 0x7f4fdc158b60 2024-04-25T01:17:33.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.461+0000 7f4fe1600640 1 --2- 172.21.15.90:0/3046015826 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4fbc06be30 0x7f4fbc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:33.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.461+0000 7f4fe1600640 1 --2- 172.21.15.90:0/3046015826 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4fbc06be30 0x7f4fbc06e2f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4fd0000c00 tx=0x7f4fd0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:33.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.465+0000 7f4fd9600640 1 -- 172.21.15.90:0/3046015826 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4fd4059590 con 0x7f4fdc158b60 2024-04-25T01:17:33.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.649+0000 7f4fe2a00640 1 -- 172.21.15.90:0/3046015826 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4fdc073cd0 con 0x7f4fdc158b60 2024-04-25T01:17:33.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.650+0000 7f4fd9600640 1 -- 172.21.15.90:0/3046015826 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) ==== 93+0+10 (secure 0 0 0) 0x7f4fd4058e90 con 0x7f4fdc158b60 2024-04-25T01:17:33.635 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T01:17:33.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.653+0000 7f4fe2a00640 1 -- 172.21.15.90:0/3046015826 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4fbc06be30 msgr2=0x7f4fbc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:33.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.653+0000 7f4fe2a00640 1 --2- 172.21.15.90:0/3046015826 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4fbc06be30 0x7f4fbc06e2f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4fd0000c00 tx=0x7f4fd0009040 comp rx=0 tx=0).stop 2024-04-25T01:17:33.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.653+0000 7f4fe2a00640 1 -- 172.21.15.90:0/3046015826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fdc158b60 msgr2=0x7f4fdc158fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:33.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.653+0000 7f4fe2a00640 1 --2- 172.21.15.90:0/3046015826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fdc158b60 0x7f4fdc158fe0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc161900 tx=0x7f4fd4009da0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.653+0000 7f4fe2a00640 1 -- 172.21.15.90:0/3046015826 shutdown_connections 2024-04-25T01:17:33.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.653+0000 7f4fe2a00640 1 --2- 172.21.15.90:0/3046015826 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4fbc06be30 0x7f4fbc06e2f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.653+0000 7f4fe2a00640 1 --2- 172.21.15.90:0/3046015826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fdc158b60 0x7f4fdc158fe0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.653+0000 7f4fe2a00640 1 --2- 172.21.15.90:0/3046015826 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fdc1425c0 0x7f4fdc144650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.653+0000 7f4fe2a00640 1 --2- 172.21.15.90:0/3046015826 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fc4004800 0x7f4fdc142ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.653+0000 7f4fe2a00640 1 -- 172.21.15.90:0/3046015826 >> 172.21.15.90:0/3046015826 conn(0x7f4fdc06b120 msgr2=0x7f4fdc06d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:33.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.653+0000 7f4fe2a00640 1 -- 172.21.15.90:0/3046015826 shutdown_connections 2024-04-25T01:17:33.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.653+0000 7f4fe2a00640 1 -- 172.21.15.90:0/3046015826 wait complete. 2024-04-25T01:17:33.655 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:17:33.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.869+0000 7f01d0a00640 1 Processor -- start 2024-04-25T01:17:33.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.869+0000 7f01d0a00640 1 -- start start 2024-04-25T01:17:33.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.870+0000 7f01d0a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f01cc0722e0 0x7f01cc06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.870+0000 7f01d0a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01cc15fc40 0x7f01cc1600c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.870+0000 7f01d0a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01cc06e410 con 0x7f01cc0cedc0 2024-04-25T01:17:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.870+0000 7f01d0a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f01cc160aa0 con 0x7f01cc0722e0 2024-04-25T01:17:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.870+0000 7f01d0a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f01cc06da40 con 0x7f01cc15fc40 2024-04-25T01:17:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.870+0000 7f01c9e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f01cc0722e0 0x7f01cc06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.870+0000 7f01c9e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f01cc0722e0 0x7f01cc06ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33440/0 (socket says 172.21.15.90:33440) 2024-04-25T01:17:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.870+0000 7f01c9e00640 1 -- 172.21.15.90:0/3353180903 learned_addr learned my addr 172.21.15.90:0/3353180903 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.871+0000 7f01c9e00640 1 -- 172.21.15.90:0/3353180903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01cc15fc40 msgr2=0x7f01cc1600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.871+0000 7f01c9400640 1 --2- 172.21.15.90:0/3353180903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01cc15fc40 0x7f01cc1600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.871+0000 7f01c9e00640 1 --2- 172.21.15.90:0/3353180903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01cc15fc40 0x7f01cc1600c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.871+0000 7f01c9e00640 1 -- 172.21.15.90:0/3353180903 >> v1:172.21.15.90:6789/0 conn(0x7f01cc0cedc0 legacy=0x7f01cc0cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.871+0000 7f01c8a00640 1 -- 172.21.15.90:0/3353180903 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2141081755 0 0) 0x7f01cc06e410 con 0x7f01cc0cedc0 2024-04-25T01:17:33.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.871+0000 7f01c9e00640 1 -- 172.21.15.90:0/3353180903 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01cc06d110 con 0x7f01cc0722e0 2024-04-25T01:17:33.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.871+0000 7f01c9400640 1 --2- 172.21.15.90:0/3353180903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01cc15fc40 0x7f01cc1600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:33.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.872+0000 7f01c9e00640 1 --2- 172.21.15.90:0/3353180903 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01cc0722e0 0x7f01cc06ded0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f01bc009870 tx=0x7f01bc02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3db424b504dd049 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:33.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.872+0000 7f01c8a00640 1 -- 172.21.15.90:0/3353180903 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01bc037070 con 0x7f01cc0722e0 2024-04-25T01:17:33.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.872+0000 7f01c8a00640 1 -- 172.21.15.90:0/3353180903 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01bc032440 con 0x7f01cc0722e0 2024-04-25T01:17:33.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.872+0000 7f01c8a00640 1 -- 172.21.15.90:0/3353180903 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01bc032810 con 0x7f01cc0722e0 2024-04-25T01:17:33.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.873+0000 7f01d0a00640 1 -- 172.21.15.90:0/3353180903 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01cc0722e0 msgr2=0x7f01cc06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:33.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.873+0000 7f01d0a00640 1 --2- 172.21.15.90:0/3353180903 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01cc0722e0 0x7f01cc06ded0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f01bc009870 tx=0x7f01bc02a1c0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.873+0000 7f01d0a00640 1 -- 172.21.15.90:0/3353180903 shutdown_connections 2024-04-25T01:17:33.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.873+0000 7f01d0a00640 1 --2- 172.21.15.90:0/3353180903 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01cc15fc40 0x7f01cc1600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.873+0000 7f01d0a00640 1 --2- 172.21.15.90:0/3353180903 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01cc0722e0 0x7f01cc06ded0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.873+0000 7f01d0a00640 1 -- 172.21.15.90:0/3353180903 >> 172.21.15.90:0/3353180903 conn(0x7f01cc06b720 msgr2=0x7f01cc0b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:33.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.873+0000 7f01d0a00640 1 -- 172.21.15.90:0/3353180903 shutdown_connections 2024-04-25T01:17:33.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.873+0000 7f01d0a00640 1 -- 172.21.15.90:0/3353180903 wait complete. 2024-04-25T01:17:33.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.874+0000 7f01d0a00640 1 Processor -- start 2024-04-25T01:17:33.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.875+0000 7f01d0a00640 1 -- start start 2024-04-25T01:17:33.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.876+0000 7f01d0a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01cc0722e0 0x7f01cc1e6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:33.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.876+0000 7f01d0a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01cc0cedc0 0x7f01cc1e5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:33.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.876+0000 7f01d0a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01cc15fc40 0x7f01cc155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:33.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.876+0000 7f01d0a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f01cc06d6d0 con 0x7f01cc15fc40 2024-04-25T01:17:33.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.876+0000 7f01d0a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f01cc06cec0 con 0x7f01cc0cedc0 2024-04-25T01:17:33.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.876+0000 7f01d0a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f01cc070b50 con 0x7f01cc0722e0 2024-04-25T01:17:33.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.876+0000 7f01ca800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01cc15fc40 0x7f01cc155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:33.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.876+0000 7f01ca800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01cc15fc40 0x7f01cc155fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33444/0 (socket says 172.21.15.90:33444) 2024-04-25T01:17:33.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.876+0000 7f01ca800640 1 -- 172.21.15.90:0/548612550 learned_addr learned my addr 172.21.15.90:0/548612550 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:33.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.876+0000 7f01c9e00640 1 --2- 172.21.15.90:0/548612550 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01cc0722e0 0x7f01cc1e6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:33.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.876+0000 7f01c9400640 1 --2- 172.21.15.90:0/548612550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01cc0cedc0 0x7f01cc1e5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:33.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.876+0000 7f01ca800640 1 -- 172.21.15.90:0/548612550 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01cc0722e0 msgr2=0x7f01cc1e6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:33.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.876+0000 7f01ca800640 1 --2- 172.21.15.90:0/548612550 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01cc0722e0 0x7f01cc1e6fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.876+0000 7f01ca800640 1 -- 172.21.15.90:0/548612550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01cc0cedc0 msgr2=0x7f01cc1e5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:33.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.876+0000 7f01ca800640 1 --2- 172.21.15.90:0/548612550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01cc0cedc0 0x7f01cc1e5930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:33.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.876+0000 7f01ca800640 1 -- 172.21.15.90:0/548612550 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01cc0ad8e0 con 0x7f01cc15fc40 2024-04-25T01:17:33.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.877+0000 7f01ca800640 1 --2- 172.21.15.90:0/548612550 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01cc15fc40 0x7f01cc155fa0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f01c400c790 tx=0x7f01c400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:33.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.877+0000 7f01c2000640 1 -- 172.21.15.90:0/548612550 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c4003da0 con 0x7f01cc15fc40 2024-04-25T01:17:33.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.877+0000 7f01c2000640 1 -- 172.21.15.90:0/548612550 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01c4005070 con 0x7f01cc15fc40 2024-04-25T01:17:33.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.877+0000 7f01d0a00640 1 -- 172.21.15.90:0/548612550 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01cc1e6710 con 0x7f01cc15fc40 2024-04-25T01:17:33.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.877+0000 7f01c2000640 1 -- 172.21.15.90:0/548612550 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c4005460 con 0x7f01cc15fc40 2024-04-25T01:17:33.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.877+0000 7f01d0a00640 1 -- 172.21.15.90:0/548612550 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01cc0adb80 con 0x7f01cc15fc40 2024-04-25T01:17:33.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.878+0000 7f01d0a00640 1 -- 172.21.15.90:0/548612550 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01cc068fb0 con 0x7f01cc15fc40 2024-04-25T01:17:33.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.879+0000 7f01c2000640 1 -- 172.21.15.90:0/548612550 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f01c4005600 con 0x7f01cc15fc40 2024-04-25T01:17:33.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.880+0000 7f01c2000640 1 --2- 172.21.15.90:0/548612550 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f01a806be30 0x7f01a806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:33.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.880+0000 7f01c2000640 1 -- 172.21.15.90:0/548612550 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f01c408b190 con 0x7f01cc15fc40 2024-04-25T01:17:33.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.881+0000 7f01c9e00640 1 --2- 172.21.15.90:0/548612550 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f01a806be30 0x7f01a806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:33.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.881+0000 7f01c9e00640 1 --2- 172.21.15.90:0/548612550 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f01a806be30 0x7f01a806e2f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f01bc034000 tx=0x7f01bc02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:33.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:33.885+0000 7f01c2000640 1 -- 172.21.15.90:0/548612550 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f01c4059730 con 0x7f01cc15fc40 2024-04-25T01:17:34.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.081+0000 7f01d0a00640 1 -- 172.21.15.90:0/548612550 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f01cc0add50 con 0x7f01cc15fc40 2024-04-25T01:17:34.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.082+0000 7f01c2000640 1 -- 172.21.15.90:0/548612550 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) ==== 104+0+11 (secure 0 0 0) 0x7f01c4059030 con 0x7f01cc15fc40 2024-04-25T01:17:34.066 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:17:34.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.085+0000 7f01d0a00640 1 -- 172.21.15.90:0/548612550 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f01a806be30 msgr2=0x7f01a806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:34.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.085+0000 7f01d0a00640 1 --2- 172.21.15.90:0/548612550 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f01a806be30 0x7f01a806e2f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f01bc034000 tx=0x7f01bc02e000 comp rx=0 tx=0).stop 2024-04-25T01:17:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.085+0000 7f01d0a00640 1 -- 172.21.15.90:0/548612550 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01cc15fc40 msgr2=0x7f01cc155fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.085+0000 7f01d0a00640 1 --2- 172.21.15.90:0/548612550 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01cc15fc40 0x7f01cc155fa0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f01c400c790 tx=0x7f01c400cc50 comp rx=0 tx=0).stop 2024-04-25T01:17:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.085+0000 7f01d0a00640 1 -- 172.21.15.90:0/548612550 shutdown_connections 2024-04-25T01:17:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.085+0000 7f01d0a00640 1 --2- 172.21.15.90:0/548612550 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f01a806be30 0x7f01a806e2f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.085+0000 7f01d0a00640 1 --2- 172.21.15.90:0/548612550 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01cc15fc40 0x7f01cc155fa0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.085+0000 7f01d0a00640 1 --2- 172.21.15.90:0/548612550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01cc0cedc0 0x7f01cc1e5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.085+0000 7f01d0a00640 1 --2- 172.21.15.90:0/548612550 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01cc0722e0 0x7f01cc1e6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.085+0000 7f01d0a00640 1 -- 172.21.15.90:0/548612550 >> 172.21.15.90:0/548612550 conn(0x7f01cc06b720 msgr2=0x7f01cc074600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.085+0000 7f01d0a00640 1 -- 172.21.15.90:0/548612550 shutdown_connections 2024-04-25T01:17:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.085+0000 7f01d0a00640 1 -- 172.21.15.90:0/548612550 wait complete. 2024-04-25T01:17:34.089 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:17:34.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.304+0000 7f00cd800640 1 Processor -- start 2024-04-25T01:17:34.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.305+0000 7f00cd800640 1 -- start start 2024-04-25T01:17:34.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.306+0000 7f00cd800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f00c8158a70 0x7f00c80b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:34.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.306+0000 7f00cd800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f00c80b37c0 0x7f00c80b4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:34.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.306+0000 7f00cd800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00c815c210 con 0x7f00c815b520 2024-04-25T01:17:34.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.306+0000 7f00cd800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f00c80b2ce0 con 0x7f00c80b37c0 2024-04-25T01:17:34.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.306+0000 7f00cd800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f00c80b2fc0 con 0x7f00c8158a70 2024-04-25T01:17:34.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.307+0000 7f00be000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f00c80b37c0 0x7f00c80b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:34.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.307+0000 7f00be000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f00c80b37c0 0x7f00c80b4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33448/0 (socket says 172.21.15.90:33448) 2024-04-25T01:17:34.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.307+0000 7f00be000640 1 -- 172.21.15.90:0/2606301743 learned_addr learned my addr 172.21.15.90:0/2606301743 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:34.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.307+0000 7f00c6a00640 1 --2- 172.21.15.90:0/2606301743 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f00c8158a70 0x7f00c80b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:34.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.307+0000 7f00be000640 1 -- 172.21.15.90:0/2606301743 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f00c8158a70 msgr2=0x7f00c80b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:34.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.307+0000 7f00be000640 1 --2- 172.21.15.90:0/2606301743 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f00c8158a70 0x7f00c80b40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:34.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.307+0000 7f00be000640 1 -- 172.21.15.90:0/2606301743 >> v1:172.21.15.103:6789/0 conn(0x7f00c815b520 legacy=0x7f00c815b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:34.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.307+0000 7f00be000640 1 -- 172.21.15.90:0/2606301743 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c80b32a0 con 0x7f00c80b37c0 2024-04-25T01:17:34.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.308+0000 7f00be000640 1 --2- 172.21.15.90:0/2606301743 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00c80b37c0 0x7f00c80b4600 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f00b0009870 tx=0x7f00b002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=516ea258270f4f90 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:34.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.309+0000 7f00c6000640 1 -- 172.21.15.90:0/2606301743 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b0037070 con 0x7f00c80b37c0 2024-04-25T01:17:34.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.309+0000 7f00c6000640 1 -- 172.21.15.90:0/2606301743 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00b0032440 con 0x7f00c80b37c0 2024-04-25T01:17:34.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.309+0000 7f00c6000640 1 -- 172.21.15.90:0/2606301743 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b0032810 con 0x7f00c80b37c0 2024-04-25T01:17:34.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.309+0000 7f00cd800640 1 -- 172.21.15.90:0/2606301743 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00c80b37c0 msgr2=0x7f00c80b4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:34.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.309+0000 7f00cd800640 1 --2- 172.21.15.90:0/2606301743 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00c80b37c0 0x7f00c80b4600 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f00b0009870 tx=0x7f00b002a270 comp rx=0 tx=0).stop 2024-04-25T01:17:34.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.309+0000 7f00cd800640 1 -- 172.21.15.90:0/2606301743 shutdown_connections 2024-04-25T01:17:34.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.309+0000 7f00cd800640 1 --2- 172.21.15.90:0/2606301743 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00c80b37c0 0x7f00c80b4600 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:34.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.309+0000 7f00cd800640 1 --2- 172.21.15.90:0/2606301743 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f00c8158a70 0x7f00c80b40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:34.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.309+0000 7f00cd800640 1 -- 172.21.15.90:0/2606301743 >> 172.21.15.90:0/2606301743 conn(0x7f00c806b720 msgr2=0x7f00c80b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:34.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.309+0000 7f00cd800640 1 -- 172.21.15.90:0/2606301743 shutdown_connections 2024-04-25T01:17:34.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.310+0000 7f00cd800640 1 -- 172.21.15.90:0/2606301743 wait complete. 2024-04-25T01:17:34.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.311+0000 7f00cd800640 1 Processor -- start 2024-04-25T01:17:34.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.311+0000 7f00cd800640 1 -- start start 2024-04-25T01:17:34.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.312+0000 7f00cd800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00c80b37c0 0x7f00c8134a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:34.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.312+0000 7f00cd800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f00c8158a70 0x7f00c815a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:34.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.312+0000 7f00cd800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00c815b520 0x7f00c815a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:34.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.312+0000 7f00cd800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f00c806cb70 con 0x7f00c815b520 2024-04-25T01:17:34.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.312+0000 7f00cd800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f00c806c9f0 con 0x7f00c80b37c0 2024-04-25T01:17:34.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.312+0000 7f00cd800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f00c806ccf0 con 0x7f00c8158a70 2024-04-25T01:17:34.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.313+0000 7f00c7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00c815b520 0x7f00c815a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:34.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.313+0000 7f00c7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00c815b520 0x7f00c815a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33458/0 (socket says 172.21.15.90:33458) 2024-04-25T01:17:34.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.313+0000 7f00c7400640 1 -- 172.21.15.90:0/4177571113 learned_addr learned my addr 172.21.15.90:0/4177571113 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:34.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.313+0000 7f00c6a00640 1 --2- 172.21.15.90:0/4177571113 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00c80b37c0 0x7f00c8134a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:34.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.313+0000 7f00be000640 1 --2- 172.21.15.90:0/4177571113 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f00c8158a70 0x7f00c815a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:34.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.313+0000 7f00c7400640 1 -- 172.21.15.90:0/4177571113 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f00c8158a70 msgr2=0x7f00c815a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:34.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.313+0000 7f00c7400640 1 --2- 172.21.15.90:0/4177571113 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f00c8158a70 0x7f00c815a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:34.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.313+0000 7f00c7400640 1 -- 172.21.15.90:0/4177571113 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00c80b37c0 msgr2=0x7f00c8134a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:34.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.313+0000 7f00c7400640 1 --2- 172.21.15.90:0/4177571113 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00c80b37c0 0x7f00c8134a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:34.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.313+0000 7f00c7400640 1 -- 172.21.15.90:0/4177571113 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c81e61f0 con 0x7f00c815b520 2024-04-25T01:17:34.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.313+0000 7f00c7400640 1 --2- 172.21.15.90:0/4177571113 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00c815b520 0x7f00c815a830 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f00c000c730 tx=0x7f00c000cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:34.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.314+0000 7f00bf400640 1 -- 172.21.15.90:0/4177571113 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c0016020 con 0x7f00c815b520 2024-04-25T01:17:34.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.314+0000 7f00bf400640 1 -- 172.21.15.90:0/4177571113 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00c0006d50 con 0x7f00c815b520 2024-04-25T01:17:34.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.314+0000 7f00bf400640 1 -- 172.21.15.90:0/4177571113 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c0003bb0 con 0x7f00c815b520 2024-04-25T01:17:34.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.314+0000 7f00cd800640 1 -- 172.21.15.90:0/4177571113 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00c81e64d0 con 0x7f00c815b520 2024-04-25T01:17:34.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.314+0000 7f00cd800640 1 -- 172.21.15.90:0/4177571113 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00c81e6a90 con 0x7f00c815b520 2024-04-25T01:17:34.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.315+0000 7f00cd800640 1 -- 172.21.15.90:0/4177571113 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00c81e7870 con 0x7f00c815b520 2024-04-25T01:17:34.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.316+0000 7f00bf400640 1 -- 172.21.15.90:0/4177571113 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f00c00050d0 con 0x7f00c815b520 2024-04-25T01:17:34.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.317+0000 7f00bf400640 1 --2- 172.21.15.90:0/4177571113 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f00a406be30 0x7f00a406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:34.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.318+0000 7f00bf400640 1 -- 172.21.15.90:0/4177571113 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f00c008adc0 con 0x7f00c815b520 2024-04-25T01:17:34.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.318+0000 7f00c6a00640 1 --2- 172.21.15.90:0/4177571113 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f00a406be30 0x7f00a406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:34.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.318+0000 7f00c6a00640 1 --2- 172.21.15.90:0/4177571113 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f00a406be30 0x7f00a406e2f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f00b8006fd0 tx=0x7f00b8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:34.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.322+0000 7f00bf400640 1 -- 172.21.15.90:0/4177571113 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f00c0020020 con 0x7f00c815b520 2024-04-25T01:17:34.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.520+0000 7f00cd800640 1 -- 172.21.15.90:0/4177571113 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f00c8068fb0 con 0x7f00c815b520 2024-04-25T01:17:34.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.521+0000 7f00bf400640 1 -- 172.21.15.90:0/4177571113 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) ==== 100+0+11 (secure 0 0 0) 0x7f00c00593f0 con 0x7f00c815b520 2024-04-25T01:17:34.506 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-25T01:17:34.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.523+0000 7f00cd800640 1 -- 172.21.15.90:0/4177571113 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f00a406be30 msgr2=0x7f00a406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:34.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.523+0000 7f00cd800640 1 --2- 172.21.15.90:0/4177571113 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f00a406be30 0x7f00a406e2f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f00b8006fd0 tx=0x7f00b8008040 comp rx=0 tx=0).stop 2024-04-25T01:17:34.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.523+0000 7f00cd800640 1 -- 172.21.15.90:0/4177571113 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00c815b520 msgr2=0x7f00c815a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:34.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.523+0000 7f00cd800640 1 --2- 172.21.15.90:0/4177571113 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00c815b520 0x7f00c815a830 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f00c000c730 tx=0x7f00c000cb30 comp rx=0 tx=0).stop 2024-04-25T01:17:34.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.523+0000 7f00cd800640 1 -- 172.21.15.90:0/4177571113 shutdown_connections 2024-04-25T01:17:34.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.523+0000 7f00cd800640 1 --2- 172.21.15.90:0/4177571113 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f00a406be30 0x7f00a406e2f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:34.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.523+0000 7f00cd800640 1 --2- 172.21.15.90:0/4177571113 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00c815b520 0x7f00c815a830 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:34.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.523+0000 7f00cd800640 1 --2- 172.21.15.90:0/4177571113 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f00c8158a70 0x7f00c815a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:34.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.523+0000 7f00cd800640 1 --2- 172.21.15.90:0/4177571113 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00c80b37c0 0x7f00c8134a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:34.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.523+0000 7f00cd800640 1 -- 172.21.15.90:0/4177571113 >> 172.21.15.90:0/4177571113 conn(0x7f00c806b720 msgr2=0x7f00c815ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:34.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.524+0000 7f00cd800640 1 -- 172.21.15.90:0/4177571113 shutdown_connections 2024-04-25T01:17:34.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.524+0000 7f00cd800640 1 -- 172.21.15.90:0/4177571113 wait complete. 2024-04-25T01:17:34.527 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:17:34.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.745+0000 7fe41ca00640 1 Processor -- start 2024-04-25T01:17:34.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.745+0000 7fe41ca00640 1 -- start start 2024-04-25T01:17:34.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.746+0000 7fe41ca00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe4180722e0 0x7fe41806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:34.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.746+0000 7fe41ca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe41815fc40 0x7fe4181600c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:34.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.746+0000 7fe41ca00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe41806e410 con 0x7fe4180cedc0 2024-04-25T01:17:34.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.746+0000 7fe41ca00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe418160aa0 con 0x7fe4180722e0 2024-04-25T01:17:34.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.746+0000 7fe41ca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe41806da40 con 0x7fe41815fc40 2024-04-25T01:17:34.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.746+0000 7fe416800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe4180cedc0 0x7fe4180cf250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47396/0 (socket says 172.21.15.90:47396) 2024-04-25T01:17:34.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.746+0000 7fe416800640 1 -- 172.21.15.90:0/2089824910 learned_addr learned my addr 172.21.15.90:0/2089824910 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:34.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.746+0000 7fe415e00640 1 --2- 172.21.15.90:0/2089824910 >> v2:172.21.15.103:3300/0 conn(0x7fe4180722e0 0x7fe41806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:34.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.746+0000 7fe40fe00640 1 --2- 172.21.15.90:0/2089824910 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe41815fc40 0x7fe4181600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:34.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.747+0000 7fe415400640 1 -- 172.21.15.90:0/2089824910 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1188359137 0 0) 0x7fe41806e410 con 0x7fe4180cedc0 2024-04-25T01:17:34.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.747+0000 7fe415400640 1 -- 172.21.15.90:0/2089824910 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3fc003520 con 0x7fe4180cedc0 2024-04-25T01:17:34.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.747+0000 7fe415400640 1 -- 172.21.15.90:0/2089824910 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 933960842 0 0) 0x7fe3fc003520 con 0x7fe4180cedc0 2024-04-25T01:17:34.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.747+0000 7fe415400640 1 -- 172.21.15.90:0/2089824910 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe41815fc40 msgr2=0x7fe4181600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:34.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.747+0000 7fe415400640 1 --2- 172.21.15.90:0/2089824910 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe41815fc40 0x7fe4181600c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:34.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.747+0000 7fe415400640 1 -- 172.21.15.90:0/2089824910 >> v2:172.21.15.103:3300/0 conn(0x7fe4180722e0 msgr2=0x7fe41806ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:34.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.747+0000 7fe415400640 1 --2- 172.21.15.90:0/2089824910 >> v2:172.21.15.103:3300/0 conn(0x7fe4180722e0 0x7fe41806ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:34.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.747+0000 7fe415400640 1 -- 172.21.15.90:0/2089824910 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe41806d110 con 0x7fe4180cedc0 2024-04-25T01:17:34.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.747+0000 7fe415400640 1 -- 172.21.15.90:0/2089824910 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7fe4100019f0 con 0x7fe4180cedc0 2024-04-25T01:17:34.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.747+0000 7fe415400640 1 -- 172.21.15.90:0/2089824910 >> v1:172.21.15.90:6789/0 conn(0x7fe4180cedc0 legacy=0x7fe4180cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:34.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.747+0000 7fe415400640 1 --2- 172.21.15.90:0/2089824910 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe41815fc40 0x7fe3fc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:34.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.747+0000 7fe415400640 1 --2- 172.21.15.90:0/2089824910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3fc003f30 0x7fe3fc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:34.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.747+0000 7fe415400640 1 --2- 172.21.15.90:0/2089824910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3fc004d40 0x7fe3fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:34.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.747+0000 7fe415400640 1 -- 172.21.15.90:0/2089824910 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe418160aa0 con 0x7fe3fc004d40 2024-04-25T01:17:34.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.747+0000 7fe415400640 1 -- 172.21.15.90:0/2089824910 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe3fc00b9c0 con 0x7fe3fc003f30 2024-04-25T01:17:34.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.747+0000 7fe415400640 1 -- 172.21.15.90:0/2089824910 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe3fc00bb90 con 0x7fe41815fc40 2024-04-25T01:17:34.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.748+0000 7fe416800640 1 --2- 172.21.15.90:0/2089824910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3fc004d40 0x7fe3fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:34.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.748+0000 7fe415e00640 1 --2- 172.21.15.90:0/2089824910 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe41815fc40 0x7fe3fc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:34.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.748+0000 7fe40fe00640 1 --2- 172.21.15.90:0/2089824910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3fc003f30 0x7fe3fc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:34.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.748+0000 7fe416800640 1 -- 172.21.15.90:0/2089824910 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe41815fc40 msgr2=0x7fe3fc0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:34.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.748+0000 7fe416800640 1 --2- 172.21.15.90:0/2089824910 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe41815fc40 0x7fe3fc0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:34.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.748+0000 7fe416800640 1 -- 172.21.15.90:0/2089824910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3fc003f30 msgr2=0x7fe3fc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:34.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.748+0000 7fe416800640 1 --2- 172.21.15.90:0/2089824910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3fc003f30 0x7fe3fc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:34.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.748+0000 7fe416800640 1 -- 172.21.15.90:0/2089824910 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3fc003520 con 0x7fe3fc004d40 2024-04-25T01:17:34.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.748+0000 7fe40fe00640 1 --2- 172.21.15.90:0/2089824910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3fc003f30 0x7fe3fc004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:34.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.749+0000 7fe416800640 1 --2- 172.21.15.90:0/2089824910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3fc004d40 0x7fe3fc0051c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe41000a8d0 tx=0x7fe3fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f15363c76e45d2d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:34.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.749+0000 7fe415400640 1 -- 172.21.15.90:0/2089824910 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41000a4c0 con 0x7fe3fc004d40 2024-04-25T01:17:34.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.749+0000 7fe415400640 1 -- 172.21.15.90:0/2089824910 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe41002f070 con 0x7fe3fc004d40 2024-04-25T01:17:34.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.749+0000 7fe415400640 1 -- 172.21.15.90:0/2089824910 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41002bd40 con 0x7fe3fc004d40 2024-04-25T01:17:34.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.750+0000 7fe41ca00640 1 -- 172.21.15.90:0/2089824910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3fc004d40 msgr2=0x7fe3fc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:34.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.750+0000 7fe41ca00640 1 --2- 172.21.15.90:0/2089824910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3fc004d40 0x7fe3fc0051c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe41000a8d0 tx=0x7fe3fc005830 comp rx=0 tx=0).stop 2024-04-25T01:17:34.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.750+0000 7fe416800640 1 -- 172.21.15.90:0/2089824910 reap_dead start 2024-04-25T01:17:34.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.750+0000 7fe41ca00640 1 -- 172.21.15.90:0/2089824910 shutdown_connections 2024-04-25T01:17:34.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.750+0000 7fe41ca00640 1 -- 172.21.15.90:0/2089824910 >> 172.21.15.90:0/2089824910 conn(0x7fe41806b720 msgr2=0x7fe4180b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:34.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.750+0000 7fe41ca00640 1 -- 172.21.15.90:0/2089824910 shutdown_connections 2024-04-25T01:17:34.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.750+0000 7fe41ca00640 1 -- 172.21.15.90:0/2089824910 wait complete. 2024-04-25T01:17:34.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.752+0000 7fe41ca00640 1 Processor -- start 2024-04-25T01:17:34.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.752+0000 7fe41ca00640 1 -- start start 2024-04-25T01:17:34.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.753+0000 7fe41ca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3fc004d40 0x7fe41813fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:34.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.753+0000 7fe41ca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe418142ed0 0x7fe4180b0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:34.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.753+0000 7fe41ca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4180b0c40 0x7fe4181410e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:34.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.753+0000 7fe41ca00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe41806d6d0 con 0x7fe3fc004d40 2024-04-25T01:17:34.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.753+0000 7fe41ca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe41806cec0 con 0x7fe418142ed0 2024-04-25T01:17:34.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.753+0000 7fe41ca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe4181600c0 con 0x7fe4180b0c40 2024-04-25T01:17:34.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.753+0000 7fe415e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3fc004d40 0x7fe41813fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:34.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.753+0000 7fe415e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3fc004d40 0x7fe41813fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33468/0 (socket says 172.21.15.90:33468) 2024-04-25T01:17:34.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.753+0000 7fe415e00640 1 -- 172.21.15.90:0/252708331 learned_addr learned my addr 172.21.15.90:0/252708331 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:34.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.753+0000 7fe415e00640 1 -- 172.21.15.90:0/252708331 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4180b0c40 msgr2=0x7fe4181410e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:34.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.753+0000 7fe415e00640 1 --2- 172.21.15.90:0/252708331 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4180b0c40 0x7fe4181410e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:34.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.754+0000 7fe415e00640 1 -- 172.21.15.90:0/252708331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe418142ed0 msgr2=0x7fe4180b0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:34.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.754+0000 7fe415e00640 1 --2- 172.21.15.90:0/252708331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe418142ed0 0x7fe4180b0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:34.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.754+0000 7fe415e00640 1 -- 172.21.15.90:0/252708331 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4180af460 con 0x7fe3fc004d40 2024-04-25T01:17:34.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.754+0000 7fe415e00640 1 --2- 172.21.15.90:0/252708331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3fc004d40 0x7fe41813fe80 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fe408011730 tx=0x7fe408011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:34.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.754+0000 7fe40e000640 1 -- 172.21.15.90:0/252708331 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe40800f940 con 0x7fe3fc004d40 2024-04-25T01:17:34.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.754+0000 7fe40e000640 1 -- 172.21.15.90:0/252708331 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe40800fae0 con 0x7fe3fc004d40 2024-04-25T01:17:34.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.754+0000 7fe40e000640 1 -- 172.21.15.90:0/252708331 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe40800fe90 con 0x7fe3fc004d40 2024-04-25T01:17:34.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.754+0000 7fe41ca00640 1 -- 172.21.15.90:0/252708331 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe418143350 con 0x7fe3fc004d40 2024-04-25T01:17:34.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.754+0000 7fe41ca00640 1 -- 172.21.15.90:0/252708331 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe418158be0 con 0x7fe3fc004d40 2024-04-25T01:17:34.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.755+0000 7fe41ca00640 1 -- 172.21.15.90:0/252708331 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4180b1040 con 0x7fe3fc004d40 2024-04-25T01:17:34.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.756+0000 7fe40e000640 1 -- 172.21.15.90:0/252708331 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe408015460 con 0x7fe3fc004d40 2024-04-25T01:17:34.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.757+0000 7fe40e000640 1 --2- 172.21.15.90:0/252708331 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe3f406bfd0 0x7fe3f406e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:34.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.757+0000 7fe40e000640 1 -- 172.21.15.90:0/252708331 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7fe40808b770 con 0x7fe3fc004d40 2024-04-25T01:17:34.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.757+0000 7fe40fe00640 1 --2- 172.21.15.90:0/252708331 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe3f406bfd0 0x7fe3f406e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:34.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.758+0000 7fe40fe00640 1 --2- 172.21.15.90:0/252708331 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe3f406bfd0 0x7fe3f406e490 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe418155d60 tx=0x7fe400008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:34.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.763+0000 7fe40e000640 1 -- 172.21.15.90:0/252708331 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe418143350 con 0x7fe3fc004d40 2024-04-25T01:17:34.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.960+0000 7fe41ca00640 1 -- 172.21.15.90:0/252708331 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe418072c90 con 0x7fe3fc004d40 2024-04-25T01:17:34.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.961+0000 7fe40e000640 1 -- 172.21.15.90:0/252708331 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) ==== 103+0+11 (secure 0 0 0) 0x7fe408019030 con 0x7fe3fc004d40 2024-04-25T01:17:34.945 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:17:34.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.964+0000 7fe41ca00640 1 -- 172.21.15.90:0/252708331 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe3f406bfd0 msgr2=0x7fe3f406e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:34.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.964+0000 7fe41ca00640 1 --2- 172.21.15.90:0/252708331 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe3f406bfd0 0x7fe3f406e490 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe418155d60 tx=0x7fe400008040 comp rx=0 tx=0).stop 2024-04-25T01:17:34.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.964+0000 7fe41ca00640 1 -- 172.21.15.90:0/252708331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3fc004d40 msgr2=0x7fe41813fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:34.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.964+0000 7fe41ca00640 1 --2- 172.21.15.90:0/252708331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3fc004d40 0x7fe41813fe80 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fe408011730 tx=0x7fe408011bf0 comp rx=0 tx=0).stop 2024-04-25T01:17:34.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.964+0000 7fe41ca00640 1 -- 172.21.15.90:0/252708331 shutdown_connections 2024-04-25T01:17:34.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.964+0000 7fe41ca00640 1 --2- 172.21.15.90:0/252708331 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe3f406bfd0 0x7fe3f406e490 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:34.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.964+0000 7fe41ca00640 1 --2- 172.21.15.90:0/252708331 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4180b0c40 0x7fe4181410e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:34.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.964+0000 7fe41ca00640 1 --2- 172.21.15.90:0/252708331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe418142ed0 0x7fe4180b0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:34.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.964+0000 7fe41ca00640 1 --2- 172.21.15.90:0/252708331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3fc004d40 0x7fe41813fe80 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:34.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.964+0000 7fe41ca00640 1 -- 172.21.15.90:0/252708331 >> 172.21.15.90:0/252708331 conn(0x7fe41806b720 msgr2=0x7fe41815ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:34.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.964+0000 7fe41ca00640 1 -- 172.21.15.90:0/252708331 shutdown_connections 2024-04-25T01:17:34.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:34.964+0000 7fe41ca00640 1 -- 172.21.15.90:0/252708331 wait complete. 2024-04-25T01:17:34.969 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T01:17:35.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.187+0000 7f4c29000640 1 Processor -- start 2024-04-25T01:17:35.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.187+0000 7f4c29000640 1 -- start start 2024-04-25T01:17:35.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.189+0000 7f4c29000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4c240cefe0 0x7f4c240b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:35.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.189+0000 7f4c29000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c2406e000 0x7f4c24163440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:35.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.189+0000 7f4c29000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4c240b2b20 0x7f4c240b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:35.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.189+0000 7f4c29000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f4c24163980 con 0x7f4c240b2b20 2024-04-25T01:17:35.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.189+0000 7f4c29000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4c2406cf60 con 0x7f4c240cefe0 2024-04-25T01:17:35.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.189+0000 7f4c29000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4c2406d240 con 0x7f4c2406e000 2024-04-25T01:17:35.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.189+0000 7f4c22800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4c240b2b20 0x7f4c240b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:35.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.189+0000 7f4c22800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4c240b2b20 0x7f4c240b2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56932/0 (socket says 172.21.15.90:56932) 2024-04-25T01:17:35.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.189+0000 7f4c22800640 1 -- 172.21.15.90:0/1510835176 learned_addr learned my addr 172.21.15.90:0/1510835176 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:35.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.189+0000 7f4c21e00640 1 --2- 172.21.15.90:0/1510835176 >> v2:172.21.15.103:3300/0 conn(0x7f4c240cefe0 0x7f4c240b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:35.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.189+0000 7f4c21400640 1 --2- 172.21.15.90:0/1510835176 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c2406e000 0x7f4c24163440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:35.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.189+0000 7f4c22800640 1 -- 172.21.15.90:0/1510835176 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c2406e000 msgr2=0x7f4c24163440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:35.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.189+0000 7f4c22800640 1 --2- 172.21.15.90:0/1510835176 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c2406e000 0x7f4c24163440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:35.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.189+0000 7f4c22800640 1 -- 172.21.15.90:0/1510835176 >> v2:172.21.15.103:3300/0 conn(0x7f4c240cefe0 msgr2=0x7f4c240b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:35.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.189+0000 7f4c22800640 1 --2- 172.21.15.90:0/1510835176 >> v2:172.21.15.103:3300/0 conn(0x7f4c240cefe0 0x7f4c240b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:35.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.189+0000 7f4c22800640 1 -- 172.21.15.90:0/1510835176 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c2406d520 con 0x7f4c240b2b20 2024-04-25T01:17:35.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.189+0000 7f4c21400640 1 --2- 172.21.15.90:0/1510835176 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c2406e000 0x7f4c24163440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:35.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.190+0000 7f4c22800640 1 --2- 172.21.15.90:0/1510835176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c240b2b20 0x7f4c240b2fb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4c240739a0 tx=0x7f4c1c02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f31d3eda3db8574 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:35.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.190+0000 7f4c20a00640 1 -- 172.21.15.90:0/1510835176 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c036070 con 0x7f4c240b2b20 2024-04-25T01:17:35.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.190+0000 7f4c20a00640 1 -- 172.21.15.90:0/1510835176 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c1c031600 con 0x7f4c240b2b20 2024-04-25T01:17:35.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.190+0000 7f4c20a00640 1 -- 172.21.15.90:0/1510835176 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c0319d0 con 0x7f4c240b2b20 2024-04-25T01:17:35.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.191+0000 7f4c29000640 1 -- 172.21.15.90:0/1510835176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c240b2b20 msgr2=0x7f4c240b2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:35.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.191+0000 7f4c29000640 1 --2- 172.21.15.90:0/1510835176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c240b2b20 0x7f4c240b2fb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4c240739a0 tx=0x7f4c1c02a610 comp rx=0 tx=0).stop 2024-04-25T01:17:35.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.191+0000 7f4c29000640 1 -- 172.21.15.90:0/1510835176 shutdown_connections 2024-04-25T01:17:35.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.191+0000 7f4c29000640 1 --2- 172.21.15.90:0/1510835176 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c2406e000 0x7f4c24163440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:35.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.191+0000 7f4c29000640 1 --2- 172.21.15.90:0/1510835176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c240b2b20 0x7f4c240b2fb0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:35.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.191+0000 7f4c29000640 1 --2- 172.21.15.90:0/1510835176 >> v2:172.21.15.103:3300/0 conn(0x7f4c240cefe0 0x7f4c240b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:35.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.191+0000 7f4c29000640 1 -- 172.21.15.90:0/1510835176 >> 172.21.15.90:0/1510835176 conn(0x7f4c240b1820 msgr2=0x7f4c240b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:35.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.191+0000 7f4c29000640 1 -- 172.21.15.90:0/1510835176 shutdown_connections 2024-04-25T01:17:35.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.191+0000 7f4c29000640 1 -- 172.21.15.90:0/1510835176 wait complete. 2024-04-25T01:17:35.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.193+0000 7f4c29000640 1 Processor -- start 2024-04-25T01:17:35.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.193+0000 7f4c29000640 1 -- start start 2024-04-25T01:17:35.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.194+0000 7f4c29000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c2406e000 0x7f4c24147d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:35.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.194+0000 7f4c29000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c240b2b20 0x7f4c24156fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:35.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.194+0000 7f4c29000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c240cefe0 0x7f4c240b08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:35.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.194+0000 7f4c29000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2406e490 con 0x7f4c2406e000 2024-04-25T01:17:35.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.194+0000 7f4c29000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2406dae0 con 0x7f4c240cefe0 2024-04-25T01:17:35.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.194+0000 7f4c29000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4c24070c50 con 0x7f4c240b2b20 2024-04-25T01:17:35.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.194+0000 7f4c21e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c2406e000 0x7f4c24147d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:35.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.194+0000 7f4c21e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c2406e000 0x7f4c24147d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56944/0 (socket says 172.21.15.90:56944) 2024-04-25T01:17:35.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.194+0000 7f4c21e00640 1 -- 172.21.15.90:0/3382235419 learned_addr learned my addr 172.21.15.90:0/3382235419 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:35.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.194+0000 7f4c21e00640 1 -- 172.21.15.90:0/3382235419 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c240b2b20 msgr2=0x7f4c24156fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:35.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.194+0000 7f4c21400640 1 --2- 172.21.15.90:0/3382235419 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c240b2b20 0x7f4c24156fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:35.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.195+0000 7f4c22800640 1 --2- 172.21.15.90:0/3382235419 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c240cefe0 0x7f4c240b08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:35.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.195+0000 7f4c21e00640 1 --2- 172.21.15.90:0/3382235419 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c240b2b20 0x7f4c24156fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:35.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.195+0000 7f4c21e00640 1 -- 172.21.15.90:0/3382235419 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c240cefe0 msgr2=0x7f4c240b08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:35.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.195+0000 7f4c21e00640 1 --2- 172.21.15.90:0/3382235419 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c240cefe0 0x7f4c240b08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:35.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.195+0000 7f4c21e00640 1 -- 172.21.15.90:0/3382235419 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c241ebdd0 con 0x7f4c2406e000 2024-04-25T01:17:35.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.195+0000 7f4c22800640 1 --2- 172.21.15.90:0/3382235419 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c240cefe0 0x7f4c240b08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:35.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.195+0000 7f4c21e00640 1 --2- 172.21.15.90:0/3382235419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c2406e000 0x7f4c24147d60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4c1400d5f0 tx=0x7f4c1400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:35.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.195+0000 7f4c1a000640 1 -- 172.21.15.90:0/3382235419 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c140125f0 con 0x7f4c2406e000 2024-04-25T01:17:35.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.195+0000 7f4c1a000640 1 -- 172.21.15.90:0/3382235419 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c14019070 con 0x7f4c2406e000 2024-04-25T01:17:35.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.195+0000 7f4c1a000640 1 -- 172.21.15.90:0/3382235419 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c140129d0 con 0x7f4c2406e000 2024-04-25T01:17:35.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.195+0000 7f4c29000640 1 -- 172.21.15.90:0/3382235419 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c241ec0b0 con 0x7f4c2406e000 2024-04-25T01:17:35.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.195+0000 7f4c29000640 1 -- 172.21.15.90:0/3382235419 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c241ec670 con 0x7f4c2406e000 2024-04-25T01:17:35.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.196+0000 7f4c29000640 1 -- 172.21.15.90:0/3382235419 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c240679d0 con 0x7f4c2406e000 2024-04-25T01:17:35.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.197+0000 7f4c1a000640 1 -- 172.21.15.90:0/3382235419 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4c14012b70 con 0x7f4c2406e000 2024-04-25T01:17:35.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.198+0000 7f4c1a000640 1 --2- 172.21.15.90:0/3382235419 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4c0406be30 0x7f4c0406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:35.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.198+0000 7f4c1a000640 1 -- 172.21.15.90:0/3382235419 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f4c1408b670 con 0x7f4c2406e000 2024-04-25T01:17:35.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.199+0000 7f4c21400640 1 --2- 172.21.15.90:0/3382235419 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4c0406be30 0x7f4c0406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:35.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.199+0000 7f4c21400640 1 --2- 172.21.15.90:0/3382235419 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4c0406be30 0x7f4c0406e2f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4c10006fd0 tx=0x7f4c10008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:35.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.203+0000 7f4c1a000640 1 -- 172.21.15.90:0/3382235419 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4c240679d0 con 0x7f4c2406e000 2024-04-25T01:17:35.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.415+0000 7f4c29000640 1 -- 172.21.15.90:0/3382235419 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4c24068fb0 con 0x7f4c2406e000 2024-04-25T01:17:35.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.416+0000 7f4c1a000640 1 -- 172.21.15.90:0/3382235419 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v28) ==== 80+0+0 (secure 0 0 0) 0x7f4c14059c10 con 0x7f4c2406e000 2024-04-25T01:17:35.401 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-04-25T01:17:35.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.419+0000 7f4c29000640 1 -- 172.21.15.90:0/3382235419 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4c0406be30 msgr2=0x7f4c0406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:35.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.419+0000 7f4c29000640 1 --2- 172.21.15.90:0/3382235419 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4c0406be30 0x7f4c0406e2f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4c10006fd0 tx=0x7f4c10008040 comp rx=0 tx=0).stop 2024-04-25T01:17:35.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.419+0000 7f4c29000640 1 -- 172.21.15.90:0/3382235419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c2406e000 msgr2=0x7f4c24147d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:35.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.419+0000 7f4c29000640 1 --2- 172.21.15.90:0/3382235419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c2406e000 0x7f4c24147d60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4c1400d5f0 tx=0x7f4c1400dab0 comp rx=0 tx=0).stop 2024-04-25T01:17:35.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.419+0000 7f4c29000640 1 -- 172.21.15.90:0/3382235419 shutdown_connections 2024-04-25T01:17:35.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.419+0000 7f4c29000640 1 --2- 172.21.15.90:0/3382235419 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4c0406be30 0x7f4c0406e2f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:35.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.419+0000 7f4c29000640 1 --2- 172.21.15.90:0/3382235419 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c240cefe0 0x7f4c240b08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:35.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.419+0000 7f4c29000640 1 --2- 172.21.15.90:0/3382235419 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c240b2b20 0x7f4c24156fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:35.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.419+0000 7f4c29000640 1 --2- 172.21.15.90:0/3382235419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c2406e000 0x7f4c24147d60 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:35.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.419+0000 7f4c29000640 1 -- 172.21.15.90:0/3382235419 >> 172.21.15.90:0/3382235419 conn(0x7f4c240b1820 msgr2=0x7f4c24163b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:35.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.419+0000 7f4c29000640 1 -- 172.21.15.90:0/3382235419 shutdown_connections 2024-04-25T01:17:35.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.419+0000 7f4c29000640 1 -- 172.21.15.90:0/3382235419 wait complete. 2024-04-25T01:17:35.423 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-25T01:17:35.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:35.423 DEBUG:teuthology.orchestra.run.smithi090:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-04-25T01:17:35.510 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-04-25T01:17:35.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:35.511 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T01:17:35.564 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:17:35.565 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:17:35.621 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:17:35.622 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:17:35.675 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:17:35.675 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:17:35.727 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:17:35.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.957+0000 7fcfe9600640 1 Processor -- start 2024-04-25T01:17:35.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.957+0000 7fcfe9600640 1 -- start start 2024-04-25T01:17:35.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.958+0000 7fcfe9600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcfe415ad40 0x7fcfe406b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:35.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.959+0000 7fcfe9600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcfe406d910 0x7fcfe40cd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:35.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.959+0000 7fcfe9600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfe406ba90 con 0x7fcfe40cf520 2024-04-25T01:17:35.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.959+0000 7fcfe9600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fcfe406cb50 con 0x7fcfe415ad40 2024-04-25T01:17:35.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.959+0000 7fcfe9600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcfe406ce30 con 0x7fcfe406d910 2024-04-25T01:17:35.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.959+0000 7fcfe3e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcfe415ad40 0x7fcfe406b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:35.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.959+0000 7fcfe3e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcfe415ad40 0x7fcfe406b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56958/0 (socket says 172.21.15.90:56958) 2024-04-25T01:17:35.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.959+0000 7fcfe3e00640 1 -- 172.21.15.90:0/2117564236 learned_addr learned my addr 172.21.15.90:0/2117564236 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:35.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.959+0000 7fcfe3e00640 1 -- 172.21.15.90:0/2117564236 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcfe406d910 msgr2=0x7fcfe40cd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:35.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.959+0000 7fcfe3e00640 1 --2- 172.21.15.90:0/2117564236 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcfe406d910 0x7fcfe40cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:35.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.959+0000 7fcfe3e00640 1 -- 172.21.15.90:0/2117564236 >> v1:172.21.15.103:6789/0 conn(0x7fcfe40cf520 legacy=0x7fcfe40ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:35.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.959+0000 7fcfe3e00640 1 -- 172.21.15.90:0/2117564236 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfe406d110 con 0x7fcfe415ad40 2024-04-25T01:17:35.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.960+0000 7fcfe3e00640 1 --2- 172.21.15.90:0/2117564236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfe415ad40 0x7fcfe406b120 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4009870 tx=0x7fcfd402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbadc4db6bfaabc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:35.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.961+0000 7fcfe3400640 1 -- 172.21.15.90:0/2117564236 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd4037070 con 0x7fcfe415ad40 2024-04-25T01:17:35.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.961+0000 7fcfe3400640 1 -- 172.21.15.90:0/2117564236 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfd4032440 con 0x7fcfe415ad40 2024-04-25T01:17:35.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.961+0000 7fcfe3400640 1 -- 172.21.15.90:0/2117564236 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd4032810 con 0x7fcfe415ad40 2024-04-25T01:17:35.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.961+0000 7fcfe9600640 1 -- 172.21.15.90:0/2117564236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfe415ad40 msgr2=0x7fcfe406b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:35.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.961+0000 7fcfe9600640 1 --2- 172.21.15.90:0/2117564236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfe415ad40 0x7fcfe406b120 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4009870 tx=0x7fcfd402a230 comp rx=0 tx=0).stop 2024-04-25T01:17:35.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.962+0000 7fcfe9600640 1 -- 172.21.15.90:0/2117564236 shutdown_connections 2024-04-25T01:17:35.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.962+0000 7fcfe9600640 1 --2- 172.21.15.90:0/2117564236 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcfe406d910 0x7fcfe40cd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:35.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.962+0000 7fcfe9600640 1 --2- 172.21.15.90:0/2117564236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfe415ad40 0x7fcfe406b120 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:35.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.962+0000 7fcfe9600640 1 -- 172.21.15.90:0/2117564236 >> 172.21.15.90:0/2117564236 conn(0x7fcfe40b1660 msgr2=0x7fcfe40b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:35.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.962+0000 7fcfe9600640 1 -- 172.21.15.90:0/2117564236 shutdown_connections 2024-04-25T01:17:35.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.962+0000 7fcfe9600640 1 -- 172.21.15.90:0/2117564236 wait complete. 2024-04-25T01:17:35.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.963+0000 7fcfe9600640 1 Processor -- start 2024-04-25T01:17:35.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.964+0000 7fcfe9600640 1 -- start start 2024-04-25T01:17:35.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.964+0000 7fcfe9600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfe406d910 0x7fcfe414c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:35.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.965+0000 7fcfe9600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcfe40cf520 0x7fcfe415bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:35.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.965+0000 7fcfe9600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfe415ad40 0x7fcfe41f7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:35.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.965+0000 7fcfe9600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe406b7e0 con 0x7fcfe415ad40 2024-04-25T01:17:35.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.965+0000 7fcfe9600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe406b660 con 0x7fcfe406d910 2024-04-25T01:17:35.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.965+0000 7fcfe9600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcfe40aee40 con 0x7fcfe40cf520 2024-04-25T01:17:35.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.965+0000 7fcfe8c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfe415ad40 0x7fcfe41f7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:35.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.965+0000 7fcfe8c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfe415ad40 0x7fcfe41f7c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56964/0 (socket says 172.21.15.90:56964) 2024-04-25T01:17:35.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.965+0000 7fcfe8c00640 1 -- 172.21.15.90:0/1121463476 learned_addr learned my addr 172.21.15.90:0/1121463476 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:35.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.965+0000 7fcfdbe00640 1 --2- 172.21.15.90:0/1121463476 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcfe40cf520 0x7fcfe415bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:35.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.965+0000 7fcfe3e00640 1 --2- 172.21.15.90:0/1121463476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfe406d910 0x7fcfe414c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:35.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.965+0000 7fcfe8c00640 1 -- 172.21.15.90:0/1121463476 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcfe40cf520 msgr2=0x7fcfe415bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:35.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.965+0000 7fcfe8c00640 1 --2- 172.21.15.90:0/1121463476 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcfe40cf520 0x7fcfe415bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:35.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.965+0000 7fcfe8c00640 1 -- 172.21.15.90:0/1121463476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfe406d910 msgr2=0x7fcfe414c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:35.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.965+0000 7fcfe8c00640 1 --2- 172.21.15.90:0/1121463476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfe406d910 0x7fcfe414c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:35.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.965+0000 7fcfe8c00640 1 -- 172.21.15.90:0/1121463476 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfe41f86a0 con 0x7fcfe415ad40 2024-04-25T01:17:35.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.965+0000 7fcfe3e00640 1 --2- 172.21.15.90:0/1121463476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfe406d910 0x7fcfe414c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:35.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.966+0000 7fcfe8c00640 1 --2- 172.21.15.90:0/1121463476 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfe415ad40 0x7fcfe41f7c40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc00b9e0 tx=0x7fcfdc00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:35.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.966+0000 7fcfe0c00640 1 -- 172.21.15.90:0/1121463476 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc019070 con 0x7fcfe415ad40 2024-04-25T01:17:35.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.966+0000 7fcfe0c00640 1 -- 172.21.15.90:0/1121463476 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfdc00cb40 con 0x7fcfe415ad40 2024-04-25T01:17:35.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.966+0000 7fcfe9600640 1 -- 172.21.15.90:0/1121463476 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfe41f8980 con 0x7fcfe415ad40 2024-04-25T01:17:35.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.966+0000 7fcfe0c00640 1 -- 172.21.15.90:0/1121463476 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc014440 con 0x7fcfe415ad40 2024-04-25T01:17:35.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.966+0000 7fcfe9600640 1 -- 172.21.15.90:0/1121463476 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfe414e770 con 0x7fcfe415ad40 2024-04-25T01:17:35.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.967+0000 7fcfe9600640 1 -- 172.21.15.90:0/1121463476 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfe415b120 con 0x7fcfe415ad40 2024-04-25T01:17:35.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.968+0000 7fcfe0c00640 1 -- 172.21.15.90:0/1121463476 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fcfdc01c070 con 0x7fcfe415ad40 2024-04-25T01:17:35.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.969+0000 7fcfe0c00640 1 --2- 172.21.15.90:0/1121463476 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fcfc006be30 0x7fcfc006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:35.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.969+0000 7fcfe0c00640 1 -- 172.21.15.90:0/1121463476 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7fcfdc08af40 con 0x7fcfe415ad40 2024-04-25T01:17:35.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.969+0000 7fcfe3e00640 1 --2- 172.21.15.90:0/1121463476 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fcfc006be30 0x7fcfc006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:35.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.970+0000 7fcfe3e00640 1 --2- 172.21.15.90:0/1121463476 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fcfc006be30 0x7fcfc006e2f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4034000 tx=0x7fcfd402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:35.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:35.974+0000 7fcfe0c00640 1 -- 172.21.15.90:0/1121463476 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fcfe415b120 con 0x7fcfe415ad40 2024-04-25T01:17:36.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.196+0000 7fcfe9600640 1 -- 172.21.15.90:0/1121463476 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcfe4075350 con 0x7fcfe415ad40 2024-04-25T01:17:36.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.198+0000 7fcfe0c00640 1 -- 172.21.15.90:0/1121463476 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18576 (secure 0 0 0) 0x7fcfdc0594e0 con 0x7fcfe415ad40 2024-04-25T01:17:36.182 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:17:36.183 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:17:09.346008+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:16:20.094512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:16:21.078268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:16:24.670642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.e","mappings":[{"from":0,"to":7}]},{"pgid":"4.f","mappings":[{"from":7,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:17:36.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.201+0000 7fcfe9600640 1 -- 172.21.15.90:0/1121463476 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fcfc006be30 msgr2=0x7fcfc006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:36.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.201+0000 7fcfe9600640 1 --2- 172.21.15.90:0/1121463476 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fcfc006be30 0x7fcfc006e2f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4034000 tx=0x7fcfd402e000 comp rx=0 tx=0).stop 2024-04-25T01:17:36.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.201+0000 7fcfe9600640 1 -- 172.21.15.90:0/1121463476 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfe415ad40 msgr2=0x7fcfe41f7c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:36.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.201+0000 7fcfe9600640 1 --2- 172.21.15.90:0/1121463476 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfe415ad40 0x7fcfe41f7c40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc00b9e0 tx=0x7fcfdc00bea0 comp rx=0 tx=0).stop 2024-04-25T01:17:36.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.202+0000 7fcfe9600640 1 -- 172.21.15.90:0/1121463476 shutdown_connections 2024-04-25T01:17:36.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.202+0000 7fcfe9600640 1 --2- 172.21.15.90:0/1121463476 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fcfc006be30 0x7fcfc006e2f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:36.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.202+0000 7fcfe9600640 1 --2- 172.21.15.90:0/1121463476 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfe415ad40 0x7fcfe41f7c40 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:36.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.202+0000 7fcfe9600640 1 --2- 172.21.15.90:0/1121463476 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcfe40cf520 0x7fcfe415bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:36.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.202+0000 7fcfe9600640 1 --2- 172.21.15.90:0/1121463476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfe406d910 0x7fcfe414c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:36.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.202+0000 7fcfe9600640 1 -- 172.21.15.90:0/1121463476 >> 172.21.15.90:0/1121463476 conn(0x7fcfe40b1660 msgr2=0x7fcfe40b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:36.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.202+0000 7fcfe9600640 1 -- 172.21.15.90:0/1121463476 shutdown_connections 2024-04-25T01:17:36.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.202+0000 7fcfe9600640 1 -- 172.21.15.90:0/1121463476 wait complete. 2024-04-25T01:17:36.212 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:16:15.894126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:16:20.094512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:16:21.078268+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:16:24.670642+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:17:36.213 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:17:36.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.426+0000 7f14d4e00640 1 Processor -- start 2024-04-25T01:17:36.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.426+0000 7f14d4e00640 1 -- start start 2024-04-25T01:17:36.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.427+0000 7f14d4e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14d00722e0 0x7f14d006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:36.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.427+0000 7f14d4e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f14d00cf4d0 0x7f14d01600c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:36.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.427+0000 7f14d4e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14d006e410 con 0x7f14d015fc40 2024-04-25T01:17:36.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.427+0000 7f14d4e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f14d0160aa0 con 0x7f14d00cf4d0 2024-04-25T01:17:36.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.427+0000 7f14d4e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f14d006da40 con 0x7f14d00722e0 2024-04-25T01:17:36.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.428+0000 7f14ce800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f14d00cf4d0 0x7f14d01600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:36.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.428+0000 7f14ce800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f14d00cf4d0 0x7f14d01600c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56976/0 (socket says 172.21.15.90:56976) 2024-04-25T01:17:36.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.428+0000 7f14ce800640 1 -- 172.21.15.90:0/135657454 learned_addr learned my addr 172.21.15.90:0/135657454 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:36.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.428+0000 7f14cde00640 1 --2- 172.21.15.90:0/135657454 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14d00722e0 0x7f14d006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:36.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.428+0000 7f14ce800640 1 -- 172.21.15.90:0/135657454 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14d00722e0 msgr2=0x7f14d006ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:36.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.428+0000 7f14ce800640 1 --2- 172.21.15.90:0/135657454 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14d00722e0 0x7f14d006ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:36.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.428+0000 7f14ce800640 1 -- 172.21.15.90:0/135657454 >> v1:172.21.15.103:6789/0 conn(0x7f14d015fc40 legacy=0x7f14d00cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:36.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.428+0000 7f14ce800640 1 -- 172.21.15.90:0/135657454 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14d006d110 con 0x7f14d00cf4d0 2024-04-25T01:17:36.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.429+0000 7f14ce800640 1 --2- 172.21.15.90:0/135657454 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14d00cf4d0 0x7f14d01600c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f14c8009e50 tx=0x7f14c802a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aa0bee1342657b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:36.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.430+0000 7f14cd400640 1 -- 172.21.15.90:0/135657454 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c8036070 con 0x7f14d00cf4d0 2024-04-25T01:17:36.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.430+0000 7f14cd400640 1 -- 172.21.15.90:0/135657454 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14c8031600 con 0x7f14d00cf4d0 2024-04-25T01:17:36.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.430+0000 7f14cd400640 1 -- 172.21.15.90:0/135657454 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c80319d0 con 0x7f14d00cf4d0 2024-04-25T01:17:36.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.430+0000 7f14d4e00640 1 -- 172.21.15.90:0/135657454 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14d00cf4d0 msgr2=0x7f14d01600c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:36.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.430+0000 7f14d4e00640 1 --2- 172.21.15.90:0/135657454 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14d00cf4d0 0x7f14d01600c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f14c8009e50 tx=0x7f14c802a630 comp rx=0 tx=0).stop 2024-04-25T01:17:36.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.430+0000 7f14d4e00640 1 -- 172.21.15.90:0/135657454 shutdown_connections 2024-04-25T01:17:36.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.430+0000 7f14d4e00640 1 --2- 172.21.15.90:0/135657454 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14d00cf4d0 0x7f14d01600c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:36.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.430+0000 7f14d4e00640 1 --2- 172.21.15.90:0/135657454 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14d00722e0 0x7f14d006ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:36.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.430+0000 7f14d4e00640 1 -- 172.21.15.90:0/135657454 >> 172.21.15.90:0/135657454 conn(0x7f14d006b720 msgr2=0x7f14d00b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:36.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.431+0000 7f14d4e00640 1 -- 172.21.15.90:0/135657454 shutdown_connections 2024-04-25T01:17:36.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.431+0000 7f14d4e00640 1 -- 172.21.15.90:0/135657454 wait complete. 2024-04-25T01:17:36.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.432+0000 7f14d4e00640 1 Processor -- start 2024-04-25T01:17:36.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.432+0000 7f14d4e00640 1 -- start start 2024-04-25T01:17:36.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.433+0000 7f14d4e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14d00722e0 0x7f14d0144760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:36.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.433+0000 7f14d4e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14d00cf4d0 0x7f14d0159a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:36.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.433+0000 7f14d4e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14d015fc40 0x7f14d0157050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:36.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.433+0000 7f14d4e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f14d006d6d0 con 0x7f14d00722e0 2024-04-25T01:17:36.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.433+0000 7f14d4e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f14d006cec0 con 0x7f14d015fc40 2024-04-25T01:17:36.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.433+0000 7f14d4e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f14d0070b50 con 0x7f14d00cf4d0 2024-04-25T01:17:36.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.433+0000 7f14cde00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14d00722e0 0x7f14d0144760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:36.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.433+0000 7f14cde00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14d00722e0 0x7f14d0144760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56990/0 (socket says 172.21.15.90:56990) 2024-04-25T01:17:36.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.433+0000 7f14cde00640 1 -- 172.21.15.90:0/987415750 learned_addr learned my addr 172.21.15.90:0/987415750 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:36.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.434+0000 7f14cde00640 1 -- 172.21.15.90:0/987415750 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14d00cf4d0 msgr2=0x7f14d0159a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:36.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.434+0000 7f14cde00640 1 --2- 172.21.15.90:0/987415750 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14d00cf4d0 0x7f14d0159a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:36.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.434+0000 7f14cde00640 1 -- 172.21.15.90:0/987415750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14d015fc40 msgr2=0x7f14d0157050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:36.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.434+0000 7f14cde00640 1 --2- 172.21.15.90:0/987415750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14d015fc40 0x7f14d0157050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:36.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.434+0000 7f14cde00640 1 -- 172.21.15.90:0/987415750 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14d0145150 con 0x7f14d00722e0 2024-04-25T01:17:36.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.434+0000 7f14cde00640 1 --2- 172.21.15.90:0/987415750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14d00722e0 0x7f14d0144760 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f14c000d830 tx=0x7f14c000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:36.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.434+0000 7f14c6a00640 1 -- 172.21.15.90:0/987415750 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c000b6c0 con 0x7f14d00722e0 2024-04-25T01:17:36.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.434+0000 7f14c6a00640 1 -- 172.21.15.90:0/987415750 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14c0019070 con 0x7f14d00722e0 2024-04-25T01:17:36.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.434+0000 7f14c6a00640 1 -- 172.21.15.90:0/987415750 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c000ba80 con 0x7f14d00722e0 2024-04-25T01:17:36.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.434+0000 7f14d4e00640 1 -- 172.21.15.90:0/987415750 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14d0145430 con 0x7f14d00722e0 2024-04-25T01:17:36.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.434+0000 7f14d4e00640 1 -- 172.21.15.90:0/987415750 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14d01459f0 con 0x7f14d00722e0 2024-04-25T01:17:36.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.436+0000 7f14d4e00640 1 -- 172.21.15.90:0/987415750 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14d0068fb0 con 0x7f14d00722e0 2024-04-25T01:17:36.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.436+0000 7f14c6a00640 1 -- 172.21.15.90:0/987415750 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f14c001e460 con 0x7f14d00722e0 2024-04-25T01:17:36.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.437+0000 7f14c6a00640 1 --2- 172.21.15.90:0/987415750 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f14ac06be30 0x7f14ac06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:36.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.437+0000 7f14c6a00640 1 -- 172.21.15.90:0/987415750 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f14c008b970 con 0x7f14d00722e0 2024-04-25T01:17:36.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.438+0000 7f14c5400640 1 --2- 172.21.15.90:0/987415750 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f14ac06be30 0x7f14ac06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:36.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.438+0000 7f14c5400640 1 --2- 172.21.15.90:0/987415750 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f14ac06be30 0x7f14ac06e2f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f14b8006fd0 tx=0x7f14b8006bc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:36.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.441+0000 7f14c6a00640 1 -- 172.21.15.90:0/987415750 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f14c0059e80 con 0x7f14d00722e0 2024-04-25T01:17:36.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.633+0000 7f14d4e00640 1 -- 172.21.15.90:0/987415750 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f14d0072c90 con 0x7f14d00722e0 2024-04-25T01:17:36.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.634+0000 7f14c6a00640 1 -- 172.21.15.90:0/987415750 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) ==== 93+0+10 (secure 0 0 0) 0x7f14c0059780 con 0x7f14d00722e0 2024-04-25T01:17:36.617 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T01:17:36.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.636+0000 7f14d4e00640 1 -- 172.21.15.90:0/987415750 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f14ac06be30 msgr2=0x7f14ac06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:36.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.636+0000 7f14d4e00640 1 --2- 172.21.15.90:0/987415750 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f14ac06be30 0x7f14ac06e2f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f14b8006fd0 tx=0x7f14b8006bc0 comp rx=0 tx=0).stop 2024-04-25T01:17:36.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.636+0000 7f14d4e00640 1 -- 172.21.15.90:0/987415750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14d00722e0 msgr2=0x7f14d0144760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:36.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.636+0000 7f14d4e00640 1 --2- 172.21.15.90:0/987415750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14d00722e0 0x7f14d0144760 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f14c000d830 tx=0x7f14c000dcf0 comp rx=0 tx=0).stop 2024-04-25T01:17:36.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.636+0000 7f14d4e00640 1 -- 172.21.15.90:0/987415750 shutdown_connections 2024-04-25T01:17:36.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.636+0000 7f14d4e00640 1 --2- 172.21.15.90:0/987415750 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f14ac06be30 0x7f14ac06e2f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:36.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.636+0000 7f14d4e00640 1 --2- 172.21.15.90:0/987415750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14d015fc40 0x7f14d0157050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:36.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.636+0000 7f14d4e00640 1 --2- 172.21.15.90:0/987415750 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f14d00cf4d0 0x7f14d0159a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:36.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.636+0000 7f14d4e00640 1 --2- 172.21.15.90:0/987415750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14d00722e0 0x7f14d0144760 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:36.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.636+0000 7f14d4e00640 1 -- 172.21.15.90:0/987415750 >> 172.21.15.90:0/987415750 conn(0x7f14d006b720 msgr2=0x7f14d00ae510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:36.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.636+0000 7f14d4e00640 1 -- 172.21.15.90:0/987415750 shutdown_connections 2024-04-25T01:17:36.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.636+0000 7f14d4e00640 1 -- 172.21.15.90:0/987415750 wait complete. 2024-04-25T01:17:36.638 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:17:36.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.849+0000 7f57b5a00640 1 Processor -- start 2024-04-25T01:17:36.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.849+0000 7f57b5a00640 1 -- start start 2024-04-25T01:17:36.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.851+0000 7f57b5a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f57b00722e0 0x7f57b006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:36.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.851+0000 7f57b5a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57b00cf4d0 0x7f57b01600c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:36.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.851+0000 7f57b5a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57b006e410 con 0x7f57b015fc40 2024-04-25T01:17:36.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.851+0000 7f57b5a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f57b0160aa0 con 0x7f57b00722e0 2024-04-25T01:17:36.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.851+0000 7f57b5a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f57b006da40 con 0x7f57b00cf4d0 2024-04-25T01:17:36.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.851+0000 7f57a6000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f57b015fc40 0x7f57b00cedc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:35154/0 (socket says 172.21.15.90:35154) 2024-04-25T01:17:36.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.851+0000 7f57a6000640 1 -- 172.21.15.90:0/2573527147 learned_addr learned my addr 172.21.15.90:0/2573527147 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:36.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.851+0000 7f57aea00640 1 --2- 172.21.15.90:0/2573527147 >> v2:172.21.15.90:3300/0 conn(0x7f57b00722e0 0x7f57b006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:36.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.851+0000 7f57ae000640 1 -- 172.21.15.90:0/2573527147 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 483446816 0 0) 0x7f57b006e410 con 0x7f57b015fc40 2024-04-25T01:17:36.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.851+0000 7f57af400640 1 --2- 172.21.15.90:0/2573527147 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57b00cf4d0 0x7f57b01600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:36.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.852+0000 7f57ae000640 1 -- 172.21.15.90:0/2573527147 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5794003520 con 0x7f57b015fc40 2024-04-25T01:17:36.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.852+0000 7f57aea00640 1 -- 172.21.15.90:0/2573527147 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57b00cf4d0 msgr2=0x7f57b01600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:36.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.852+0000 7f57aea00640 1 --2- 172.21.15.90:0/2573527147 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57b00cf4d0 0x7f57b01600c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:36.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.852+0000 7f57aea00640 1 -- 172.21.15.90:0/2573527147 >> v1:172.21.15.90:6789/0 conn(0x7f57b015fc40 legacy=0x7f57b00cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:36.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.852+0000 7f57aea00640 1 -- 172.21.15.90:0/2573527147 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57b006d110 con 0x7f57b00722e0 2024-04-25T01:17:36.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.852+0000 7f57af400640 1 --2- 172.21.15.90:0/2573527147 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57b00cf4d0 0x7f57b01600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:36.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.853+0000 7f57aea00640 1 --2- 172.21.15.90:0/2573527147 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57b00722e0 0x7f57b006ded0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f57a00097a0 tx=0x7f57a002a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ca5aaf70e627e45 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:36.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.853+0000 7f57ae000640 1 -- 172.21.15.90:0/2573527147 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a0037070 con 0x7f57b00722e0 2024-04-25T01:17:36.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.853+0000 7f57ae000640 1 -- 172.21.15.90:0/2573527147 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57a002ae50 con 0x7f57b00722e0 2024-04-25T01:17:36.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.853+0000 7f57ae000640 1 -- 172.21.15.90:0/2573527147 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a0032660 con 0x7f57b00722e0 2024-04-25T01:17:36.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.853+0000 7f57b5a00640 1 -- 172.21.15.90:0/2573527147 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57b00722e0 msgr2=0x7f57b006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:36.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.853+0000 7f57b5a00640 1 --2- 172.21.15.90:0/2573527147 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57b00722e0 0x7f57b006ded0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f57a00097a0 tx=0x7f57a002a100 comp rx=0 tx=0).stop 2024-04-25T01:17:36.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.854+0000 7f57b5a00640 1 -- 172.21.15.90:0/2573527147 shutdown_connections 2024-04-25T01:17:36.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.854+0000 7f57b5a00640 1 --2- 172.21.15.90:0/2573527147 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57b00cf4d0 0x7f57b01600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:36.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.854+0000 7f57b5a00640 1 --2- 172.21.15.90:0/2573527147 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57b00722e0 0x7f57b006ded0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:36.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.854+0000 7f57b5a00640 1 -- 172.21.15.90:0/2573527147 >> 172.21.15.90:0/2573527147 conn(0x7f57b006b720 msgr2=0x7f57b00b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:36.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.854+0000 7f57b5a00640 1 -- 172.21.15.90:0/2573527147 shutdown_connections 2024-04-25T01:17:36.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.854+0000 7f57b5a00640 1 -- 172.21.15.90:0/2573527147 wait complete. 2024-04-25T01:17:36.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.855+0000 7f57b5a00640 1 Processor -- start 2024-04-25T01:17:36.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.855+0000 7f57b5a00640 1 -- start start 2024-04-25T01:17:36.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.856+0000 7f57b5a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57b00722e0 0x7f57b0156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:36.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.856+0000 7f57b5a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57b00cf4d0 0x7f57b015ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:36.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.856+0000 7f57b5a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57b015fc40 0x7f57b015f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:36.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.856+0000 7f57b5a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f57b006d6d0 con 0x7f57b015fc40 2024-04-25T01:17:36.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.856+0000 7f57b5a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f57b006cec0 con 0x7f57b00722e0 2024-04-25T01:17:36.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.856+0000 7f57b5a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f57b0070b50 con 0x7f57b00cf4d0 2024-04-25T01:17:36.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.857+0000 7f57af400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57b015fc40 0x7f57b015f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:36.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.857+0000 7f57af400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57b015fc40 0x7f57b015f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57010/0 (socket says 172.21.15.90:57010) 2024-04-25T01:17:36.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.857+0000 7f57af400640 1 -- 172.21.15.90:0/2237156264 learned_addr learned my addr 172.21.15.90:0/2237156264 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:36.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.857+0000 7f57a6000640 1 --2- 172.21.15.90:0/2237156264 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57b00cf4d0 0x7f57b015ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:36.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.857+0000 7f57aea00640 1 --2- 172.21.15.90:0/2237156264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57b00722e0 0x7f57b0156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:36.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.857+0000 7f57af400640 1 -- 172.21.15.90:0/2237156264 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57b00cf4d0 msgr2=0x7f57b015ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:36.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.857+0000 7f57af400640 1 --2- 172.21.15.90:0/2237156264 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57b00cf4d0 0x7f57b015ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:36.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.857+0000 7f57af400640 1 -- 172.21.15.90:0/2237156264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57b00722e0 msgr2=0x7f57b0156d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:36.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.857+0000 7f57af400640 1 --2- 172.21.15.90:0/2237156264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57b00722e0 0x7f57b0156d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:36.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.857+0000 7f57af400640 1 -- 172.21.15.90:0/2237156264 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57b0157410 con 0x7f57b015fc40 2024-04-25T01:17:36.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.857+0000 7f57aea00640 1 --2- 172.21.15.90:0/2237156264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57b00722e0 0x7f57b0156d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:36.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.857+0000 7f57af400640 1 --2- 172.21.15.90:0/2237156264 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57b015fc40 0x7f57b015f320 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f57a800cea0 tx=0x7f57a800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:36.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.858+0000 7f57a7400640 1 -- 172.21.15.90:0/2237156264 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a8003b90 con 0x7f57b015fc40 2024-04-25T01:17:36.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.858+0000 7f57a7400640 1 -- 172.21.15.90:0/2237156264 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57a8003d50 con 0x7f57b015fc40 2024-04-25T01:17:36.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.858+0000 7f57a7400640 1 -- 172.21.15.90:0/2237156264 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a8004260 con 0x7f57b015fc40 2024-04-25T01:17:36.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.858+0000 7f57b5a00640 1 -- 172.21.15.90:0/2237156264 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57b01576f0 con 0x7f57b015fc40 2024-04-25T01:17:36.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.858+0000 7f57b5a00640 1 -- 172.21.15.90:0/2237156264 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57b0157cb0 con 0x7f57b015fc40 2024-04-25T01:17:36.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.859+0000 7f57b5a00640 1 -- 172.21.15.90:0/2237156264 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57b0068fb0 con 0x7f57b015fc40 2024-04-25T01:17:36.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.860+0000 7f57a7400640 1 -- 172.21.15.90:0/2237156264 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f57a8004400 con 0x7f57b015fc40 2024-04-25T01:17:36.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.861+0000 7f57a7400640 1 --2- 172.21.15.90:0/2237156264 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f578c06be30 0x7f578c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:36.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.861+0000 7f57a7400640 1 -- 172.21.15.90:0/2237156264 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f57a808bc80 con 0x7f57b015fc40 2024-04-25T01:17:36.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.861+0000 7f57aea00640 1 --2- 172.21.15.90:0/2237156264 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f578c06be30 0x7f578c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:36.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.862+0000 7f57aea00640 1 --2- 172.21.15.90:0/2237156264 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f578c06be30 0x7f578c06e2f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f57a0034000 tx=0x7f57a002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:36.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:36.865+0000 7f57a7400640 1 -- 172.21.15.90:0/2237156264 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f57a805a220 con 0x7f57b015fc40 2024-04-25T01:17:37.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.060+0000 7f57b5a00640 1 -- 172.21.15.90:0/2237156264 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f57b0072c90 con 0x7f57b015fc40 2024-04-25T01:17:37.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.061+0000 7f57a7400640 1 -- 172.21.15.90:0/2237156264 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) ==== 104+0+11 (secure 0 0 0) 0x7f57a8059b20 con 0x7f57b015fc40 2024-04-25T01:17:37.044 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:17:37.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.063+0000 7f57b5a00640 1 -- 172.21.15.90:0/2237156264 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f578c06be30 msgr2=0x7f578c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:37.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.063+0000 7f57b5a00640 1 --2- 172.21.15.90:0/2237156264 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f578c06be30 0x7f578c06e2f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f57a0034000 tx=0x7f57a002e000 comp rx=0 tx=0).stop 2024-04-25T01:17:37.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.063+0000 7f57b5a00640 1 -- 172.21.15.90:0/2237156264 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57b015fc40 msgr2=0x7f57b015f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:37.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.063+0000 7f57b5a00640 1 --2- 172.21.15.90:0/2237156264 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57b015fc40 0x7f57b015f320 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f57a800cea0 tx=0x7f57a800cf90 comp rx=0 tx=0).stop 2024-04-25T01:17:37.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.063+0000 7f57b5a00640 1 -- 172.21.15.90:0/2237156264 shutdown_connections 2024-04-25T01:17:37.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.063+0000 7f57b5a00640 1 --2- 172.21.15.90:0/2237156264 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f578c06be30 0x7f578c06e2f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:37.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.063+0000 7f57b5a00640 1 --2- 172.21.15.90:0/2237156264 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57b015fc40 0x7f57b015f320 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:37.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.063+0000 7f57b5a00640 1 --2- 172.21.15.90:0/2237156264 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57b00cf4d0 0x7f57b015ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:37.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.063+0000 7f57b5a00640 1 --2- 172.21.15.90:0/2237156264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57b00722e0 0x7f57b0156d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:37.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.063+0000 7f57b5a00640 1 -- 172.21.15.90:0/2237156264 >> 172.21.15.90:0/2237156264 conn(0x7f57b006b720 msgr2=0x7f57b0070f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:37.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.064+0000 7f57b5a00640 1 -- 172.21.15.90:0/2237156264 shutdown_connections 2024-04-25T01:17:37.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.064+0000 7f57b5a00640 1 -- 172.21.15.90:0/2237156264 wait complete. 2024-04-25T01:17:37.066 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:17:37.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.285+0000 7fdb0c000640 1 Processor -- start 2024-04-25T01:17:37.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.285+0000 7fdb0c000640 1 -- start start 2024-04-25T01:17:37.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.287+0000 7fdb0c000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb040b37c0 0x7fdb0415bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:37.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.287+0000 7fdb0c000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb0415c210 con 0x7fdb040b47d0 2024-04-25T01:17:37.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.287+0000 7fdb0c000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb040b2ce0 con 0x7fdb04158a70 2024-04-25T01:17:37.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.287+0000 7fdb0c000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdb040b2fc0 con 0x7fdb040b37c0 2024-04-25T01:17:37.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.287+0000 7fdb08c00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fdb040b47d0 0x7fdb0415b520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:35170/0 (socket says 172.21.15.90:35170) 2024-04-25T01:17:37.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.287+0000 7fdb08c00640 1 -- 172.21.15.90:0/385412773 learned_addr learned my addr 172.21.15.90:0/385412773 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:37.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.288+0000 7fdb0a000640 1 --2- 172.21.15.90:0/385412773 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb040b37c0 0x7fdb0415bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:37.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.288+0000 7fdb03e00640 1 -- 172.21.15.90:0/385412773 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 977881728 0 0) 0x7fdb0415c210 con 0x7fdb040b47d0 2024-04-25T01:17:37.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.288+0000 7fdb03e00640 1 -- 172.21.15.90:0/385412773 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdaec005730 con 0x7fdb040b47d0 2024-04-25T01:17:37.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.288+0000 7fdb03e00640 1 -- 172.21.15.90:0/385412773 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1979970911 0 0) 0x7fdaec005730 con 0x7fdb040b47d0 2024-04-25T01:17:37.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.288+0000 7fdb03e00640 1 -- 172.21.15.90:0/385412773 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb040b37c0 msgr2=0x7fdb0415bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:37.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.288+0000 7fdb03e00640 1 --2- 172.21.15.90:0/385412773 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb040b37c0 0x7fdb0415bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:37.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.288+0000 7fdb03e00640 1 -- 172.21.15.90:0/385412773 >> v1:172.21.15.103:6789/0 conn(0x7fdb04158a70 legacy=0x7fdb040b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:37.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.288+0000 7fdb03e00640 1 -- 172.21.15.90:0/385412773 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb040b32a0 con 0x7fdb040b47d0 2024-04-25T01:17:37.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.288+0000 7fdb03e00640 1 -- 172.21.15.90:0/385412773 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7fdaf0001670 con 0x7fdb040b47d0 2024-04-25T01:17:37.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.288+0000 7fdb03e00640 1 -- 172.21.15.90:0/385412773 >> v1:172.21.15.90:6789/0 conn(0x7fdb040b47d0 legacy=0x7fdb0415b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:37.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.288+0000 7fdb03e00640 1 --2- 172.21.15.90:0/385412773 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb040b37c0 0x7fdaec005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:37.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.288+0000 7fdb03e00640 1 --2- 172.21.15.90:0/385412773 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdaec0061c0 0x7fdaec006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:37.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.288+0000 7fdb03e00640 1 --2- 172.21.15.90:0/385412773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaec006f50 0x7fdaec0073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:37.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.288+0000 7fdb03e00640 1 -- 172.21.15.90:0/385412773 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdb040b2fc0 con 0x7fdaec0061c0 2024-04-25T01:17:37.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.288+0000 7fdb03e00640 1 -- 172.21.15.90:0/385412773 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdaec00dd10 con 0x7fdaec006f50 2024-04-25T01:17:37.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.288+0000 7fdb03e00640 1 -- 172.21.15.90:0/385412773 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdaec00df10 con 0x7fdb040b37c0 2024-04-25T01:17:37.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.289+0000 7fdb08c00640 1 --2- 172.21.15.90:0/385412773 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdaec0061c0 0x7fdaec006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:37.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.289+0000 7fdb09600640 1 --2- 172.21.15.90:0/385412773 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb040b37c0 0x7fdaec005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:37.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.289+0000 7fdb0a000640 1 --2- 172.21.15.90:0/385412773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaec006f50 0x7fdaec0073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:37.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.289+0000 7fdb08c00640 1 -- 172.21.15.90:0/385412773 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb040b37c0 msgr2=0x7fdaec005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:37.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.289+0000 7fdb08c00640 1 --2- 172.21.15.90:0/385412773 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb040b37c0 0x7fdaec005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:37.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.289+0000 7fdb08c00640 1 -- 172.21.15.90:0/385412773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaec006f50 msgr2=0x7fdaec0073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:37.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.289+0000 7fdb08c00640 1 --2- 172.21.15.90:0/385412773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaec006f50 0x7fdaec0073d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:37.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.289+0000 7fdb08c00640 1 -- 172.21.15.90:0/385412773 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaec005730 con 0x7fdaec0061c0 2024-04-25T01:17:37.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.290+0000 7fdb08c00640 1 --2- 172.21.15.90:0/385412773 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdaec0061c0 0x7fdaec006a10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fdaf000c0e0 tx=0x7fdaec007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0e3e0165d8d5eb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:37.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.290+0000 7fdb03e00640 1 -- 172.21.15.90:0/385412773 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf0007c20 con 0x7fdaec0061c0 2024-04-25T01:17:37.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.290+0000 7fdb03e00640 1 -- 172.21.15.90:0/385412773 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaf0005af0 con 0x7fdaec0061c0 2024-04-25T01:17:37.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.290+0000 7fdb03e00640 1 -- 172.21.15.90:0/385412773 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf0007070 con 0x7fdaec0061c0 2024-04-25T01:17:37.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.291+0000 7fdb0c000640 1 -- 172.21.15.90:0/385412773 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdaec0061c0 msgr2=0x7fdaec006a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:37.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.291+0000 7fdb0c000640 1 --2- 172.21.15.90:0/385412773 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdaec0061c0 0x7fdaec006a10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fdaf000c0e0 tx=0x7fdaec007a40 comp rx=0 tx=0).stop 2024-04-25T01:17:37.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.291+0000 7fdb0a000640 1 -- 172.21.15.90:0/385412773 reap_dead start 2024-04-25T01:17:37.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.291+0000 7fdb0c000640 1 -- 172.21.15.90:0/385412773 shutdown_connections 2024-04-25T01:17:37.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.291+0000 7fdb0c000640 1 -- 172.21.15.90:0/385412773 >> 172.21.15.90:0/385412773 conn(0x7fdb0406b720 msgr2=0x7fdb040b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:37.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.291+0000 7fdb0c000640 1 -- 172.21.15.90:0/385412773 shutdown_connections 2024-04-25T01:17:37.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.291+0000 7fdb0c000640 1 -- 172.21.15.90:0/385412773 wait complete. 2024-04-25T01:17:37.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.293+0000 7fdb0c000640 1 Processor -- start 2024-04-25T01:17:37.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.293+0000 7fdb0c000640 1 -- start start 2024-04-25T01:17:37.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.294+0000 7fdb0c000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdaec0061c0 0x7fdb040cc060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:37.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.294+0000 7fdb0c000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb040cb5b0 0x7fdb040c9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:37.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.294+0000 7fdb0c000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb04157830 0x7fdb04072380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:37.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.294+0000 7fdb0c000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdb0415bdb0 con 0x7fdaec0061c0 2024-04-25T01:17:37.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.294+0000 7fdb0c000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdb0415bc30 con 0x7fdb040cb5b0 2024-04-25T01:17:37.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.294+0000 7fdb0c000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdb0415bf30 con 0x7fdb04157830 2024-04-25T01:17:37.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.294+0000 7fdb09600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdaec0061c0 0x7fdb040cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:37.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.294+0000 7fdb09600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdaec0061c0 0x7fdb040cc060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57034/0 (socket says 172.21.15.90:57034) 2024-04-25T01:17:37.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.294+0000 7fdb09600640 1 -- 172.21.15.90:0/400862987 learned_addr learned my addr 172.21.15.90:0/400862987 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:37.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.294+0000 7fdb09600640 1 -- 172.21.15.90:0/400862987 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb04157830 msgr2=0x7fdb04072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:37.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.294+0000 7fdb09600640 1 --2- 172.21.15.90:0/400862987 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb04157830 0x7fdb04072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:37.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.294+0000 7fdb09600640 1 -- 172.21.15.90:0/400862987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb040cb5b0 msgr2=0x7fdb040c9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:37.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.294+0000 7fdb09600640 1 --2- 172.21.15.90:0/400862987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb040cb5b0 0x7fdb040c9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:37.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.294+0000 7fdb09600640 1 -- 172.21.15.90:0/400862987 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb040ca510 con 0x7fdaec0061c0 2024-04-25T01:17:37.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.295+0000 7fdb09600640 1 --2- 172.21.15.90:0/400862987 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdaec0061c0 0x7fdb040cc060 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fdaf800e4f0 tx=0x7fdaf800e9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:37.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.295+0000 7fdb01600640 1 -- 172.21.15.90:0/400862987 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf80052e0 con 0x7fdaec0061c0 2024-04-25T01:17:37.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.295+0000 7fdb01600640 1 -- 172.21.15.90:0/400862987 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaf8018070 con 0x7fdaec0061c0 2024-04-25T01:17:37.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.295+0000 7fdb0c000640 1 -- 172.21.15.90:0/400862987 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb040ca7f0 con 0x7fdaec0061c0 2024-04-25T01:17:37.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.295+0000 7fdb01600640 1 -- 172.21.15.90:0/400862987 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf8003830 con 0x7fdaec0061c0 2024-04-25T01:17:37.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.295+0000 7fdb0c000640 1 -- 172.21.15.90:0/400862987 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb040cadb0 con 0x7fdaec0061c0 2024-04-25T01:17:37.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.296+0000 7fdb0c000640 1 -- 172.21.15.90:0/400862987 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb04068fb0 con 0x7fdaec0061c0 2024-04-25T01:17:37.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.297+0000 7fdb01600640 1 -- 172.21.15.90:0/400862987 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fdaf80039d0 con 0x7fdaec0061c0 2024-04-25T01:17:37.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.298+0000 7fdb01600640 1 --2- 172.21.15.90:0/400862987 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdae406be30 0x7fdae406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:37.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.298+0000 7fdb01600640 1 -- 172.21.15.90:0/400862987 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7fdaf808aea0 con 0x7fdaec0061c0 2024-04-25T01:17:37.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.299+0000 7fdb08c00640 1 --2- 172.21.15.90:0/400862987 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdae406be30 0x7fdae406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:37.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.299+0000 7fdb08c00640 1 --2- 172.21.15.90:0/400862987 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdae406be30 0x7fdae406e2f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdaf0032000 tx=0x7fdaf0007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:37.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.303+0000 7fdb01600640 1 -- 172.21.15.90:0/400862987 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fdaf8059500 con 0x7fdaec0061c0 2024-04-25T01:17:37.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.501+0000 7fdb0c000640 1 -- 172.21.15.90:0/400862987 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdb040b3c70 con 0x7fdaec0061c0 2024-04-25T01:17:37.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.502+0000 7fdb01600640 1 -- 172.21.15.90:0/400862987 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) ==== 100+0+11 (secure 0 0 0) 0x7fdaf8058e00 con 0x7fdaec0061c0 2024-04-25T01:17:37.486 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-25T01:17:37.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.505+0000 7fdb0c000640 1 -- 172.21.15.90:0/400862987 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdae406be30 msgr2=0x7fdae406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:37.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.505+0000 7fdb0c000640 1 --2- 172.21.15.90:0/400862987 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdae406be30 0x7fdae406e2f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdaf0032000 tx=0x7fdaf0007000 comp rx=0 tx=0).stop 2024-04-25T01:17:37.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.505+0000 7fdb0c000640 1 -- 172.21.15.90:0/400862987 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdaec0061c0 msgr2=0x7fdb040cc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:37.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.505+0000 7fdb0c000640 1 --2- 172.21.15.90:0/400862987 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdaec0061c0 0x7fdb040cc060 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fdaf800e4f0 tx=0x7fdaf800e9b0 comp rx=0 tx=0).stop 2024-04-25T01:17:37.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.505+0000 7fdb0c000640 1 -- 172.21.15.90:0/400862987 shutdown_connections 2024-04-25T01:17:37.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.505+0000 7fdb0c000640 1 --2- 172.21.15.90:0/400862987 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdae406be30 0x7fdae406e2f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:37.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.505+0000 7fdb0c000640 1 --2- 172.21.15.90:0/400862987 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdb04157830 0x7fdb04072380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:37.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.505+0000 7fdb0c000640 1 --2- 172.21.15.90:0/400862987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb040cb5b0 0x7fdb040c9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:37.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.505+0000 7fdb0c000640 1 --2- 172.21.15.90:0/400862987 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdaec0061c0 0x7fdb040cc060 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:37.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.505+0000 7fdb0c000640 1 -- 172.21.15.90:0/400862987 >> 172.21.15.90:0/400862987 conn(0x7fdb0406b720 msgr2=0x7fdb040b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:37.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.505+0000 7fdb0c000640 1 -- 172.21.15.90:0/400862987 shutdown_connections 2024-04-25T01:17:37.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.505+0000 7fdb0c000640 1 -- 172.21.15.90:0/400862987 wait complete. 2024-04-25T01:17:37.507 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:17:37.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.723+0000 7f60e1600640 1 Processor -- start 2024-04-25T01:17:37.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.723+0000 7f60e1600640 1 -- start start 2024-04-25T01:17:37.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.724+0000 7f60e1600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f60dc155130 0x7f60dc06db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:37.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.724+0000 7f60e1600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60dc06e0b0 0x7f60dc15b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:37.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.724+0000 7f60e1600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f60dc15b780 0x7f60dc0ae620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:37.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.724+0000 7f60e1600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f60dc15bfe0 con 0x7f60dc155130 2024-04-25T01:17:37.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.724+0000 7f60e1600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f60dc155510 con 0x7f60dc15b780 2024-04-25T01:17:37.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.724+0000 7f60e1600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f60dc071af0 con 0x7f60dc06e0b0 2024-04-25T01:17:37.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.725+0000 7f60dbe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f60dc155130 0x7f60dc06db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:37.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.725+0000 7f60dbe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f60dc155130 0x7f60dc06db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57038/0 (socket says 172.21.15.90:57038) 2024-04-25T01:17:37.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.725+0000 7f60dbe00640 1 -- 172.21.15.90:0/3298305826 learned_addr learned my addr 172.21.15.90:0/3298305826 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:37.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.725+0000 7f60d3400640 1 --2- 172.21.15.90:0/3298305826 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60dc06e0b0 0x7f60dc15b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:37.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.725+0000 7f60e0c00640 1 --2- 172.21.15.90:0/3298305826 >> v2:172.21.15.103:3300/0 conn(0x7f60dc15b780 0x7f60dc0ae620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:37.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.725+0000 7f60dbe00640 1 -- 172.21.15.90:0/3298305826 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60dc06e0b0 msgr2=0x7f60dc15b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:37.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.725+0000 7f60dbe00640 1 --2- 172.21.15.90:0/3298305826 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60dc06e0b0 0x7f60dc15b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:37.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.725+0000 7f60dbe00640 1 -- 172.21.15.90:0/3298305826 >> v2:172.21.15.103:3300/0 conn(0x7f60dc15b780 msgr2=0x7f60dc0ae620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:37.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.725+0000 7f60dbe00640 1 --2- 172.21.15.90:0/3298305826 >> v2:172.21.15.103:3300/0 conn(0x7f60dc15b780 0x7f60dc0ae620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:37.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.725+0000 7f60dbe00640 1 -- 172.21.15.90:0/3298305826 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60dc071dd0 con 0x7f60dc155130 2024-04-25T01:17:37.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.725+0000 7f60d3400640 1 --2- 172.21.15.90:0/3298305826 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60dc06e0b0 0x7f60dc15b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:37.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.725+0000 7f60e0c00640 1 --2- 172.21.15.90:0/3298305826 >> v2:172.21.15.103:3300/0 conn(0x7f60dc15b780 0x7f60dc0ae620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:37.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.726+0000 7f60dbe00640 1 --2- 172.21.15.90:0/3298305826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60dc155130 0x7f60dc06db70 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f60dc158bf0 tx=0x7f60cc02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=718697453051c5a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:37.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.727+0000 7f60db400640 1 -- 172.21.15.90:0/3298305826 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc036070 con 0x7f60dc155130 2024-04-25T01:17:37.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.727+0000 7f60db400640 1 -- 172.21.15.90:0/3298305826 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60cc031600 con 0x7f60dc155130 2024-04-25T01:17:37.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.727+0000 7f60db400640 1 -- 172.21.15.90:0/3298305826 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc0319d0 con 0x7f60dc155130 2024-04-25T01:17:37.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.727+0000 7f60e1600640 1 -- 172.21.15.90:0/3298305826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60dc155130 msgr2=0x7f60dc06db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:37.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.727+0000 7f60e1600640 1 --2- 172.21.15.90:0/3298305826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60dc155130 0x7f60dc06db70 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f60dc158bf0 tx=0x7f60cc02a610 comp rx=0 tx=0).stop 2024-04-25T01:17:37.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.727+0000 7f60e1600640 1 -- 172.21.15.90:0/3298305826 shutdown_connections 2024-04-25T01:17:37.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.727+0000 7f60e1600640 1 --2- 172.21.15.90:0/3298305826 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60dc06e0b0 0x7f60dc15b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:37.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.727+0000 7f60e1600640 1 --2- 172.21.15.90:0/3298305826 >> v2:172.21.15.103:3300/0 conn(0x7f60dc15b780 0x7f60dc0ae620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:37.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.727+0000 7f60e1600640 1 --2- 172.21.15.90:0/3298305826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60dc155130 0x7f60dc06db70 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:37.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.727+0000 7f60e1600640 1 -- 172.21.15.90:0/3298305826 >> 172.21.15.90:0/3298305826 conn(0x7f60dc06b120 msgr2=0x7f60dc06b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:37.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.728+0000 7f60e1600640 1 -- 172.21.15.90:0/3298305826 shutdown_connections 2024-04-25T01:17:37.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.728+0000 7f60e1600640 1 -- 172.21.15.90:0/3298305826 wait complete. 2024-04-25T01:17:37.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.729+0000 7f60e1600640 1 Processor -- start 2024-04-25T01:17:37.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.730+0000 7f60e1600640 1 -- start start 2024-04-25T01:17:37.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.730+0000 7f60e1600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60dc06e0b0 0x7f60dc1f6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:37.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.730+0000 7f60e1600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60dc155130 0x7f60dc0cd560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:37.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.731+0000 7f60e1600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60dc15b780 0x7f60dc0cbf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:37.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.731+0000 7f60e1600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f60dc072a30 con 0x7f60dc15b780 2024-04-25T01:17:37.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.731+0000 7f60e1600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f60dc0728b0 con 0x7f60dc06e0b0 2024-04-25T01:17:37.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.731+0000 7f60e1600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f60dc15bc10 con 0x7f60dc155130 2024-04-25T01:17:37.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.731+0000 7f60e0c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60dc15b780 0x7f60dc0cbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:37.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.731+0000 7f60e0c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60dc15b780 0x7f60dc0cbf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57044/0 (socket says 172.21.15.90:57044) 2024-04-25T01:17:37.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.731+0000 7f60e0c00640 1 -- 172.21.15.90:0/660657517 learned_addr learned my addr 172.21.15.90:0/660657517 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:37.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.731+0000 7f60dbe00640 1 --2- 172.21.15.90:0/660657517 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60dc06e0b0 0x7f60dc1f6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:37.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.731+0000 7f60d3400640 1 --2- 172.21.15.90:0/660657517 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60dc155130 0x7f60dc0cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:37.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.731+0000 7f60e0c00640 1 -- 172.21.15.90:0/660657517 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60dc155130 msgr2=0x7f60dc0cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:37.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.731+0000 7f60e0c00640 1 --2- 172.21.15.90:0/660657517 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60dc155130 0x7f60dc0cd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:37.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.731+0000 7f60e0c00640 1 -- 172.21.15.90:0/660657517 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60dc06e0b0 msgr2=0x7f60dc1f6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:37.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.731+0000 7f60e0c00640 1 --2- 172.21.15.90:0/660657517 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60dc06e0b0 0x7f60dc1f6eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:37.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.731+0000 7f60e0c00640 1 -- 172.21.15.90:0/660657517 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60dc1f75e0 con 0x7f60dc15b780 2024-04-25T01:17:37.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.732+0000 7f60e0c00640 1 --2- 172.21.15.90:0/660657517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60dc15b780 0x7f60dc0cbf20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f60d400c7c0 tx=0x7f60d400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:37.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.732+0000 7f60d8c00640 1 -- 172.21.15.90:0/660657517 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d4015020 con 0x7f60dc15b780 2024-04-25T01:17:37.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.732+0000 7f60d8c00640 1 -- 172.21.15.90:0/660657517 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60d4004230 con 0x7f60dc15b780 2024-04-25T01:17:37.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.732+0000 7f60d8c00640 1 -- 172.21.15.90:0/660657517 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d4007620 con 0x7f60dc15b780 2024-04-25T01:17:37.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.732+0000 7f60e1600640 1 -- 172.21.15.90:0/660657517 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60dc1f78c0 con 0x7f60dc15b780 2024-04-25T01:17:37.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.732+0000 7f60e1600640 1 -- 172.21.15.90:0/660657517 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60dc0cc460 con 0x7f60dc15b780 2024-04-25T01:17:37.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.733+0000 7f60e1600640 1 -- 172.21.15.90:0/660657517 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60dc074c90 con 0x7f60dc15b780 2024-04-25T01:17:37.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.734+0000 7f60d8c00640 1 -- 172.21.15.90:0/660657517 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f60d4018020 con 0x7f60dc15b780 2024-04-25T01:17:37.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.735+0000 7f60d8c00640 1 --2- 172.21.15.90:0/660657517 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f60b806be60 0x7f60b806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:37.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.735+0000 7f60d8c00640 1 -- 172.21.15.90:0/660657517 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f60d408b290 con 0x7f60dc15b780 2024-04-25T01:17:37.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.735+0000 7f60dbe00640 1 --2- 172.21.15.90:0/660657517 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f60b806be60 0x7f60b806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:37.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.736+0000 7f60dbe00640 1 --2- 172.21.15.90:0/660657517 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f60b806be60 0x7f60b806e320 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f60cc033000 tx=0x7f60cc009000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:37.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.740+0000 7f60d8c00640 1 -- 172.21.15.90:0/660657517 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f60dc074c90 con 0x7f60dc15b780 2024-04-25T01:17:37.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.938+0000 7f60e1600640 1 -- 172.21.15.90:0/660657517 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f60dc1f38c0 con 0x7f60dc15b780 2024-04-25T01:17:37.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.938+0000 7f60d8c00640 1 -- 172.21.15.90:0/660657517 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) ==== 103+0+11 (secure 0 0 0) 0x7f60d401f020 con 0x7f60dc15b780 2024-04-25T01:17:37.922 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:17:37.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.941+0000 7f60e1600640 1 -- 172.21.15.90:0/660657517 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f60b806be60 msgr2=0x7f60b806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:37.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.941+0000 7f60e1600640 1 --2- 172.21.15.90:0/660657517 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f60b806be60 0x7f60b806e320 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f60cc033000 tx=0x7f60cc009000 comp rx=0 tx=0).stop 2024-04-25T01:17:37.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.941+0000 7f60e1600640 1 -- 172.21.15.90:0/660657517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60dc15b780 msgr2=0x7f60dc0cbf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:37.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.941+0000 7f60e1600640 1 --2- 172.21.15.90:0/660657517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60dc15b780 0x7f60dc0cbf20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f60d400c7c0 tx=0x7f60d400cc80 comp rx=0 tx=0).stop 2024-04-25T01:17:37.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.941+0000 7f60e1600640 1 -- 172.21.15.90:0/660657517 shutdown_connections 2024-04-25T01:17:37.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.941+0000 7f60e1600640 1 --2- 172.21.15.90:0/660657517 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f60b806be60 0x7f60b806e320 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:37.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.941+0000 7f60e1600640 1 --2- 172.21.15.90:0/660657517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60dc15b780 0x7f60dc0cbf20 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:37.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.941+0000 7f60e1600640 1 --2- 172.21.15.90:0/660657517 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60dc155130 0x7f60dc0cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:37.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.941+0000 7f60e1600640 1 --2- 172.21.15.90:0/660657517 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60dc06e0b0 0x7f60dc1f6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:37.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.941+0000 7f60e1600640 1 -- 172.21.15.90:0/660657517 >> 172.21.15.90:0/660657517 conn(0x7f60dc06b120 msgr2=0x7f60dc154400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:37.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.941+0000 7f60e1600640 1 -- 172.21.15.90:0/660657517 shutdown_connections 2024-04-25T01:17:37.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:37.941+0000 7f60e1600640 1 -- 172.21.15.90:0/660657517 wait complete. 2024-04-25T01:17:37.944 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T01:17:38.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.165+0000 7f2fe5a00640 1 Processor -- start 2024-04-25T01:17:38.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.165+0000 7f2fe5a00640 1 -- start start 2024-04-25T01:17:38.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.167+0000 7f2fe5a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2fe00b41f0 0x7f2fe006b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:38.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.167+0000 7f2fe5a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2fe015ee00 0x7f2fe006b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:38.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.167+0000 7f2fe5a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fe00adc10 con 0x7f2fe00ae9d0 2024-04-25T01:17:38.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.167+0000 7f2fe5a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f2fe00adef0 con 0x7f2fe015ee00 2024-04-25T01:17:38.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.167+0000 7f2fe5a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2fe00ae1d0 con 0x7f2fe00b41f0 2024-04-25T01:17:38.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.167+0000 7f2fe5000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2fe015ee00 0x7f2fe006b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:38.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.167+0000 7f2fe5000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2fe015ee00 0x7f2fe006b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57060/0 (socket says 172.21.15.90:57060) 2024-04-25T01:17:38.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.167+0000 7f2fe5000640 1 -- 172.21.15.90:0/2137414647 learned_addr learned my addr 172.21.15.90:0/2137414647 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:38.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.168+0000 7f2fdfe00640 1 --2- 172.21.15.90:0/2137414647 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2fe00b41f0 0x7f2fe006b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:38.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.168+0000 7f2fe5000640 1 -- 172.21.15.90:0/2137414647 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2fe00b41f0 msgr2=0x7f2fe006b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:38.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.168+0000 7f2fe5000640 1 --2- 172.21.15.90:0/2137414647 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2fe00b41f0 0x7f2fe006b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:38.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.168+0000 7f2fe5000640 1 -- 172.21.15.90:0/2137414647 >> v1:172.21.15.103:6789/0 conn(0x7f2fe00ae9d0 legacy=0x7f2fe015e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:38.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.168+0000 7f2fe5000640 1 -- 172.21.15.90:0/2137414647 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fe00ae4b0 con 0x7f2fe015ee00 2024-04-25T01:17:38.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.169+0000 7f2fe5000640 1 --2- 172.21.15.90:0/2137414647 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fe015ee00 0x7f2fe006b660 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8009870 tx=0x7f2fd802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11b1094db4efacb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:38.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.169+0000 7f2fdea00640 1 -- 172.21.15.90:0/2137414647 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd8037070 con 0x7f2fe015ee00 2024-04-25T01:17:38.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.169+0000 7f2fdea00640 1 -- 172.21.15.90:0/2137414647 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fd8032440 con 0x7f2fe015ee00 2024-04-25T01:17:38.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.169+0000 7f2fdea00640 1 -- 172.21.15.90:0/2137414647 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd8032810 con 0x7f2fe015ee00 2024-04-25T01:17:38.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.170+0000 7f2fe5a00640 1 -- 172.21.15.90:0/2137414647 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fe015ee00 msgr2=0x7f2fe006b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:38.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.170+0000 7f2fe5a00640 1 --2- 172.21.15.90:0/2137414647 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fe015ee00 0x7f2fe006b660 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8009870 tx=0x7f2fd802a210 comp rx=0 tx=0).stop 2024-04-25T01:17:38.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.170+0000 7f2fe5a00640 1 -- 172.21.15.90:0/2137414647 shutdown_connections 2024-04-25T01:17:38.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.170+0000 7f2fe5a00640 1 --2- 172.21.15.90:0/2137414647 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fe015ee00 0x7f2fe006b660 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:38.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.170+0000 7f2fe5a00640 1 --2- 172.21.15.90:0/2137414647 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2fe00b41f0 0x7f2fe006b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:38.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.170+0000 7f2fe5a00640 1 -- 172.21.15.90:0/2137414647 >> 172.21.15.90:0/2137414647 conn(0x7f2fe00b1660 msgr2=0x7f2fe00b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:38.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.170+0000 7f2fe5a00640 1 -- 172.21.15.90:0/2137414647 shutdown_connections 2024-04-25T01:17:38.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.170+0000 7f2fe5a00640 1 -- 172.21.15.90:0/2137414647 wait complete. 2024-04-25T01:17:38.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.172+0000 7f2fe5a00640 1 Processor -- start 2024-04-25T01:17:38.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.172+0000 7f2fe5a00640 1 -- start start 2024-04-25T01:17:38.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.173+0000 7f2fe5a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fe00ae9d0 0x7f2fe0151b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:38.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.173+0000 7f2fe5a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fe00b41f0 0x7f2fe0070a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:38.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.173+0000 7f2fe5a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2fe015ee00 0x7f2fe0070f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:38.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.173+0000 7f2fe5a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe0157fc0 con 0x7f2fe00b41f0 2024-04-25T01:17:38.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.173+0000 7f2fe5a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe0157e40 con 0x7f2fe00ae9d0 2024-04-25T01:17:38.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.173+0000 7f2fe5a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2fe0158140 con 0x7f2fe015ee00 2024-04-25T01:17:38.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.173+0000 7f2fdf400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fe00b41f0 0x7f2fe0070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:38.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.173+0000 7f2fdf400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fe00b41f0 0x7f2fe0070a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57072/0 (socket says 172.21.15.90:57072) 2024-04-25T01:17:38.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.173+0000 7f2fdf400640 1 -- 172.21.15.90:0/1024749741 learned_addr learned my addr 172.21.15.90:0/1024749741 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:38.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.173+0000 7f2fdf400640 1 -- 172.21.15.90:0/1024749741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2fe015ee00 msgr2=0x7f2fe0070f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:38.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.173+0000 7f2fdf400640 1 --2- 172.21.15.90:0/1024749741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2fe015ee00 0x7f2fe0070f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:38.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.173+0000 7f2fdfe00640 1 --2- 172.21.15.90:0/1024749741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fe00ae9d0 0x7f2fe0151b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:38.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.173+0000 7f2fdf400640 1 -- 172.21.15.90:0/1024749741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fe00ae9d0 msgr2=0x7f2fe0151b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:38.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.173+0000 7f2fdf400640 1 --2- 172.21.15.90:0/1024749741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fe00ae9d0 0x7f2fe0151b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:38.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.173+0000 7f2fdf400640 1 -- 172.21.15.90:0/1024749741 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fe0153670 con 0x7f2fe00b41f0 2024-04-25T01:17:38.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.174+0000 7f2fdf400640 1 --2- 172.21.15.90:0/1024749741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fe00b41f0 0x7f2fe0070a50 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2fd000c7c0 tx=0x7f2fd000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:38.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.174+0000 7f2fd7e00640 1 -- 172.21.15.90:0/1024749741 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd0016020 con 0x7f2fe00b41f0 2024-04-25T01:17:38.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.174+0000 7f2fd7e00640 1 -- 172.21.15.90:0/1024749741 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fd000aa50 con 0x7f2fe00b41f0 2024-04-25T01:17:38.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.174+0000 7f2fe5a00640 1 -- 172.21.15.90:0/1024749741 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fe01e66d0 con 0x7f2fe00b41f0 2024-04-25T01:17:38.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.174+0000 7f2fd7e00640 1 -- 172.21.15.90:0/1024749741 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd000ae40 con 0x7f2fe00b41f0 2024-04-25T01:17:38.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.174+0000 7f2fe5a00640 1 -- 172.21.15.90:0/1024749741 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fe0152040 con 0x7f2fe00b41f0 2024-04-25T01:17:38.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.175+0000 7f2fe5a00640 1 -- 172.21.15.90:0/1024749741 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fe0075350 con 0x7f2fe00b41f0 2024-04-25T01:17:38.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.176+0000 7f2fd7e00640 1 -- 172.21.15.90:0/1024749741 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2fd0019020 con 0x7f2fe00b41f0 2024-04-25T01:17:38.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.177+0000 7f2fd7e00640 1 --2- 172.21.15.90:0/1024749741 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2fbc06be30 0x7f2fbc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:38.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.177+0000 7f2fd7e00640 1 -- 172.21.15.90:0/1024749741 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f2fd008aed0 con 0x7f2fe00b41f0 2024-04-25T01:17:38.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.177+0000 7f2fdfe00640 1 --2- 172.21.15.90:0/1024749741 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2fbc06be30 0x7f2fbc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:38.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.178+0000 7f2fdfe00640 1 --2- 172.21.15.90:0/1024749741 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2fbc06be30 0x7f2fbc06e2f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8006fd0 tx=0x7f2fc8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:38.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.181+0000 7f2fd7e00640 1 -- 172.21.15.90:0/1024749741 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2fd0020070 con 0x7f2fe00b41f0 2024-04-25T01:17:38.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.390+0000 7f2fe5a00640 1 -- 172.21.15.90:0/1024749741 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2fe0152a80 con 0x7f2fe00b41f0 2024-04-25T01:17:38.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.391+0000 7f2fd7e00640 1 -- 172.21.15.90:0/1024749741 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v28) ==== 80+0+0 (secure 0 0 0) 0x7f2fd0059510 con 0x7f2fe00b41f0 2024-04-25T01:17:38.375 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-04-25T01:17:38.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.394+0000 7f2fe5a00640 1 -- 172.21.15.90:0/1024749741 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2fbc06be30 msgr2=0x7f2fbc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:38.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.394+0000 7f2fe5a00640 1 --2- 172.21.15.90:0/1024749741 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2fbc06be30 0x7f2fbc06e2f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8006fd0 tx=0x7f2fc8008040 comp rx=0 tx=0).stop 2024-04-25T01:17:38.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.394+0000 7f2fe5a00640 1 -- 172.21.15.90:0/1024749741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fe00b41f0 msgr2=0x7f2fe0070a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:38.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.394+0000 7f2fe5a00640 1 --2- 172.21.15.90:0/1024749741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fe00b41f0 0x7f2fe0070a50 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2fd000c7c0 tx=0x7f2fd000cc80 comp rx=0 tx=0).stop 2024-04-25T01:17:38.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.394+0000 7f2fe5a00640 1 -- 172.21.15.90:0/1024749741 shutdown_connections 2024-04-25T01:17:38.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.394+0000 7f2fe5a00640 1 --2- 172.21.15.90:0/1024749741 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2fbc06be30 0x7f2fbc06e2f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:38.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.394+0000 7f2fe5a00640 1 --2- 172.21.15.90:0/1024749741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2fe015ee00 0x7f2fe0070f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:38.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.394+0000 7f2fe5a00640 1 --2- 172.21.15.90:0/1024749741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fe00b41f0 0x7f2fe0070a50 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:38.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.394+0000 7f2fe5a00640 1 --2- 172.21.15.90:0/1024749741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fe00ae9d0 0x7f2fe0151b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:38.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.394+0000 7f2fe5a00640 1 -- 172.21.15.90:0/1024749741 >> 172.21.15.90:0/1024749741 conn(0x7f2fe00b1660 msgr2=0x7f2fe00af730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:38.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.394+0000 7f2fe5a00640 1 -- 172.21.15.90:0/1024749741 shutdown_connections 2024-04-25T01:17:38.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.394+0000 7f2fe5a00640 1 -- 172.21.15.90:0/1024749741 wait complete. 2024-04-25T01:17:38.397 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-04-25T01:17:38.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:38.397 DEBUG:teuthology.orchestra.run.smithi103:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-04-25T01:17:38.487 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi103.front.sepia.ceph.com 2024-04-25T01:17:38.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:38.487 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-25T01:17:38.547 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:17:38.547 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:17:38.606 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:17:38.607 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:17:38.664 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:17:38.664 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:17:38.712 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:17:38.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.973+0000 7efc14800640 1 Processor -- start 2024-04-25T01:17:38.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.973+0000 7efc14800640 1 -- start start 2024-04-25T01:17:38.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.975+0000 7efc14800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc0c143940 0x7efc0c06e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:38.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.975+0000 7efc14800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7efc0c15f400 0x7efc0c06eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:38.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.975+0000 7efc14800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc0c070e60 con 0x7efc0c143d20 2024-04-25T01:17:38.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.975+0000 7efc14800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7efc0c071140 con 0x7efc0c15f400 2024-04-25T01:17:38.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.975+0000 7efc14800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efc0c071420 con 0x7efc0c143940 2024-04-25T01:17:38.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.975+0000 7efc12800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7efc0c15f400 0x7efc0c06eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:38.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.975+0000 7efc12800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7efc0c15f400 0x7efc0c06eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57076/0 (socket says 172.21.15.90:57076) 2024-04-25T01:17:38.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.975+0000 7efc12800640 1 -- 172.21.15.90:0/1917754798 learned_addr learned my addr 172.21.15.90:0/1917754798 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:38.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.976+0000 7efc12800640 1 -- 172.21.15.90:0/1917754798 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc0c143940 msgr2=0x7efc0c06e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:38.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.976+0000 7efc12800640 1 --2- 172.21.15.90:0/1917754798 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc0c143940 0x7efc0c06e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:38.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.976+0000 7efc12800640 1 -- 172.21.15.90:0/1917754798 >> v1:172.21.15.103:6789/0 conn(0x7efc0c143d20 legacy=0x7efc0c15ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:38.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.976+0000 7efc12800640 1 -- 172.21.15.90:0/1917754798 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc0c071700 con 0x7efc0c15f400 2024-04-25T01:17:38.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.977+0000 7efc12800640 1 --2- 172.21.15.90:0/1917754798 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc0c15f400 0x7efc0c06eb90 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7efc08009e50 tx=0x7efc0802a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aca1dc6abaad440 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:38.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.977+0000 7efc10a00640 1 -- 172.21.15.90:0/1917754798 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc08036070 con 0x7efc0c15f400 2024-04-25T01:17:38.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.977+0000 7efc10a00640 1 -- 172.21.15.90:0/1917754798 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc08031600 con 0x7efc0c15f400 2024-04-25T01:17:38.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.977+0000 7efc10a00640 1 -- 172.21.15.90:0/1917754798 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc080319d0 con 0x7efc0c15f400 2024-04-25T01:17:38.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.978+0000 7efc14800640 1 -- 172.21.15.90:0/1917754798 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc0c15f400 msgr2=0x7efc0c06eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:38.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.978+0000 7efc14800640 1 --2- 172.21.15.90:0/1917754798 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc0c15f400 0x7efc0c06eb90 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7efc08009e50 tx=0x7efc0802a6c0 comp rx=0 tx=0).stop 2024-04-25T01:17:38.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.978+0000 7efc14800640 1 -- 172.21.15.90:0/1917754798 shutdown_connections 2024-04-25T01:17:38.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.978+0000 7efc14800640 1 --2- 172.21.15.90:0/1917754798 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc0c15f400 0x7efc0c06eb90 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:38.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.978+0000 7efc14800640 1 --2- 172.21.15.90:0/1917754798 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc0c143940 0x7efc0c06e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:38.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.978+0000 7efc14800640 1 -- 172.21.15.90:0/1917754798 >> 172.21.15.90:0/1917754798 conn(0x7efc0c0b1820 msgr2=0x7efc0c0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:38.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.978+0000 7efc14800640 1 -- 172.21.15.90:0/1917754798 shutdown_connections 2024-04-25T01:17:38.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.978+0000 7efc14800640 1 -- 172.21.15.90:0/1917754798 wait complete. 2024-04-25T01:17:38.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.980+0000 7efc14800640 1 Processor -- start 2024-04-25T01:17:38.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.980+0000 7efc14800640 1 -- start start 2024-04-25T01:17:38.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.981+0000 7efc14800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc0c143940 0x7efc0c13f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:38.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.981+0000 7efc14800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc0c143d20 0x7efc0c141f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:38.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.981+0000 7efc14800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc0c15f400 0x7efc0c1424c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:38.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.981+0000 7efc14800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7efc0c071fb0 con 0x7efc0c143d20 2024-04-25T01:17:38.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.981+0000 7efc14800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efc0c071e30 con 0x7efc0c143940 2024-04-25T01:17:38.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.981+0000 7efc14800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7efc0c072130 con 0x7efc0c15f400 2024-04-25T01:17:38.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.981+0000 7efc11400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc0c143d20 0x7efc0c141f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:38.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.981+0000 7efc11400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc0c143d20 0x7efc0c141f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57084/0 (socket says 172.21.15.90:57084) 2024-04-25T01:17:38.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.981+0000 7efc11400640 1 -- 172.21.15.90:0/3650251969 learned_addr learned my addr 172.21.15.90:0/3650251969 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:38.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.981+0000 7efc11400640 1 -- 172.21.15.90:0/3650251969 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc0c15f400 msgr2=0x7efc0c1424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:38.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.981+0000 7efc11400640 1 --2- 172.21.15.90:0/3650251969 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc0c15f400 0x7efc0c1424c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:38.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.981+0000 7efc11400640 1 -- 172.21.15.90:0/3650251969 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc0c143940 msgr2=0x7efc0c13f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:38.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.981+0000 7efc11400640 1 --2- 172.21.15.90:0/3650251969 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc0c143940 0x7efc0c13f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:38.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.981+0000 7efc11400640 1 -- 172.21.15.90:0/3650251969 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc0c0b0380 con 0x7efc0c143d20 2024-04-25T01:17:38.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.982+0000 7efc11400640 1 --2- 172.21.15.90:0/3650251969 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc0c143d20 0x7efc0c141f80 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7efbfc00c460 tx=0x7efbfc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:38.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.982+0000 7efc06000640 1 -- 172.21.15.90:0/3650251969 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbfc016020 con 0x7efc0c143d20 2024-04-25T01:17:38.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.982+0000 7efc06000640 1 -- 172.21.15.90:0/3650251969 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efbfc012870 con 0x7efc0c143d20 2024-04-25T01:17:38.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.982+0000 7efc06000640 1 -- 172.21.15.90:0/3650251969 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbfc012c60 con 0x7efc0c143d20 2024-04-25T01:17:38.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.982+0000 7efc14800640 1 -- 172.21.15.90:0/3650251969 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc0c1e48d0 con 0x7efc0c143d20 2024-04-25T01:17:38.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.982+0000 7efc14800640 1 -- 172.21.15.90:0/3650251969 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc0c1e4e90 con 0x7efc0c143d20 2024-04-25T01:17:38.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.983+0000 7efc14800640 1 -- 172.21.15.90:0/3650251969 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc0c1e5090 con 0x7efc0c143d20 2024-04-25T01:17:38.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.984+0000 7efc06000640 1 -- 172.21.15.90:0/3650251969 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7efbfc019020 con 0x7efc0c143d20 2024-04-25T01:17:38.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.985+0000 7efc06000640 1 --2- 172.21.15.90:0/3650251969 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7efbf006be30 0x7efbf006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:38.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.985+0000 7efc06000640 1 -- 172.21.15.90:0/3650251969 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7efbfc08aee0 con 0x7efc0c143d20 2024-04-25T01:17:38.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.985+0000 7efc11e00640 1 --2- 172.21.15.90:0/3650251969 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7efbf006be30 0x7efbf006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:38.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.986+0000 7efc11e00640 1 --2- 172.21.15.90:0/3650251969 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7efbf006be30 0x7efbf006e2f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7efc00009710 tx=0x7efc00006c90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:38.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:38.989+0000 7efc06000640 1 -- 172.21.15.90:0/3650251969 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7efc0c1e5090 con 0x7efc0c143d20 2024-04-25T01:17:39.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.188+0000 7efc14800640 1 -- 172.21.15.90:0/3650251969 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efc0c068fb0 con 0x7efc0c143d20 2024-04-25T01:17:39.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.190+0000 7efc06000640 1 -- 172.21.15.90:0/3650251969 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18576 (secure 0 0 0) 0x7efbfc020030 con 0x7efc0c143d20 2024-04-25T01:17:39.173 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:17:39.174 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:17:09.346008+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:16:20.094512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:16:21.078268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:16:24.670642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.e","mappings":[{"from":0,"to":7}]},{"pgid":"4.f","mappings":[{"from":7,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:17:39.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.192+0000 7efc14800640 1 -- 172.21.15.90:0/3650251969 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7efbf006be30 msgr2=0x7efbf006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:39.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.192+0000 7efc14800640 1 --2- 172.21.15.90:0/3650251969 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7efbf006be30 0x7efbf006e2f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7efc00009710 tx=0x7efc00006c90 comp rx=0 tx=0).stop 2024-04-25T01:17:39.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.192+0000 7efc14800640 1 -- 172.21.15.90:0/3650251969 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc0c143d20 msgr2=0x7efc0c141f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:39.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.192+0000 7efc14800640 1 --2- 172.21.15.90:0/3650251969 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc0c143d20 0x7efc0c141f80 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7efbfc00c460 tx=0x7efbfc00c920 comp rx=0 tx=0).stop 2024-04-25T01:17:39.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.192+0000 7efc14800640 1 -- 172.21.15.90:0/3650251969 shutdown_connections 2024-04-25T01:17:39.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.192+0000 7efc14800640 1 --2- 172.21.15.90:0/3650251969 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7efbf006be30 0x7efbf006e2f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:39.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.192+0000 7efc14800640 1 --2- 172.21.15.90:0/3650251969 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7efc0c15f400 0x7efc0c1424c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:39.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.192+0000 7efc14800640 1 --2- 172.21.15.90:0/3650251969 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc0c143d20 0x7efc0c141f80 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:39.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.192+0000 7efc14800640 1 --2- 172.21.15.90:0/3650251969 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc0c143940 0x7efc0c13f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:39.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.192+0000 7efc14800640 1 -- 172.21.15.90:0/3650251969 >> 172.21.15.90:0/3650251969 conn(0x7efc0c0b1820 msgr2=0x7efc0c0b3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:39.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.192+0000 7efc14800640 1 -- 172.21.15.90:0/3650251969 shutdown_connections 2024-04-25T01:17:39.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.192+0000 7efc14800640 1 -- 172.21.15.90:0/3650251969 wait complete. 2024-04-25T01:17:39.195 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:16:15.894126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:16:20.094512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:16:21.078268+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:16:24.670642+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:17:39.196 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:17:39.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.421+0000 7fc00c800640 1 Processor -- start 2024-04-25T01:17:39.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.421+0000 7fc00c800640 1 -- start start 2024-04-25T01:17:39.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.423+0000 7fc00c800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc0040cf4d0 0x7fc0041600a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:39.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.423+0000 7fc00c800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc00406d110 con 0x7fc0040722e0 2024-04-25T01:17:39.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.423+0000 7fc00c800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc00406d3f0 con 0x7fc00415fc40 2024-04-25T01:17:39.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.423+0000 7fc00c800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fc004160aa0 con 0x7fc0040cf4d0 2024-04-25T01:17:39.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.423+0000 7fc009e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc0040722e0 0x7fc00406ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:35180/0 (socket says 172.21.15.90:35180) 2024-04-25T01:17:39.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.423+0000 7fc009e00640 1 -- 172.21.15.90:0/3140856193 learned_addr learned my addr 172.21.15.90:0/3140856193 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:39.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.423+0000 7fc00a800640 1 --2- 172.21.15.90:0/3140856193 >> v2:172.21.15.90:3300/0 conn(0x7fc0040cf4d0 0x7fc0041600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:39.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.424+0000 7fc008a00640 1 -- 172.21.15.90:0/3140856193 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2707212064 0 0) 0x7fc00406d110 con 0x7fc0040722e0 2024-04-25T01:17:39.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.424+0000 7fc008a00640 1 -- 172.21.15.90:0/3140856193 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfec003520 con 0x7fc0040722e0 2024-04-25T01:17:39.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.424+0000 7fc00a800640 1 -- 172.21.15.90:0/3140856193 >> v1:172.21.15.103:6789/0 conn(0x7fc00415fc40 legacy=0x7fc0040cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:39.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.424+0000 7fc00a800640 1 -- 172.21.15.90:0/3140856193 >> v1:172.21.15.90:6789/0 conn(0x7fc0040722e0 legacy=0x7fc00406ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:39.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.424+0000 7fc00a800640 1 -- 172.21.15.90:0/3140856193 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc00406d6d0 con 0x7fc0040cf4d0 2024-04-25T01:17:39.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.425+0000 7fc00a800640 1 --2- 172.21.15.90:0/3140856193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0040cf4d0 0x7fc0041600a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc000009d90 tx=0x7fc00002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37fb00999fdd7597 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:39.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.426+0000 7fc008a00640 1 -- 172.21.15.90:0/3140856193 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc000007740 con 0x7fc0040cf4d0 2024-04-25T01:17:39.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.426+0000 7fc008a00640 1 -- 172.21.15.90:0/3140856193 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0000078e0 con 0x7fc0040cf4d0 2024-04-25T01:17:39.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.426+0000 7fc008a00640 1 -- 172.21.15.90:0/3140856193 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc000007cb0 con 0x7fc0040cf4d0 2024-04-25T01:17:39.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.426+0000 7fc00c800640 1 -- 172.21.15.90:0/3140856193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0040cf4d0 msgr2=0x7fc0041600a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:39.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.426+0000 7fc00c800640 1 --2- 172.21.15.90:0/3140856193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0040cf4d0 0x7fc0041600a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc000009d90 tx=0x7fc00002a660 comp rx=0 tx=0).stop 2024-04-25T01:17:39.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.426+0000 7fc00c800640 1 -- 172.21.15.90:0/3140856193 shutdown_connections 2024-04-25T01:17:39.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.426+0000 7fc00c800640 1 --2- 172.21.15.90:0/3140856193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0040cf4d0 0x7fc0041600a0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:39.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.426+0000 7fc00c800640 1 -- 172.21.15.90:0/3140856193 >> 172.21.15.90:0/3140856193 conn(0x7fc00406b720 msgr2=0x7fc0040b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:39.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.427+0000 7fc00c800640 1 -- 172.21.15.90:0/3140856193 shutdown_connections 2024-04-25T01:17:39.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.427+0000 7fc00c800640 1 -- 172.21.15.90:0/3140856193 wait complete. 2024-04-25T01:17:39.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.429+0000 7fc00c800640 1 Processor -- start 2024-04-25T01:17:39.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.429+0000 7fc00c800640 1 -- start start 2024-04-25T01:17:39.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.430+0000 7fc00c800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0040722e0 0x7fc0041593c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:39.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.430+0000 7fc00c800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0040cf4d0 0x7fc00415d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:39.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.430+0000 7fc00c800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc00415fc40 0x7fc00415dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:39.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.430+0000 7fc00c800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc004070f60 con 0x7fc0040cf4d0 2024-04-25T01:17:39.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.430+0000 7fc00c800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc004070de0 con 0x7fc0040722e0 2024-04-25T01:17:39.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.430+0000 7fc00c800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc0040710e0 con 0x7fc00415fc40 2024-04-25T01:17:39.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.430+0000 7fc009400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0040cf4d0 0x7fc00415d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:39.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.430+0000 7fc009400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0040cf4d0 0x7fc00415d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57108/0 (socket says 172.21.15.90:57108) 2024-04-25T01:17:39.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.430+0000 7fc009400640 1 -- 172.21.15.90:0/2890406988 learned_addr learned my addr 172.21.15.90:0/2890406988 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:39.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.431+0000 7fc009e00640 1 --2- 172.21.15.90:0/2890406988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0040722e0 0x7fc0041593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:39.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.431+0000 7fc00a800640 1 --2- 172.21.15.90:0/2890406988 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc00415fc40 0x7fc00415dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:39.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.431+0000 7fc009400640 1 -- 172.21.15.90:0/2890406988 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc00415fc40 msgr2=0x7fc00415dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:39.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.431+0000 7fc009400640 1 --2- 172.21.15.90:0/2890406988 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc00415fc40 0x7fc00415dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:39.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.431+0000 7fc009400640 1 -- 172.21.15.90:0/2890406988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0040722e0 msgr2=0x7fc0041593c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:39.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.431+0000 7fc009400640 1 --2- 172.21.15.90:0/2890406988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0040722e0 0x7fc0041593c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:39.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.431+0000 7fc009400640 1 -- 172.21.15.90:0/2890406988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0041f4130 con 0x7fc0040cf4d0 2024-04-25T01:17:39.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.431+0000 7fc00a800640 1 --2- 172.21.15.90:0/2890406988 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc00415fc40 0x7fc00415dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:39.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.431+0000 7fc009400640 1 --2- 172.21.15.90:0/2890406988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0040cf4d0 0x7fc00415d7f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fbff400c400 tx=0x7fbff400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:39.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.431+0000 7fbffe000640 1 -- 172.21.15.90:0/2890406988 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff4017070 con 0x7fc0040cf4d0 2024-04-25T01:17:39.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.431+0000 7fbffe000640 1 -- 172.21.15.90:0/2890406988 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbff40052d0 con 0x7fc0040cf4d0 2024-04-25T01:17:39.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.432+0000 7fbffe000640 1 -- 172.21.15.90:0/2890406988 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff4003830 con 0x7fc0040cf4d0 2024-04-25T01:17:39.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.432+0000 7fc00c800640 1 -- 172.21.15.90:0/2890406988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0041f4410 con 0x7fc0040cf4d0 2024-04-25T01:17:39.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.432+0000 7fc00c800640 1 -- 172.21.15.90:0/2890406988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0041f49d0 con 0x7fc0040cf4d0 2024-04-25T01:17:39.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.433+0000 7fc00c800640 1 -- 172.21.15.90:0/2890406988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0041f4db0 con 0x7fc0040cf4d0 2024-04-25T01:17:39.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.434+0000 7fbffe000640 1 -- 172.21.15.90:0/2890406988 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbff401a070 con 0x7fc0040cf4d0 2024-04-25T01:17:39.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.435+0000 7fbffe000640 1 --2- 172.21.15.90:0/2890406988 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbfe806be30 0x7fbfe806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:39.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.435+0000 7fbffe000640 1 -- 172.21.15.90:0/2890406988 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7fbff408a4a0 con 0x7fc0040cf4d0 2024-04-25T01:17:39.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.435+0000 7fc009e00640 1 --2- 172.21.15.90:0/2890406988 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbfe806be30 0x7fbfe806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:39.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.436+0000 7fc009e00640 1 --2- 172.21.15.90:0/2890406988 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbfe806be30 0x7fbfe806e2f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbff80011e0 tx=0x7fbff8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:39.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.440+0000 7fbffe000640 1 -- 172.21.15.90:0/2890406988 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc0041f4db0 con 0x7fc0040cf4d0 2024-04-25T01:17:39.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.640+0000 7fc00c800640 1 -- 172.21.15.90:0/2890406988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc004068fb0 con 0x7fc0040cf4d0 2024-04-25T01:17:39.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.640+0000 7fbffe000640 1 -- 172.21.15.90:0/2890406988 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) ==== 93+0+10 (secure 0 0 0) 0x7fbff4058b70 con 0x7fc0040cf4d0 2024-04-25T01:17:39.624 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T01:17:39.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.643+0000 7fc00c800640 1 -- 172.21.15.90:0/2890406988 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbfe806be30 msgr2=0x7fbfe806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:39.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.643+0000 7fc00c800640 1 --2- 172.21.15.90:0/2890406988 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbfe806be30 0x7fbfe806e2f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbff80011e0 tx=0x7fbff8009040 comp rx=0 tx=0).stop 2024-04-25T01:17:39.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.643+0000 7fc00c800640 1 -- 172.21.15.90:0/2890406988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0040cf4d0 msgr2=0x7fc00415d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:39.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.643+0000 7fc00c800640 1 --2- 172.21.15.90:0/2890406988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0040cf4d0 0x7fc00415d7f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fbff400c400 tx=0x7fbff400c8c0 comp rx=0 tx=0).stop 2024-04-25T01:17:39.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.643+0000 7fc00c800640 1 -- 172.21.15.90:0/2890406988 shutdown_connections 2024-04-25T01:17:39.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.643+0000 7fc00c800640 1 --2- 172.21.15.90:0/2890406988 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbfe806be30 0x7fbfe806e2f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:39.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.643+0000 7fc00c800640 1 --2- 172.21.15.90:0/2890406988 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc00415fc40 0x7fc00415dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:39.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.643+0000 7fc00c800640 1 --2- 172.21.15.90:0/2890406988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0040cf4d0 0x7fc00415d7f0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:39.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.643+0000 7fc00c800640 1 --2- 172.21.15.90:0/2890406988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0040722e0 0x7fc0041593c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:39.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.643+0000 7fc00c800640 1 -- 172.21.15.90:0/2890406988 >> 172.21.15.90:0/2890406988 conn(0x7fc00406b720 msgr2=0x7fc0041559f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:39.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.644+0000 7fc00c800640 1 -- 172.21.15.90:0/2890406988 shutdown_connections 2024-04-25T01:17:39.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.644+0000 7fc00c800640 1 -- 172.21.15.90:0/2890406988 wait complete. 2024-04-25T01:17:39.645 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:17:39.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.865+0000 7f2450a00640 1 Processor -- start 2024-04-25T01:17:39.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.866+0000 7f2450a00640 1 -- start start 2024-04-25T01:17:39.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.867+0000 7f2450a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f244c06fe50 0x7f244c0b3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:39.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.867+0000 7f2450a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f244c06edc0 0x7f244c06f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:39.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.867+0000 7f2450a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f244c06dd20 con 0x7f244c06b120 2024-04-25T01:17:39.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.867+0000 7f2450a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f244c15e4f0 con 0x7f244c06edc0 2024-04-25T01:17:39.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.867+0000 7f2450a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f244c0b0430 con 0x7f244c06fe50 2024-04-25T01:17:39.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.868+0000 7f244b200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f244c06edc0 0x7f244c06f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:39.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.868+0000 7f2443e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f244c06fe50 0x7f244c0b3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:39.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.868+0000 7f244b200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f244c06edc0 0x7f244c06f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.90:37454/0 (socket says 172.21.15.90:37454) 2024-04-25T01:17:39.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.868+0000 7f244b200640 1 -- 172.21.15.90:0/2925344299 learned_addr learned my addr 172.21.15.90:0/2925344299 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:39.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.869+0000 7f244b200640 1 -- 172.21.15.90:0/2925344299 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f244c06fe50 msgr2=0x7f244c0b3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:39.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.869+0000 7f244b200640 1 --2- 172.21.15.90:0/2925344299 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f244c06fe50 0x7f244c0b3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:39.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.869+0000 7f244b200640 1 -- 172.21.15.90:0/2925344299 >> v1:172.21.15.103:6789/0 conn(0x7f244c06b120 legacy=0x7f244c06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:39.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.869+0000 7f244b200640 1 -- 172.21.15.90:0/2925344299 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f244c06e000 con 0x7f244c06edc0 2024-04-25T01:17:39.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.870+0000 7f244b200640 1 --2- 172.21.15.90:0/2925344299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f244c06edc0 0x7f244c06f250 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2444009e50 tx=0x7f244402a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f613b410b6da8e77 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:39.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.871+0000 7f2449e00640 1 -- 172.21.15.90:0/2925344299 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2444036070 con 0x7f244c06edc0 2024-04-25T01:17:39.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.871+0000 7f2449e00640 1 -- 172.21.15.90:0/2925344299 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2444031600 con 0x7f244c06edc0 2024-04-25T01:17:39.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.871+0000 7f2449e00640 1 -- 172.21.15.90:0/2925344299 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24440319d0 con 0x7f244c06edc0 2024-04-25T01:17:39.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.872+0000 7f2450a00640 1 -- 172.21.15.90:0/2925344299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f244c06edc0 msgr2=0x7f244c06f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:39.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.872+0000 7f2450a00640 1 --2- 172.21.15.90:0/2925344299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f244c06edc0 0x7f244c06f250 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2444009e50 tx=0x7f244402a680 comp rx=0 tx=0).stop 2024-04-25T01:17:39.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.872+0000 7f2450a00640 1 -- 172.21.15.90:0/2925344299 shutdown_connections 2024-04-25T01:17:39.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.872+0000 7f2450a00640 1 --2- 172.21.15.90:0/2925344299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f244c06edc0 0x7f244c06f250 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:39.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.872+0000 7f2450a00640 1 --2- 172.21.15.90:0/2925344299 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f244c06fe50 0x7f244c0b3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:39.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.872+0000 7f2450a00640 1 -- 172.21.15.90:0/2925344299 >> 172.21.15.90:0/2925344299 conn(0x7f244c0d0980 msgr2=0x7f244c0d0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:39.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.872+0000 7f2450a00640 1 -- 172.21.15.90:0/2925344299 shutdown_connections 2024-04-25T01:17:39.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.872+0000 7f2450a00640 1 -- 172.21.15.90:0/2925344299 wait complete. 2024-04-25T01:17:39.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.874+0000 7f2450a00640 1 Processor -- start 2024-04-25T01:17:39.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.874+0000 7f2450a00640 1 -- start start 2024-04-25T01:17:39.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.875+0000 7f2450a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f244c06b120 0x7f244c157ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:39.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.875+0000 7f2450a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f244c06edc0 0x7f244c1f4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:39.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.875+0000 7f2450a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f244c06fe50 0x7f244c0adcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:39.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.875+0000 7f2450a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f244c06e8a0 con 0x7f244c06fe50 2024-04-25T01:17:39.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.875+0000 7f2450a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f244c06e5c0 con 0x7f244c06edc0 2024-04-25T01:17:39.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.875+0000 7f2450a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f244c06d9d0 con 0x7f244c06b120 2024-04-25T01:17:39.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.875+0000 7f244b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f244c06fe50 0x7f244c0adcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:39.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.876+0000 7f244b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f244c06fe50 0x7f244c0adcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57112/0 (socket says 172.21.15.90:57112) 2024-04-25T01:17:39.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.876+0000 7f244b200640 1 -- 172.21.15.90:0/4292442094 learned_addr learned my addr 172.21.15.90:0/4292442094 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:39.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.876+0000 7f2443e00640 1 --2- 172.21.15.90:0/4292442094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f244c06edc0 0x7f244c1f4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:39.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.876+0000 7f244a800640 1 --2- 172.21.15.90:0/4292442094 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f244c06b120 0x7f244c157ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:39.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.876+0000 7f244b200640 1 -- 172.21.15.90:0/4292442094 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f244c06b120 msgr2=0x7f244c157ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:39.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.876+0000 7f244b200640 1 --2- 172.21.15.90:0/4292442094 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f244c06b120 0x7f244c157ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:39.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.876+0000 7f244b200640 1 -- 172.21.15.90:0/4292442094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f244c06edc0 msgr2=0x7f244c1f4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:39.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.876+0000 7f244b200640 1 --2- 172.21.15.90:0/4292442094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f244c06edc0 0x7f244c1f4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:39.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.876+0000 7f244b200640 1 -- 172.21.15.90:0/4292442094 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f244c158010 con 0x7f244c06fe50 2024-04-25T01:17:39.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.876+0000 7f244b200640 1 --2- 172.21.15.90:0/4292442094 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f244c06fe50 0x7f244c0adcc0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2444009e50 tx=0x7f2444004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:39.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.877+0000 7f2442a00640 1 -- 172.21.15.90:0/4292442094 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2444036050 con 0x7f244c06fe50 2024-04-25T01:17:39.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.877+0000 7f2442a00640 1 -- 172.21.15.90:0/4292442094 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2444042070 con 0x7f244c06fe50 2024-04-25T01:17:39.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.877+0000 7f2442a00640 1 -- 172.21.15.90:0/4292442094 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2444009510 con 0x7f244c06fe50 2024-04-25T01:17:39.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.877+0000 7f2450a00640 1 -- 172.21.15.90:0/4292442094 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f244c0cde70 con 0x7f244c06fe50 2024-04-25T01:17:39.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.877+0000 7f2450a00640 1 -- 172.21.15.90:0/4292442094 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f244c0ce430 con 0x7f244c06fe50 2024-04-25T01:17:39.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.878+0000 7f2450a00640 1 -- 172.21.15.90:0/4292442094 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f244c075350 con 0x7f244c06fe50 2024-04-25T01:17:39.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.879+0000 7f2442a00640 1 -- 172.21.15.90:0/4292442094 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2444007bf0 con 0x7f244c06fe50 2024-04-25T01:17:39.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.880+0000 7f2442a00640 1 --2- 172.21.15.90:0/4292442094 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f242806be30 0x7f242806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:39.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.880+0000 7f2442a00640 1 -- 172.21.15.90:0/4292442094 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f24440abfb0 con 0x7f244c06fe50 2024-04-25T01:17:39.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.880+0000 7f244a800640 1 --2- 172.21.15.90:0/4292442094 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f242806be30 0x7f242806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:39.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.881+0000 7f244a800640 1 --2- 172.21.15.90:0/4292442094 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f242806be30 0x7f242806e2f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f243c000f60 tx=0x7f243c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:39.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:39.884+0000 7f2442a00640 1 -- 172.21.15.90:0/4292442094 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f244c158010 con 0x7f244c06fe50 2024-04-25T01:17:40.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.085+0000 7f2450a00640 1 -- 172.21.15.90:0/4292442094 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f244c0d0730 con 0x7f244c06fe50 2024-04-25T01:17:40.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.086+0000 7f2442a00640 1 -- 172.21.15.90:0/4292442094 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) ==== 104+0+11 (secure 0 0 0) 0x7f244407a5e0 con 0x7f244c06fe50 2024-04-25T01:17:40.070 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:17:40.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.088+0000 7f2450a00640 1 -- 172.21.15.90:0/4292442094 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f242806be30 msgr2=0x7f242806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:40.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.089+0000 7f2450a00640 1 --2- 172.21.15.90:0/4292442094 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f242806be30 0x7f242806e2f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f243c000f60 tx=0x7f243c009040 comp rx=0 tx=0).stop 2024-04-25T01:17:40.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.089+0000 7f2450a00640 1 -- 172.21.15.90:0/4292442094 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f244c06fe50 msgr2=0x7f244c0adcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:40.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.089+0000 7f2450a00640 1 --2- 172.21.15.90:0/4292442094 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f244c06fe50 0x7f244c0adcc0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2444009e50 tx=0x7f2444004a40 comp rx=0 tx=0).stop 2024-04-25T01:17:40.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.089+0000 7f2450a00640 1 -- 172.21.15.90:0/4292442094 shutdown_connections 2024-04-25T01:17:40.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.089+0000 7f2450a00640 1 --2- 172.21.15.90:0/4292442094 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f242806be30 0x7f242806e2f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:40.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.089+0000 7f2450a00640 1 --2- 172.21.15.90:0/4292442094 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f244c06fe50 0x7f244c0adcc0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:40.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.089+0000 7f2450a00640 1 --2- 172.21.15.90:0/4292442094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f244c06edc0 0x7f244c1f4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:40.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.089+0000 7f2450a00640 1 --2- 172.21.15.90:0/4292442094 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f244c06b120 0x7f244c157ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:40.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.089+0000 7f2450a00640 1 -- 172.21.15.90:0/4292442094 >> 172.21.15.90:0/4292442094 conn(0x7f244c0d0980 msgr2=0x7f244c0b4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:40.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.089+0000 7f2450a00640 1 -- 172.21.15.90:0/4292442094 shutdown_connections 2024-04-25T01:17:40.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.089+0000 7f2450a00640 1 -- 172.21.15.90:0/4292442094 wait complete. 2024-04-25T01:17:40.092 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:17:40.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.310+0000 7f65f3200640 1 Processor -- start 2024-04-25T01:17:40.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.310+0000 7f65f3200640 1 -- start start 2024-04-25T01:17:40.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.312+0000 7f65f3200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f65f40aef80 0x7f65f40b3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:40.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.312+0000 7f65f3200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65f406ccb0 con 0x7f65f40cf850 2024-04-25T01:17:40.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.312+0000 7f65f3200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65f406cf90 con 0x7f65f40cfcb0 2024-04-25T01:17:40.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.312+0000 7f65f3200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f65f406ca60 con 0x7f65f40aef80 2024-04-25T01:17:40.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.312+0000 7f65f1400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f65f40cf850 0x7f65f4070a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:35196/0 (socket says 172.21.15.90:35196) 2024-04-25T01:17:40.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.312+0000 7f65f1400640 1 -- 172.21.15.90:0/49242426 learned_addr learned my addr 172.21.15.90:0/49242426 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:40.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.312+0000 7f65f1e00640 1 --2- 172.21.15.90:0/49242426 >> v2:172.21.15.103:3300/0 conn(0x7f65f40aef80 0x7f65f40b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:40.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.313+0000 7f65f0a00640 1 -- 172.21.15.90:0/49242426 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 392942762 0 0) 0x7f65f406ccb0 con 0x7f65f40cf850 2024-04-25T01:17:40.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.313+0000 7f65f0a00640 1 -- 172.21.15.90:0/49242426 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65d8003520 con 0x7f65f40cf850 2024-04-25T01:17:40.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.313+0000 7f65f0a00640 1 -- 172.21.15.90:0/49242426 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2522748739 0 0) 0x7f65d8003520 con 0x7f65f40cf850 2024-04-25T01:17:40.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.313+0000 7f65f0a00640 1 -- 172.21.15.90:0/49242426 >> v2:172.21.15.103:3300/0 conn(0x7f65f40aef80 msgr2=0x7f65f40b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:40.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.313+0000 7f65f0a00640 1 --2- 172.21.15.90:0/49242426 >> v2:172.21.15.103:3300/0 conn(0x7f65f40aef80 0x7f65f40b3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:40.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.313+0000 7f65f0a00640 1 -- 172.21.15.90:0/49242426 >> v1:172.21.15.103:6789/0 conn(0x7f65f40cfcb0 legacy=0x7f65f406da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:40.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.313+0000 7f65f0a00640 1 -- 172.21.15.90:0/49242426 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65f406d270 con 0x7f65f40cf850 2024-04-25T01:17:40.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.313+0000 7f65f0a00640 1 -- 172.21.15.90:0/49242426 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f65e0001670 con 0x7f65f40cf850 2024-04-25T01:17:40.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.313+0000 7f65f0a00640 1 -- 172.21.15.90:0/49242426 >> v1:172.21.15.90:6789/0 conn(0x7f65f40cf850 legacy=0x7f65f4070a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:40.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.313+0000 7f65f0a00640 1 --2- 172.21.15.90:0/49242426 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f65d80039b0 0x7f65d8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:40.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.313+0000 7f65f0a00640 1 --2- 172.21.15.90:0/49242426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65d8004800 0x7f65d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:40.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.313+0000 7f65f0a00640 1 --2- 172.21.15.90:0/49242426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65d80052e0 0x7f65d800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:40.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.313+0000 7f65f0a00640 1 -- 172.21.15.90:0/49242426 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f65f406ca60 con 0x7f65d80052e0 2024-04-25T01:17:40.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.313+0000 7f65f0a00640 1 -- 172.21.15.90:0/49242426 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f65d800c1b0 con 0x7f65d8004800 2024-04-25T01:17:40.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.313+0000 7f65f0a00640 1 -- 172.21.15.90:0/49242426 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f65d800c3b0 con 0x7f65d80039b0 2024-04-25T01:17:40.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.313+0000 7f65f2800640 1 --2- 172.21.15.90:0/49242426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65d80052e0 0x7f65d800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:40.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.314+0000 7f65f1400640 1 --2- 172.21.15.90:0/49242426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65d8004800 0x7f65d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:40.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.314+0000 7f65f1e00640 1 --2- 172.21.15.90:0/49242426 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f65d80039b0 0x7f65d8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:40.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.314+0000 7f65f2800640 1 -- 172.21.15.90:0/49242426 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f65d80039b0 msgr2=0x7f65d8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:40.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.314+0000 7f65f2800640 1 --2- 172.21.15.90:0/49242426 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f65d80039b0 0x7f65d8003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:40.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.314+0000 7f65f2800640 1 -- 172.21.15.90:0/49242426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65d8004800 msgr2=0x7f65d8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:40.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.314+0000 7f65f2800640 1 --2- 172.21.15.90:0/49242426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65d8004800 0x7f65d8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:40.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.314+0000 7f65f2800640 1 -- 172.21.15.90:0/49242426 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65d8003520 con 0x7f65d80052e0 2024-04-25T01:17:40.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.315+0000 7f65f2800640 1 -- 172.21.15.90:0/49242426 reap_dead start 2024-04-25T01:17:40.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.315+0000 7f65f2800640 1 --2- 172.21.15.90:0/49242426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65d80052e0 0x7f65d800b9c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f65f40d00b0 tx=0x7f65d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=855579c64807d98b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:40.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.315+0000 7f65f0a00640 1 -- 172.21.15.90:0/49242426 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ec001370 con 0x7f65d80052e0 2024-04-25T01:17:40.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.315+0000 7f65f0a00640 1 -- 172.21.15.90:0/49242426 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65ec02cd60 con 0x7f65d80052e0 2024-04-25T01:17:40.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.315+0000 7f65f0a00640 1 -- 172.21.15.90:0/49242426 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ec0052f0 con 0x7f65d80052e0 2024-04-25T01:17:40.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.316+0000 7f65f3200640 1 -- 172.21.15.90:0/49242426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65d80052e0 msgr2=0x7f65d800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:40.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.316+0000 7f65f3200640 1 --2- 172.21.15.90:0/49242426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65d80052e0 0x7f65d800b9c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f65f40d00b0 tx=0x7f65d8005830 comp rx=0 tx=0).stop 2024-04-25T01:17:40.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.316+0000 7f65f3200640 1 -- 172.21.15.90:0/49242426 shutdown_connections 2024-04-25T01:17:40.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.316+0000 7f65f3200640 1 --2- 172.21.15.90:0/49242426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65d80052e0 0x7f65d800b9c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:40.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.316+0000 7f65f3200640 1 -- 172.21.15.90:0/49242426 >> 172.21.15.90:0/49242426 conn(0x7f65f406b120 msgr2=0x7f65f406b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:40.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.316+0000 7f65f3200640 1 -- 172.21.15.90:0/49242426 shutdown_connections 2024-04-25T01:17:40.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.316+0000 7f65f3200640 1 -- 172.21.15.90:0/49242426 wait complete. 2024-04-25T01:17:40.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.318+0000 7f65f3200640 1 Processor -- start 2024-04-25T01:17:40.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.318+0000 7f65f3200640 1 -- start start 2024-04-25T01:17:40.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.319+0000 7f65f3200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65d80052e0 0x7f65f41529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:40.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.319+0000 7f65f3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f65f4152f20 0x7f65f41ef710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:40.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.319+0000 7f65f3200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65f41efc50 0x7f65f41f00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:40.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.319+0000 7f65f3200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f65f40b3620 con 0x7f65f41efc50 2024-04-25T01:17:40.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.319+0000 7f65f3200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f65f40b34a0 con 0x7f65d80052e0 2024-04-25T01:17:40.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.319+0000 7f65f3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f65f40b37a0 con 0x7f65f4152f20 2024-04-25T01:17:40.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.319+0000 7f65f2800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65f41efc50 0x7f65f41f00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:40.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.319+0000 7f65f2800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65f41efc50 0x7f65f41f00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57124/0 (socket says 172.21.15.90:57124) 2024-04-25T01:17:40.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.319+0000 7f65f2800640 1 -- 172.21.15.90:0/2926906787 learned_addr learned my addr 172.21.15.90:0/2926906787 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:40.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.319+0000 7f65f1e00640 1 --2- 172.21.15.90:0/2926906787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65d80052e0 0x7f65f41529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:40.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.319+0000 7f65f2800640 1 -- 172.21.15.90:0/2926906787 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f65f4152f20 msgr2=0x7f65f41ef710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:40.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.319+0000 7f65f2800640 1 --2- 172.21.15.90:0/2926906787 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f65f4152f20 0x7f65f41ef710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:40.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.320+0000 7f65f2800640 1 -- 172.21.15.90:0/2926906787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65d80052e0 msgr2=0x7f65f41529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:40.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.320+0000 7f65f2800640 1 --2- 172.21.15.90:0/2926906787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65d80052e0 0x7f65f41529e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:40.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.320+0000 7f65f2800640 1 -- 172.21.15.90:0/2926906787 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65f4150c00 con 0x7f65f41efc50 2024-04-25T01:17:40.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.320+0000 7f65f2800640 1 --2- 172.21.15.90:0/2926906787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65f41efc50 0x7f65f41f00e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f65ec02c750 tx=0x7f65ec001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:40.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.320+0000 7f65ea000640 1 -- 172.21.15.90:0/2926906787 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ec043070 con 0x7f65f41efc50 2024-04-25T01:17:40.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.320+0000 7f65ea000640 1 -- 172.21.15.90:0/2926906787 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65ec02cb60 con 0x7f65f41efc50 2024-04-25T01:17:40.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.320+0000 7f65ea000640 1 -- 172.21.15.90:0/2926906787 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ec007a30 con 0x7f65f41efc50 2024-04-25T01:17:40.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.320+0000 7f65f3200640 1 -- 172.21.15.90:0/2926906787 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65f4150ee0 con 0x7f65f41efc50 2024-04-25T01:17:40.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.320+0000 7f65f3200640 1 -- 172.21.15.90:0/2926906787 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65f41514a0 con 0x7f65f41efc50 2024-04-25T01:17:40.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.321+0000 7f65f3200640 1 -- 172.21.15.90:0/2926906787 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65f4075350 con 0x7f65f41efc50 2024-04-25T01:17:40.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.322+0000 7f65ea000640 1 -- 172.21.15.90:0/2926906787 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f65ec046070 con 0x7f65f41efc50 2024-04-25T01:17:40.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.323+0000 7f65ea000640 1 --2- 172.21.15.90:0/2926906787 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f65d406be60 0x7f65d406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:40.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.323+0000 7f65ea000640 1 -- 172.21.15.90:0/2926906787 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f65ec0ac330 con 0x7f65f41efc50 2024-04-25T01:17:40.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.323+0000 7f65f1e00640 1 --2- 172.21.15.90:0/2926906787 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f65d406be60 0x7f65d406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:40.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.324+0000 7f65f1e00640 1 --2- 172.21.15.90:0/2926906787 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f65d406be60 0x7f65d406e320 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f65f40cefd0 tx=0x7f65e400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:40.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.327+0000 7f65ea000640 1 -- 172.21.15.90:0/2926906787 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f65ec07a320 con 0x7f65f41efc50 2024-04-25T01:17:40.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.522+0000 7f65f3200640 1 -- 172.21.15.90:0/2926906787 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f65f4151810 con 0x7f65f41efc50 2024-04-25T01:17:40.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.523+0000 7f65ea000640 1 -- 172.21.15.90:0/2926906787 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) ==== 100+0+11 (secure 0 0 0) 0x7f65ec07e620 con 0x7f65f41efc50 2024-04-25T01:17:40.506 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-25T01:17:40.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.526+0000 7f65f3200640 1 -- 172.21.15.90:0/2926906787 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f65d406be60 msgr2=0x7f65d406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:40.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.526+0000 7f65f3200640 1 --2- 172.21.15.90:0/2926906787 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f65d406be60 0x7f65d406e320 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f65f40cefd0 tx=0x7f65e400b040 comp rx=0 tx=0).stop 2024-04-25T01:17:40.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.526+0000 7f65f3200640 1 -- 172.21.15.90:0/2926906787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65f41efc50 msgr2=0x7f65f41f00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:40.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.526+0000 7f65f3200640 1 --2- 172.21.15.90:0/2926906787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65f41efc50 0x7f65f41f00e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f65ec02c750 tx=0x7f65ec001120 comp rx=0 tx=0).stop 2024-04-25T01:17:40.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.526+0000 7f65f3200640 1 -- 172.21.15.90:0/2926906787 shutdown_connections 2024-04-25T01:17:40.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.526+0000 7f65f3200640 1 --2- 172.21.15.90:0/2926906787 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f65d406be60 0x7f65d406e320 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:40.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.526+0000 7f65f3200640 1 --2- 172.21.15.90:0/2926906787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65f41efc50 0x7f65f41f00e0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:40.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.526+0000 7f65f3200640 1 --2- 172.21.15.90:0/2926906787 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f65f4152f20 0x7f65f41ef710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:40.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.526+0000 7f65f3200640 1 --2- 172.21.15.90:0/2926906787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65d80052e0 0x7f65f41529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:40.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.526+0000 7f65f3200640 1 -- 172.21.15.90:0/2926906787 >> 172.21.15.90:0/2926906787 conn(0x7f65f406b120 msgr2=0x7f65f40b08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:40.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.526+0000 7f65f3200640 1 -- 172.21.15.90:0/2926906787 shutdown_connections 2024-04-25T01:17:40.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.526+0000 7f65f3200640 1 -- 172.21.15.90:0/2926906787 wait complete. 2024-04-25T01:17:40.529 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:17:40.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.748+0000 7f42ff200640 1 Processor -- start 2024-04-25T01:17:40.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.748+0000 7f42ff200640 1 -- start start 2024-04-25T01:17:40.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.749+0000 7f42ff200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f43001596c0 0x7f430006de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:40.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.749+0000 7f42ff200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f43000cff00 0x7f430006e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:40.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.749+0000 7f42ff200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f430015c200 con 0x7f43000b2880 2024-04-25T01:17:40.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.749+0000 7f42ff200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f43000d0330 con 0x7f43000cff00 2024-04-25T01:17:40.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.749+0000 7f42ff200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4300159aa0 con 0x7f43001596c0 2024-04-25T01:17:40.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.750+0000 7f42fe800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f43000cff00 0x7f430006e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:40.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.750+0000 7f42fe800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f43000cff00 0x7f430006e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57128/0 (socket says 172.21.15.90:57128) 2024-04-25T01:17:40.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.750+0000 7f42fe800640 1 -- 172.21.15.90:0/1904049578 learned_addr learned my addr 172.21.15.90:0/1904049578 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:40.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.750+0000 7f42fde00640 1 --2- 172.21.15.90:0/1904049578 >> v2:172.21.15.103:3300/0 conn(0x7f43001596c0 0x7f430006de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:40.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.750+0000 7f42fe800640 1 -- 172.21.15.90:0/1904049578 >> v2:172.21.15.103:3300/0 conn(0x7f43001596c0 msgr2=0x7f430006de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:40.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.750+0000 7f42fe800640 1 --2- 172.21.15.90:0/1904049578 >> v2:172.21.15.103:3300/0 conn(0x7f43001596c0 0x7f430006de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:40.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.750+0000 7f42fe800640 1 -- 172.21.15.90:0/1904049578 >> v1:172.21.15.103:6789/0 conn(0x7f43000b2880 legacy=0x7f43000cf7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:40.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.750+0000 7f42fe800640 1 -- 172.21.15.90:0/1904049578 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f430015c470 con 0x7f43000cff00 2024-04-25T01:17:40.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.750+0000 7f42fde00640 1 --2- 172.21.15.90:0/1904049578 >> v2:172.21.15.103:3300/0 conn(0x7f43001596c0 0x7f430006de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:40.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.751+0000 7f42fe800640 1 --2- 172.21.15.90:0/1904049578 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43000cff00 0x7f430006e3d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f430006ec90 tx=0x7f42f802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6c17b3d36019577 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:40.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.752+0000 7f42fca00640 1 -- 172.21.15.90:0/1904049578 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f8036070 con 0x7f43000cff00 2024-04-25T01:17:40.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.752+0000 7f42fca00640 1 -- 172.21.15.90:0/1904049578 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42f8031600 con 0x7f43000cff00 2024-04-25T01:17:40.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.752+0000 7f42fca00640 1 -- 172.21.15.90:0/1904049578 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f80319d0 con 0x7f43000cff00 2024-04-25T01:17:40.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.752+0000 7f42ff200640 1 -- 172.21.15.90:0/1904049578 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43000cff00 msgr2=0x7f430006e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:40.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.752+0000 7f42ff200640 1 --2- 172.21.15.90:0/1904049578 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43000cff00 0x7f430006e3d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f430006ec90 tx=0x7f42f802a6f0 comp rx=0 tx=0).stop 2024-04-25T01:17:40.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.753+0000 7f42ff200640 1 -- 172.21.15.90:0/1904049578 shutdown_connections 2024-04-25T01:17:40.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.753+0000 7f42ff200640 1 --2- 172.21.15.90:0/1904049578 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43000cff00 0x7f430006e3d0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:40.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.753+0000 7f42ff200640 1 --2- 172.21.15.90:0/1904049578 >> v2:172.21.15.103:3300/0 conn(0x7f43001596c0 0x7f430006de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:40.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.753+0000 7f42ff200640 1 -- 172.21.15.90:0/1904049578 >> 172.21.15.90:0/1904049578 conn(0x7f430006b120 msgr2=0x7f430006b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:40.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.753+0000 7f42ff200640 1 -- 172.21.15.90:0/1904049578 shutdown_connections 2024-04-25T01:17:40.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.753+0000 7f42ff200640 1 -- 172.21.15.90:0/1904049578 wait complete. 2024-04-25T01:17:40.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.754+0000 7f42ff200640 1 Processor -- start 2024-04-25T01:17:40.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.755+0000 7f42ff200640 1 -- start start 2024-04-25T01:17:40.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.756+0000 7f42ff200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43000b2880 0x7f4300072670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:40.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.756+0000 7f42ff200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43000cff00 0x7f43001410f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:40.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.756+0000 7f42ff200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43001596c0 0x7f430015d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:40.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.756+0000 7f42ff200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f430015cd10 con 0x7f43000cff00 2024-04-25T01:17:40.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.756+0000 7f42ff200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f430015ca30 con 0x7f43000b2880 2024-04-25T01:17:40.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.756+0000 7f42ff200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f430015f770 con 0x7f43001596c0 2024-04-25T01:17:40.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.756+0000 7f42fd400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43000cff00 0x7f43001410f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:40.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.756+0000 7f42fd400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43000cff00 0x7f43001410f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57130/0 (socket says 172.21.15.90:57130) 2024-04-25T01:17:40.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.756+0000 7f42fd400640 1 -- 172.21.15.90:0/3224086233 learned_addr learned my addr 172.21.15.90:0/3224086233 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:40.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.756+0000 7f42fde00640 1 --2- 172.21.15.90:0/3224086233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43000b2880 0x7f4300072670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:40.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.756+0000 7f42fd400640 1 -- 172.21.15.90:0/3224086233 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43001596c0 msgr2=0x7f430015d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:40.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.756+0000 7f42fd400640 1 --2- 172.21.15.90:0/3224086233 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43001596c0 0x7f430015d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:40.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.756+0000 7f42fd400640 1 -- 172.21.15.90:0/3224086233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43000b2880 msgr2=0x7f4300072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:40.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.756+0000 7f42fd400640 1 --2- 172.21.15.90:0/3224086233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43000b2880 0x7f4300072670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:40.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.756+0000 7f42fd400640 1 -- 172.21.15.90:0/3224086233 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43001570b0 con 0x7f43000cff00 2024-04-25T01:17:40.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.756+0000 7f42fde00640 1 --2- 172.21.15.90:0/3224086233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43000b2880 0x7f4300072670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:40.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.757+0000 7f42fd400640 1 --2- 172.21.15.90:0/3224086233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43000cff00 0x7f43001410f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f42ec00c410 tx=0x7f42ec00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:40.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.757+0000 7f42f6000640 1 -- 172.21.15.90:0/3224086233 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ec017070 con 0x7f43000cff00 2024-04-25T01:17:40.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.757+0000 7f42f6000640 1 -- 172.21.15.90:0/3224086233 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42ec0035f0 con 0x7f43000cff00 2024-04-25T01:17:40.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.757+0000 7f42f6000640 1 -- 172.21.15.90:0/3224086233 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ec003a20 con 0x7f43000cff00 2024-04-25T01:17:40.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.757+0000 7f42ff200640 1 -- 172.21.15.90:0/3224086233 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4300158aa0 con 0x7f43000cff00 2024-04-25T01:17:40.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.757+0000 7f42ff200640 1 -- 172.21.15.90:0/3224086233 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4300141630 con 0x7f43000cff00 2024-04-25T01:17:40.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.758+0000 7f42ff200640 1 -- 172.21.15.90:0/3224086233 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4300075350 con 0x7f43000cff00 2024-04-25T01:17:40.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.759+0000 7f42f6000640 1 -- 172.21.15.90:0/3224086233 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f42ec01a070 con 0x7f43000cff00 2024-04-25T01:17:40.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.760+0000 7f42f6000640 1 --2- 172.21.15.90:0/3224086233 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f42e006be30 0x7f42e006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:40.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.761+0000 7f42f6000640 1 -- 172.21.15.90:0/3224086233 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f42ec08a750 con 0x7f43000cff00 2024-04-25T01:17:40.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.761+0000 7f42fde00640 1 --2- 172.21.15.90:0/3224086233 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f42e006be30 0x7f42e006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:40.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.762+0000 7f42fde00640 1 --2- 172.21.15.90:0/3224086233 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f42e006be30 0x7f42e006e2f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f42f0006fd0 tx=0x7f42f0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:40.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.765+0000 7f42f6000640 1 -- 172.21.15.90:0/3224086233 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f42ec058cf0 con 0x7f43000cff00 2024-04-25T01:17:40.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.966+0000 7f42ff200640 1 -- 172.21.15.90:0/3224086233 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f430015f4b0 con 0x7f43000cff00 2024-04-25T01:17:40.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.967+0000 7f42f6000640 1 -- 172.21.15.90:0/3224086233 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) ==== 103+0+11 (secure 0 0 0) 0x7f42ec0585f0 con 0x7f43000cff00 2024-04-25T01:17:40.951 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:17:40.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.970+0000 7f42ff200640 1 -- 172.21.15.90:0/3224086233 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f42e006be30 msgr2=0x7f42e006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:40.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.970+0000 7f42ff200640 1 --2- 172.21.15.90:0/3224086233 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f42e006be30 0x7f42e006e2f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f42f0006fd0 tx=0x7f42f0008040 comp rx=0 tx=0).stop 2024-04-25T01:17:40.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.970+0000 7f42ff200640 1 -- 172.21.15.90:0/3224086233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43000cff00 msgr2=0x7f43001410f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:40.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.970+0000 7f42ff200640 1 --2- 172.21.15.90:0/3224086233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43000cff00 0x7f43001410f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f42ec00c410 tx=0x7f42ec00c8d0 comp rx=0 tx=0).stop 2024-04-25T01:17:40.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.970+0000 7f42ff200640 1 -- 172.21.15.90:0/3224086233 shutdown_connections 2024-04-25T01:17:40.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.970+0000 7f42ff200640 1 --2- 172.21.15.90:0/3224086233 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f42e006be30 0x7f42e006e2f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:40.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.970+0000 7f42ff200640 1 --2- 172.21.15.90:0/3224086233 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f43001596c0 0x7f430015d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:40.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.970+0000 7f42ff200640 1 --2- 172.21.15.90:0/3224086233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43000cff00 0x7f43001410f0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:40.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.970+0000 7f42ff200640 1 --2- 172.21.15.90:0/3224086233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43000b2880 0x7f4300072670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:40.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.970+0000 7f42ff200640 1 -- 172.21.15.90:0/3224086233 >> 172.21.15.90:0/3224086233 conn(0x7f430006b120 msgr2=0x7f430015fb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:40.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.970+0000 7f42ff200640 1 -- 172.21.15.90:0/3224086233 shutdown_connections 2024-04-25T01:17:40.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:40.970+0000 7f42ff200640 1 -- 172.21.15.90:0/3224086233 wait complete. 2024-04-25T01:17:40.972 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T01:17:41.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.186+0000 7fa4db400640 1 Processor -- start 2024-04-25T01:17:41.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.186+0000 7fa4db400640 1 -- start start 2024-04-25T01:17:41.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.188+0000 7fa4db400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa4d4143940 0x7fa4d406e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:41.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.188+0000 7fa4db400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa4d415f400 0x7fa4d406eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:41.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.188+0000 7fa4db400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4d4070e60 con 0x7fa4d4143d20 2024-04-25T01:17:41.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.188+0000 7fa4db400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fa4d4071140 con 0x7fa4d415f400 2024-04-25T01:17:41.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.188+0000 7fa4db400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa4d4071420 con 0x7fa4d4143940 2024-04-25T01:17:41.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.188+0000 7fa4d9400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa4d415f400 0x7fa4d406eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:41.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.188+0000 7fa4d9400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa4d415f400 0x7fa4d406eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57144/0 (socket says 172.21.15.90:57144) 2024-04-25T01:17:41.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.188+0000 7fa4d9400640 1 -- 172.21.15.90:0/4070899844 learned_addr learned my addr 172.21.15.90:0/4070899844 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:41.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.188+0000 7fa4d8a00640 1 --2- 172.21.15.90:0/4070899844 >> v2:172.21.15.103:3300/0 conn(0x7fa4d4143940 0x7fa4d406e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:41.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.189+0000 7fa4d9400640 1 -- 172.21.15.90:0/4070899844 >> v2:172.21.15.103:3300/0 conn(0x7fa4d4143940 msgr2=0x7fa4d406e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:41.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.189+0000 7fa4d9400640 1 --2- 172.21.15.90:0/4070899844 >> v2:172.21.15.103:3300/0 conn(0x7fa4d4143940 0x7fa4d406e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:41.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.189+0000 7fa4d9400640 1 -- 172.21.15.90:0/4070899844 >> v1:172.21.15.103:6789/0 conn(0x7fa4d4143d20 legacy=0x7fa4d415ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:41.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.189+0000 7fa4d9400640 1 -- 172.21.15.90:0/4070899844 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d4071700 con 0x7fa4d415f400 2024-04-25T01:17:41.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.190+0000 7fa4d9400640 1 --2- 172.21.15.90:0/4070899844 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa4d415f400 0x7fa4d406eb90 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc009e50 tx=0x7fa4cc02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bdbb3da3a8b8d5c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:41.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.190+0000 7fa4d3400640 1 -- 172.21.15.90:0/4070899844 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc036070 con 0x7fa4d415f400 2024-04-25T01:17:41.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.190+0000 7fa4d3400640 1 -- 172.21.15.90:0/4070899844 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4cc031600 con 0x7fa4d415f400 2024-04-25T01:17:41.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.190+0000 7fa4d3400640 1 -- 172.21.15.90:0/4070899844 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc0319d0 con 0x7fa4d415f400 2024-04-25T01:17:41.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.191+0000 7fa4db400640 1 -- 172.21.15.90:0/4070899844 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa4d415f400 msgr2=0x7fa4d406eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:41.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.191+0000 7fa4db400640 1 --2- 172.21.15.90:0/4070899844 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa4d415f400 0x7fa4d406eb90 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc009e50 tx=0x7fa4cc02a710 comp rx=0 tx=0).stop 2024-04-25T01:17:41.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.191+0000 7fa4db400640 1 -- 172.21.15.90:0/4070899844 shutdown_connections 2024-04-25T01:17:41.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.191+0000 7fa4db400640 1 --2- 172.21.15.90:0/4070899844 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa4d415f400 0x7fa4d406eb90 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:41.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.191+0000 7fa4db400640 1 --2- 172.21.15.90:0/4070899844 >> v2:172.21.15.103:3300/0 conn(0x7fa4d4143940 0x7fa4d406e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:41.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.191+0000 7fa4db400640 1 -- 172.21.15.90:0/4070899844 >> 172.21.15.90:0/4070899844 conn(0x7fa4d40b1820 msgr2=0x7fa4d40b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:41.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.191+0000 7fa4db400640 1 -- 172.21.15.90:0/4070899844 shutdown_connections 2024-04-25T01:17:41.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.191+0000 7fa4db400640 1 -- 172.21.15.90:0/4070899844 wait complete. 2024-04-25T01:17:41.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.193+0000 7fa4db400640 1 Processor -- start 2024-04-25T01:17:41.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.193+0000 7fa4db400640 1 -- start start 2024-04-25T01:17:41.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.194+0000 7fa4db400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa4d4143940 0x7fa4d4159b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:41.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.194+0000 7fa4db400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4d4143d20 0x7fa4d4157e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:41.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.194+0000 7fa4db400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4d415f400 0x7fa4d406e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:41.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.194+0000 7fa4db400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d4071ff0 con 0x7fa4d4143940 2024-04-25T01:17:41.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.194+0000 7fa4db400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d4071e70 con 0x7fa4d4143d20 2024-04-25T01:17:41.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.194+0000 7fa4db400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa4d4072170 con 0x7fa4d415f400 2024-04-25T01:17:41.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.194+0000 7fa4d8a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa4d4143940 0x7fa4d4159b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:41.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.194+0000 7fa4d8a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa4d4143940 0x7fa4d4159b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57156/0 (socket says 172.21.15.90:57156) 2024-04-25T01:17:41.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.194+0000 7fa4d8a00640 1 -- 172.21.15.90:0/2013635207 learned_addr learned my addr 172.21.15.90:0/2013635207 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:41.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.194+0000 7fa4d8a00640 1 -- 172.21.15.90:0/2013635207 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4d415f400 msgr2=0x7fa4d406e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:41.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.194+0000 7fa4d8a00640 1 --2- 172.21.15.90:0/2013635207 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4d415f400 0x7fa4d406e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:41.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.194+0000 7fa4d8a00640 1 -- 172.21.15.90:0/2013635207 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4d4143d20 msgr2=0x7fa4d4157e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:41.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.194+0000 7fa4d8a00640 1 --2- 172.21.15.90:0/2013635207 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4d4143d20 0x7fa4d4157e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:41.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.194+0000 7fa4d8a00640 1 -- 172.21.15.90:0/2013635207 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d41570a0 con 0x7fa4d4143940 2024-04-25T01:17:41.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.195+0000 7fa4d8a00640 1 --2- 172.21.15.90:0/2013635207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa4d4143940 0x7fa4d4159b60 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa4c800d800 tx=0x7fa4c800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:41.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.195+0000 7fa4d0c00640 1 -- 172.21.15.90:0/2013635207 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c800b6c0 con 0x7fa4d4143940 2024-04-25T01:17:41.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.195+0000 7fa4d0c00640 1 -- 172.21.15.90:0/2013635207 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4c8019070 con 0x7fa4d4143940 2024-04-25T01:17:41.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.195+0000 7fa4d0c00640 1 -- 172.21.15.90:0/2013635207 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c800ba80 con 0x7fa4d4143940 2024-04-25T01:17:41.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.195+0000 7fa4db400640 1 -- 172.21.15.90:0/2013635207 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4d4157380 con 0x7fa4d4143940 2024-04-25T01:17:41.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.195+0000 7fa4db400640 1 -- 172.21.15.90:0/2013635207 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4d4157940 con 0x7fa4d4143940 2024-04-25T01:17:41.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.196+0000 7fa4db400640 1 -- 172.21.15.90:0/2013635207 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4d4068fb0 con 0x7fa4d4143940 2024-04-25T01:17:41.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.197+0000 7fa4d0c00640 1 -- 172.21.15.90:0/2013635207 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa4c800bc20 con 0x7fa4d4143940 2024-04-25T01:17:41.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.198+0000 7fa4d0c00640 1 --2- 172.21.15.90:0/2013635207 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa4b406be30 0x7fa4b406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:41.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.198+0000 7fa4d0c00640 1 -- 172.21.15.90:0/2013635207 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7fa4c808b660 con 0x7fa4d4143940 2024-04-25T01:17:41.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.199+0000 7fa4d3e00640 1 --2- 172.21.15.90:0/2013635207 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa4b406be30 0x7fa4b406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:41.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.199+0000 7fa4d3e00640 1 --2- 172.21.15.90:0/2013635207 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa4b406be30 0x7fa4b406e2f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa4d415d320 tx=0x7fa4c00079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:41.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.203+0000 7fa4d0c00640 1 -- 172.21.15.90:0/2013635207 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa4c8059ca0 con 0x7fa4d4143940 2024-04-25T01:17:41.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.401+0000 7fa4db400640 1 -- 172.21.15.90:0/2013635207 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fa4d406d150 con 0x7fa4d4143940 2024-04-25T01:17:41.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.402+0000 7fa4d0c00640 1 -- 172.21.15.90:0/2013635207 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v28) ==== 80+0+0 (secure 0 0 0) 0x7fa4c80595a0 con 0x7fa4d4143940 2024-04-25T01:17:41.386 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-04-25T01:17:41.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.404+0000 7fa4db400640 1 -- 172.21.15.90:0/2013635207 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa4b406be30 msgr2=0x7fa4b406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:41.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.404+0000 7fa4db400640 1 --2- 172.21.15.90:0/2013635207 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa4b406be30 0x7fa4b406e2f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa4d415d320 tx=0x7fa4c00079b0 comp rx=0 tx=0).stop 2024-04-25T01:17:41.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.405+0000 7fa4db400640 1 -- 172.21.15.90:0/2013635207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa4d4143940 msgr2=0x7fa4d4159b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:41.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.405+0000 7fa4db400640 1 --2- 172.21.15.90:0/2013635207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa4d4143940 0x7fa4d4159b60 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa4c800d800 tx=0x7fa4c800dcc0 comp rx=0 tx=0).stop 2024-04-25T01:17:41.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.405+0000 7fa4db400640 1 -- 172.21.15.90:0/2013635207 shutdown_connections 2024-04-25T01:17:41.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.405+0000 7fa4db400640 1 --2- 172.21.15.90:0/2013635207 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa4b406be30 0x7fa4b406e2f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:41.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.405+0000 7fa4db400640 1 --2- 172.21.15.90:0/2013635207 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa4d415f400 0x7fa4d406e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:41.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.405+0000 7fa4db400640 1 --2- 172.21.15.90:0/2013635207 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4d4143d20 0x7fa4d4157e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:41.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.405+0000 7fa4db400640 1 --2- 172.21.15.90:0/2013635207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa4d4143940 0x7fa4d4159b60 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:41.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.405+0000 7fa4db400640 1 -- 172.21.15.90:0/2013635207 >> 172.21.15.90:0/2013635207 conn(0x7fa4d40b1820 msgr2=0x7fa4d4072530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:41.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.405+0000 7fa4db400640 1 -- 172.21.15.90:0/2013635207 shutdown_connections 2024-04-25T01:17:41.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.405+0000 7fa4db400640 1 -- 172.21.15.90:0/2013635207 wait complete. 2024-04-25T01:17:41.406 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-04-25T01:17:41.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:41.406 DEBUG:teuthology.orchestra.run.smithi090:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-04-25T01:17:41.484 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-04-25T01:17:41.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:41.484 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-25T01:17:41.533 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:17:41.534 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:17:41.591 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:17:41.591 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:17:41.647 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:17:41.648 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:17:41.700 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:17:41.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.923+0000 7f66cb000640 1 Processor -- start 2024-04-25T01:17:41.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.923+0000 7f66cb000640 1 -- start start 2024-04-25T01:17:41.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.925+0000 7f66cb000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f66c4154ac0 0x7f66c415b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:41.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.925+0000 7f66cb000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f66c406e550 0x7f66c406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:41.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.925+0000 7f66cb000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66c4154ea0 con 0x7f66c40cf2a0 2024-04-25T01:17:41.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.925+0000 7f66cb000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f66c406eee0 con 0x7f66c406e550 2024-04-25T01:17:41.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.925+0000 7f66cb000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f66c406d540 con 0x7f66c4154ac0 2024-04-25T01:17:41.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.925+0000 7f66c3400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f66c40cf2a0 0x7f66c40711f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:35212/0 (socket says 172.21.15.90:35212) 2024-04-25T01:17:41.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.925+0000 7f66c3400640 1 -- 172.21.15.90:0/6360386 learned_addr learned my addr 172.21.15.90:0/6360386 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:41.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.926+0000 7f66c9000640 1 --2- 172.21.15.90:0/6360386 >> v2:172.21.15.103:3300/0 conn(0x7f66c406e550 0x7f66c406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:41.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.926+0000 7f66c3e00640 1 --2- 172.21.15.90:0/6360386 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f66c4154ac0 0x7f66c415b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:41.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.926+0000 7f66c2a00640 1 -- 172.21.15.90:0/6360386 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3298074045 0 0) 0x7f66c4154ea0 con 0x7f66c40cf2a0 2024-04-25T01:17:41.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.926+0000 7f66c2a00640 1 -- 172.21.15.90:0/6360386 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66a4003440 con 0x7f66c40cf2a0 2024-04-25T01:17:41.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.926+0000 7f66c2a00640 1 -- 172.21.15.90:0/6360386 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3045767166 0 0) 0x7f66a4003440 con 0x7f66c40cf2a0 2024-04-25T01:17:41.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.926+0000 7f66c2a00640 1 -- 172.21.15.90:0/6360386 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f66c4154ac0 msgr2=0x7f66c415b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:41.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.926+0000 7f66c2a00640 1 --2- 172.21.15.90:0/6360386 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f66c4154ac0 0x7f66c415b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:41.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.926+0000 7f66c2a00640 1 -- 172.21.15.90:0/6360386 >> v2:172.21.15.103:3300/0 conn(0x7f66c406e550 msgr2=0x7f66c406e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:41.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.926+0000 7f66c2a00640 1 --2- 172.21.15.90:0/6360386 >> v2:172.21.15.103:3300/0 conn(0x7f66c406e550 0x7f66c406e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:41.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.926+0000 7f66c2a00640 1 -- 172.21.15.90:0/6360386 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66c406d790 con 0x7f66c40cf2a0 2024-04-25T01:17:41.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.926+0000 7f66c2a00640 1 -- 172.21.15.90:0/6360386 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f66ac0019f0 con 0x7f66c40cf2a0 2024-04-25T01:17:41.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.926+0000 7f66c2a00640 1 -- 172.21.15.90:0/6360386 >> v1:172.21.15.90:6789/0 conn(0x7f66c40cf2a0 legacy=0x7f66c40711f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:41.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.926+0000 7f66c2a00640 1 --2- 172.21.15.90:0/6360386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66a4003aa0 0x7f66a4003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:41.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.926+0000 7f66c2a00640 1 --2- 172.21.15.90:0/6360386 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f66c4154ac0 0x7f66a400a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:41.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.926+0000 7f66c2a00640 1 --2- 172.21.15.90:0/6360386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66a400aee0 0x7f66a400d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:41.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.926+0000 7f66c2a00640 1 -- 172.21.15.90:0/6360386 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f66c406eee0 con 0x7f66a4003aa0 2024-04-25T01:17:41.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.926+0000 7f66c2a00640 1 -- 172.21.15.90:0/6360386 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f66c406d540 con 0x7f66a400aee0 2024-04-25T01:17:41.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.926+0000 7f66c2a00640 1 -- 172.21.15.90:0/6360386 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f66a400dbd0 con 0x7f66c4154ac0 2024-04-25T01:17:41.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.927+0000 7f66c3e00640 1 --2- 172.21.15.90:0/6360386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66a4003aa0 0x7f66a4003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:41.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.927+0000 7f66c3e00640 1 -- 172.21.15.90:0/6360386 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f66c4154ac0 msgr2=0x7f66a400a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:41.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.927+0000 7f66c3e00640 1 --2- 172.21.15.90:0/6360386 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f66c4154ac0 0x7f66a400a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:41.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.927+0000 7f66c9000640 1 --2- 172.21.15.90:0/6360386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66a400aee0 0x7f66a400d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:41.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.927+0000 7f66c3e00640 1 -- 172.21.15.90:0/6360386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66a400aee0 msgr2=0x7f66a400d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:41.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.927+0000 7f66c3e00640 1 --2- 172.21.15.90:0/6360386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66a400aee0 0x7f66a400d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:41.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.927+0000 7f66c3e00640 1 -- 172.21.15.90:0/6360386 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66a4003440 con 0x7f66a4003aa0 2024-04-25T01:17:41.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.928+0000 7f66c3e00640 1 --2- 172.21.15.90:0/6360386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66a4003aa0 0x7f66a4003ec0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f66b400ba30 tx=0x7f66a4003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d674cd6a42e44cc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:41.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.928+0000 7f66c2a00640 1 -- 172.21.15.90:0/6360386 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b4030070 con 0x7f66a4003aa0 2024-04-25T01:17:41.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.928+0000 7f66c2a00640 1 -- 172.21.15.90:0/6360386 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66b402ae10 con 0x7f66a4003aa0 2024-04-25T01:17:41.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.928+0000 7f66c2a00640 1 -- 172.21.15.90:0/6360386 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b400b600 con 0x7f66a4003aa0 2024-04-25T01:17:41.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.929+0000 7f66cb000640 1 -- 172.21.15.90:0/6360386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66a4003aa0 msgr2=0x7f66a4003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:41.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.929+0000 7f66cb000640 1 --2- 172.21.15.90:0/6360386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66a4003aa0 0x7f66a4003ec0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f66b400ba30 tx=0x7f66a4003730 comp rx=0 tx=0).stop 2024-04-25T01:17:41.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.929+0000 7f66c9000640 1 -- 172.21.15.90:0/6360386 reap_dead start 2024-04-25T01:17:41.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.929+0000 7f66cb000640 1 -- 172.21.15.90:0/6360386 shutdown_connections 2024-04-25T01:17:41.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.929+0000 7f66cb000640 1 -- 172.21.15.90:0/6360386 >> 172.21.15.90:0/6360386 conn(0x7f66c40b1820 msgr2=0x7f66c40b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:41.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.929+0000 7f66cb000640 1 -- 172.21.15.90:0/6360386 shutdown_connections 2024-04-25T01:17:41.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.929+0000 7f66cb000640 1 -- 172.21.15.90:0/6360386 wait complete. 2024-04-25T01:17:41.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.931+0000 7f66cb000640 1 Processor -- start 2024-04-25T01:17:41.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.931+0000 7f66cb000640 1 -- start start 2024-04-25T01:17:41.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.932+0000 7f66cb000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f66a4003aa0 0x7f66c4155340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:41.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.932+0000 7f66cb000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66c40ad8e0 0x7f66c41426f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:41.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.932+0000 7f66cb000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66c4143060 0x7f66c41573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:41.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.932+0000 7f66cb000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f66c406e030 con 0x7f66c4143060 2024-04-25T01:17:41.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.932+0000 7f66cb000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f66c406dd50 con 0x7f66c40ad8e0 2024-04-25T01:17:41.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.932+0000 7f66cb000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f66c4071900 con 0x7f66a4003aa0 2024-04-25T01:17:41.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.932+0000 7f66c9000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66c4143060 0x7f66c41573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:41.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.932+0000 7f66c9000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66c4143060 0x7f66c41573b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57178/0 (socket says 172.21.15.90:57178) 2024-04-25T01:17:41.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.932+0000 7f66c9000640 1 -- 172.21.15.90:0/3773578587 learned_addr learned my addr 172.21.15.90:0/3773578587 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:41.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.933+0000 7f66c3400640 1 --2- 172.21.15.90:0/3773578587 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66c40ad8e0 0x7f66c41426f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:41.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.933+0000 7f66c9000640 1 -- 172.21.15.90:0/3773578587 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f66a4003aa0 msgr2=0x7f66c4155340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:41.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.933+0000 7f66c9000640 1 --2- 172.21.15.90:0/3773578587 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f66a4003aa0 0x7f66c4155340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:41.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.933+0000 7f66c9000640 1 -- 172.21.15.90:0/3773578587 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66c40ad8e0 msgr2=0x7f66c41426f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:41.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.933+0000 7f66c9000640 1 --2- 172.21.15.90:0/3773578587 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66c40ad8e0 0x7f66c41426f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:41.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.933+0000 7f66c9000640 1 -- 172.21.15.90:0/3773578587 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66c40ae110 con 0x7f66c4143060 2024-04-25T01:17:41.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.933+0000 7f66c9000640 1 --2- 172.21.15.90:0/3773578587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66c4143060 0x7f66c41573b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f66c40d0bc0 tx=0x7f66bc010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:41.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.933+0000 7f66bbe00640 1 -- 172.21.15.90:0/3773578587 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66bc00f040 con 0x7f66c4143060 2024-04-25T01:17:41.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.933+0000 7f66bbe00640 1 -- 172.21.15.90:0/3773578587 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66bc01c070 con 0x7f66c4143060 2024-04-25T01:17:41.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.933+0000 7f66bbe00640 1 -- 172.21.15.90:0/3773578587 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66bc00eb00 con 0x7f66c4143060 2024-04-25T01:17:41.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.933+0000 7f66cb000640 1 -- 172.21.15.90:0/3773578587 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66c40ae3f0 con 0x7f66c4143060 2024-04-25T01:17:41.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.933+0000 7f66cb000640 1 -- 172.21.15.90:0/3773578587 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66c4156e90 con 0x7f66c4143060 2024-04-25T01:17:41.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.934+0000 7f66cb000640 1 -- 172.21.15.90:0/3773578587 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66c4157090 con 0x7f66c4143060 2024-04-25T01:17:41.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.935+0000 7f66bbe00640 1 -- 172.21.15.90:0/3773578587 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f66bc00eca0 con 0x7f66c4143060 2024-04-25T01:17:41.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.937+0000 7f66bbe00640 1 --2- 172.21.15.90:0/3773578587 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f66a006be30 0x7f66a006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:41.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.937+0000 7f66bbe00640 1 -- 172.21.15.90:0/3773578587 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f66bc08ae10 con 0x7f66c4143060 2024-04-25T01:17:41.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.937+0000 7f66c3e00640 1 --2- 172.21.15.90:0/3773578587 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f66a006be30 0x7f66a006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:41.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.938+0000 7f66c3e00640 1 --2- 172.21.15.90:0/3773578587 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f66a006be30 0x7f66a006e2f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f66b400ba00 tx=0x7f66b402c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:41.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:41.941+0000 7f66bbe00640 1 -- 172.21.15.90:0/3773578587 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f66c4157090 con 0x7f66c4143060 2024-04-25T01:17:42.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.145+0000 7f66cb000640 1 -- 172.21.15.90:0/3773578587 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f66c415b3d0 con 0x7f66c4143060 2024-04-25T01:17:42.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.147+0000 7f66bbe00640 1 -- 172.21.15.90:0/3773578587 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18576 (secure 0 0 0) 0x7f66bc08e030 con 0x7f66c4143060 2024-04-25T01:17:42.131 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:17:42.131 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:17:09.346008+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:16:20.094512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:16:21.078268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:16:24.670642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.e","mappings":[{"from":0,"to":7}]},{"pgid":"4.f","mappings":[{"from":7,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:17:42.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.150+0000 7f66cb000640 1 -- 172.21.15.90:0/3773578587 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f66a006be30 msgr2=0x7f66a006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:42.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.150+0000 7f66cb000640 1 --2- 172.21.15.90:0/3773578587 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f66a006be30 0x7f66a006e2f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f66b400ba00 tx=0x7f66b402c000 comp rx=0 tx=0).stop 2024-04-25T01:17:42.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.150+0000 7f66cb000640 1 -- 172.21.15.90:0/3773578587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66c4143060 msgr2=0x7f66c41573b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:42.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.150+0000 7f66cb000640 1 --2- 172.21.15.90:0/3773578587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66c4143060 0x7f66c41573b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f66c40d0bc0 tx=0x7f66bc010c20 comp rx=0 tx=0).stop 2024-04-25T01:17:42.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.150+0000 7f66cb000640 1 -- 172.21.15.90:0/3773578587 shutdown_connections 2024-04-25T01:17:42.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.150+0000 7f66cb000640 1 --2- 172.21.15.90:0/3773578587 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f66a006be30 0x7f66a006e2f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:42.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.150+0000 7f66cb000640 1 --2- 172.21.15.90:0/3773578587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66c4143060 0x7f66c41573b0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:42.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.150+0000 7f66cb000640 1 --2- 172.21.15.90:0/3773578587 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66c40ad8e0 0x7f66c41426f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:42.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.150+0000 7f66cb000640 1 --2- 172.21.15.90:0/3773578587 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f66a4003aa0 0x7f66c4155340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:42.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.150+0000 7f66cb000640 1 -- 172.21.15.90:0/3773578587 >> 172.21.15.90:0/3773578587 conn(0x7f66c40b1820 msgr2=0x7f66c40711f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:42.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.150+0000 7f66cb000640 1 -- 172.21.15.90:0/3773578587 shutdown_connections 2024-04-25T01:17:42.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.151+0000 7f66cb000640 1 -- 172.21.15.90:0/3773578587 wait complete. 2024-04-25T01:17:42.153 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:16:15.894126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:16:20.094512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:16:21.078268+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:16:24.670642+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:17:42.153 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:17:42.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.373+0000 7f8275800640 1 Processor -- start 2024-04-25T01:17:42.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.374+0000 7f8275800640 1 -- start start 2024-04-25T01:17:42.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.375+0000 7f8275800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f827015fc40 0x7f82701600a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:42.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.375+0000 7f8275800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f82700cedc0 0x7f82700cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:42.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.375+0000 7f8275800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f827006d110 con 0x7f82700722e0 2024-04-25T01:17:42.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.375+0000 7f8275800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f8270160aa0 con 0x7f827015fc40 2024-04-25T01:17:42.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.375+0000 7f8275800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f827006da40 con 0x7f82700cedc0 2024-04-25T01:17:42.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.376+0000 7f826ea00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f82700722e0 0x7f827006ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:35222/0 (socket says 172.21.15.90:35222) 2024-04-25T01:17:42.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.376+0000 7f826ea00640 1 -- 172.21.15.90:0/1643817079 learned_addr learned my addr 172.21.15.90:0/1643817079 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:42.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.376+0000 7f826e000640 1 --2- 172.21.15.90:0/1643817079 >> v2:172.21.15.90:3300/0 conn(0x7f827015fc40 0x7f82701600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:42.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.376+0000 7f826d600640 1 -- 172.21.15.90:0/1643817079 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3875414901 0 0) 0x7f827006d110 con 0x7f82700722e0 2024-04-25T01:17:42.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.376+0000 7f826d600640 1 -- 172.21.15.90:0/1643817079 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8254003520 con 0x7f82700722e0 2024-04-25T01:17:42.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.376+0000 7f826e000640 1 -- 172.21.15.90:0/1643817079 >> v2:172.21.15.103:3300/0 conn(0x7f82700cedc0 msgr2=0x7f82700cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:42.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.376+0000 7f826e000640 1 --2- 172.21.15.90:0/1643817079 >> v2:172.21.15.103:3300/0 conn(0x7f82700cedc0 0x7f82700cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:42.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.376+0000 7f826e000640 1 -- 172.21.15.90:0/1643817079 >> v1:172.21.15.90:6789/0 conn(0x7f82700722e0 legacy=0x7f827006ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:42.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.376+0000 7f826e000640 1 -- 172.21.15.90:0/1643817079 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f827006d3f0 con 0x7f827015fc40 2024-04-25T01:17:42.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.377+0000 7f826e000640 1 --2- 172.21.15.90:0/1643817079 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f827015fc40 0x7f82701600a0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8270072920 tx=0x7f825c02a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa4b0585bd86fd9c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:42.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.378+0000 7f826d600640 1 -- 172.21.15.90:0/1643817079 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c007740 con 0x7f827015fc40 2024-04-25T01:17:42.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.378+0000 7f826d600640 1 -- 172.21.15.90:0/1643817079 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f825c0078e0 con 0x7f827015fc40 2024-04-25T01:17:42.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.378+0000 7f826d600640 1 -- 172.21.15.90:0/1643817079 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c007c90 con 0x7f827015fc40 2024-04-25T01:17:42.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.378+0000 7f8275800640 1 -- 172.21.15.90:0/1643817079 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f827015fc40 msgr2=0x7f82701600a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:42.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.378+0000 7f8275800640 1 --2- 172.21.15.90:0/1643817079 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f827015fc40 0x7f82701600a0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8270072920 tx=0x7f825c02a5f0 comp rx=0 tx=0).stop 2024-04-25T01:17:42.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.378+0000 7f8275800640 1 -- 172.21.15.90:0/1643817079 shutdown_connections 2024-04-25T01:17:42.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.378+0000 7f8275800640 1 --2- 172.21.15.90:0/1643817079 >> v2:172.21.15.103:3300/0 conn(0x7f82700cedc0 0x7f82700cf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:42.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.378+0000 7f8275800640 1 --2- 172.21.15.90:0/1643817079 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f827015fc40 0x7f82701600a0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:42.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.378+0000 7f8275800640 1 -- 172.21.15.90:0/1643817079 >> 172.21.15.90:0/1643817079 conn(0x7f827006b720 msgr2=0x7f82700b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:42.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.379+0000 7f8275800640 1 -- 172.21.15.90:0/1643817079 shutdown_connections 2024-04-25T01:17:42.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.379+0000 7f8275800640 1 -- 172.21.15.90:0/1643817079 wait complete. 2024-04-25T01:17:42.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.381+0000 7f8275800640 1 Processor -- start 2024-04-25T01:17:42.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.381+0000 7f8275800640 1 -- start start 2024-04-25T01:17:42.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.382+0000 7f8275800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f82700722e0 0x7f827015b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:42.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.382+0000 7f8275800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82700cedc0 0x7f827015b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:42.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.382+0000 7f8275800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f827015fc40 0x7f82701f76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:42.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.382+0000 7f8275800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f827006cec0 con 0x7f827015fc40 2024-04-25T01:17:42.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.382+0000 7f8275800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f82700cf730 con 0x7f82700cedc0 2024-04-25T01:17:42.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.382+0000 7f8275800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8270070b50 con 0x7f82700722e0 2024-04-25T01:17:42.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.382+0000 7f826f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f827015fc40 0x7f82701f76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:42.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.383+0000 7f826f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f827015fc40 0x7f82701f76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57194/0 (socket says 172.21.15.90:57194) 2024-04-25T01:17:42.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.383+0000 7f826f400640 1 -- 172.21.15.90:0/2257950564 learned_addr learned my addr 172.21.15.90:0/2257950564 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:42.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.383+0000 7f826f400640 1 -- 172.21.15.90:0/2257950564 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f82700722e0 msgr2=0x7f827015b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:42.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.383+0000 7f826ea00640 1 --2- 172.21.15.90:0/2257950564 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f82700722e0 0x7f827015b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:42.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.383+0000 7f826e000640 1 --2- 172.21.15.90:0/2257950564 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82700cedc0 0x7f827015b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:42.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.383+0000 7f826f400640 1 --2- 172.21.15.90:0/2257950564 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f82700722e0 0x7f827015b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:42.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.383+0000 7f826f400640 1 -- 172.21.15.90:0/2257950564 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82700cedc0 msgr2=0x7f827015b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:42.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.383+0000 7f826f400640 1 --2- 172.21.15.90:0/2257950564 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82700cedc0 0x7f827015b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:42.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.383+0000 7f826f400640 1 -- 172.21.15.90:0/2257950564 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82701539c0 con 0x7f827015fc40 2024-04-25T01:17:42.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.383+0000 7f826f400640 1 --2- 172.21.15.90:0/2257950564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f827015fc40 0x7f82701f76f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8268009fd0 tx=0x7f826800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:42.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.384+0000 7f8266a00640 1 -- 172.21.15.90:0/2257950564 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8268011830 con 0x7f827015fc40 2024-04-25T01:17:42.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.384+0000 7f8266a00640 1 -- 172.21.15.90:0/2257950564 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8268016070 con 0x7f827015fc40 2024-04-25T01:17:42.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.384+0000 7f8266a00640 1 -- 172.21.15.90:0/2257950564 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8268011bf0 con 0x7f827015fc40 2024-04-25T01:17:42.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.384+0000 7f8275800640 1 -- 172.21.15.90:0/2257950564 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82701f83e0 con 0x7f827015fc40 2024-04-25T01:17:42.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.384+0000 7f8275800640 1 -- 172.21.15.90:0/2257950564 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8270157830 con 0x7f827015fc40 2024-04-25T01:17:42.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.385+0000 7f8275800640 1 -- 172.21.15.90:0/2257950564 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8270068fb0 con 0x7f827015fc40 2024-04-25T01:17:42.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.386+0000 7f8266a00640 1 -- 172.21.15.90:0/2257950564 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f826801b460 con 0x7f827015fc40 2024-04-25T01:17:42.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.387+0000 7f8266a00640 1 --2- 172.21.15.90:0/2257950564 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f825006be30 0x7f825006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:42.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.387+0000 7f8266a00640 1 -- 172.21.15.90:0/2257950564 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f826808af90 con 0x7f827015fc40 2024-04-25T01:17:42.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.388+0000 7f826ea00640 1 --2- 172.21.15.90:0/2257950564 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f825006be30 0x7f825006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:42.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.388+0000 7f826ea00640 1 --2- 172.21.15.90:0/2257950564 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f825006be30 0x7f825006e2f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8260000f60 tx=0x7f8260009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:42.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.391+0000 7f8266a00640 1 -- 172.21.15.90:0/2257950564 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f82701539c0 con 0x7f827015fc40 2024-04-25T01:17:42.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.592+0000 7f8275800640 1 -- 172.21.15.90:0/2257950564 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f82700af910 con 0x7f827015fc40 2024-04-25T01:17:42.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.593+0000 7f8266a00640 1 -- 172.21.15.90:0/2257950564 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) ==== 93+0+10 (secure 0 0 0) 0x7f8268059530 con 0x7f827015fc40 2024-04-25T01:17:42.576 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T01:17:42.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.596+0000 7f8275800640 1 -- 172.21.15.90:0/2257950564 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f825006be30 msgr2=0x7f825006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:42.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.596+0000 7f8275800640 1 --2- 172.21.15.90:0/2257950564 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f825006be30 0x7f825006e2f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8260000f60 tx=0x7f8260009040 comp rx=0 tx=0).stop 2024-04-25T01:17:42.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.596+0000 7f8275800640 1 -- 172.21.15.90:0/2257950564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f827015fc40 msgr2=0x7f82701f76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:42.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.596+0000 7f8275800640 1 --2- 172.21.15.90:0/2257950564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f827015fc40 0x7f82701f76f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8268009fd0 tx=0x7f826800b8e0 comp rx=0 tx=0).stop 2024-04-25T01:17:42.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.596+0000 7f8275800640 1 -- 172.21.15.90:0/2257950564 shutdown_connections 2024-04-25T01:17:42.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.596+0000 7f8275800640 1 --2- 172.21.15.90:0/2257950564 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f825006be30 0x7f825006e2f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:42.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.596+0000 7f8275800640 1 --2- 172.21.15.90:0/2257950564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f827015fc40 0x7f82701f76f0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:42.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.596+0000 7f8275800640 1 --2- 172.21.15.90:0/2257950564 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82700cedc0 0x7f827015b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:42.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.596+0000 7f8275800640 1 --2- 172.21.15.90:0/2257950564 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f82700722e0 0x7f827015b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:42.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.596+0000 7f8275800640 1 -- 172.21.15.90:0/2257950564 >> 172.21.15.90:0/2257950564 conn(0x7f827006b720 msgr2=0x7f82700ae860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:42.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.596+0000 7f8275800640 1 -- 172.21.15.90:0/2257950564 shutdown_connections 2024-04-25T01:17:42.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.596+0000 7f8275800640 1 -- 172.21.15.90:0/2257950564 wait complete. 2024-04-25T01:17:42.599 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:17:42.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.803+0000 7fc9dcc00640 1 Processor -- start 2024-04-25T01:17:42.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.803+0000 7fc9dcc00640 1 -- start start 2024-04-25T01:17:42.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.805+0000 7fc9dcc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc9d80722e0 0x7fc9d806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:42.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.805+0000 7fc9dcc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9d80cf4d0 0x7fc9d81600c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:42.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.805+0000 7fc9dcc00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9d806e410 con 0x7fc9d815fc40 2024-04-25T01:17:42.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.805+0000 7fc9dcc00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fc9d8160aa0 con 0x7fc9d80722e0 2024-04-25T01:17:42.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.805+0000 7fc9dcc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc9d806da40 con 0x7fc9d80cf4d0 2024-04-25T01:17:42.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.805+0000 7fc9d5e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc9d80722e0 0x7fc9d806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:42.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.806+0000 7fc9d5e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc9d80722e0 0x7fc9d806ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57202/0 (socket says 172.21.15.90:57202) 2024-04-25T01:17:42.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.806+0000 7fc9d5e00640 1 -- 172.21.15.90:0/3138583216 learned_addr learned my addr 172.21.15.90:0/3138583216 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:42.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.806+0000 7fc9d5e00640 1 -- 172.21.15.90:0/3138583216 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9d80cf4d0 msgr2=0x7fc9d81600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:42.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.806+0000 7fc9d6800640 1 --2- 172.21.15.90:0/3138583216 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9d80cf4d0 0x7fc9d81600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:42.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.806+0000 7fc9d5e00640 1 --2- 172.21.15.90:0/3138583216 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9d80cf4d0 0x7fc9d81600c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:42.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.806+0000 7fc9d5e00640 1 -- 172.21.15.90:0/3138583216 >> v1:172.21.15.103:6789/0 conn(0x7fc9d815fc40 legacy=0x7fc9d80cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:42.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.806+0000 7fc9d5e00640 1 -- 172.21.15.90:0/3138583216 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d806d110 con 0x7fc9d80722e0 2024-04-25T01:17:42.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.806+0000 7fc9d6800640 1 --2- 172.21.15.90:0/3138583216 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9d80cf4d0 0x7fc9d81600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:42.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.807+0000 7fc9d5e00640 1 --2- 172.21.15.90:0/3138583216 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9d80722e0 0x7fc9d806ded0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8009870 tx=0x7fc9c802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad7c53f244e7d918 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:42.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.808+0000 7fc9d4a00640 1 -- 172.21.15.90:0/3138583216 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c8037070 con 0x7fc9d80722e0 2024-04-25T01:17:42.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.808+0000 7fc9d4a00640 1 -- 172.21.15.90:0/3138583216 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9c8032440 con 0x7fc9d80722e0 2024-04-25T01:17:42.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.808+0000 7fc9d4a00640 1 -- 172.21.15.90:0/3138583216 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c8032810 con 0x7fc9d80722e0 2024-04-25T01:17:42.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.808+0000 7fc9dcc00640 1 -- 172.21.15.90:0/3138583216 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9d80722e0 msgr2=0x7fc9d806ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:42.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.808+0000 7fc9dcc00640 1 --2- 172.21.15.90:0/3138583216 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9d80722e0 0x7fc9d806ded0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8009870 tx=0x7fc9c802a180 comp rx=0 tx=0).stop 2024-04-25T01:17:42.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.809+0000 7fc9dcc00640 1 -- 172.21.15.90:0/3138583216 shutdown_connections 2024-04-25T01:17:42.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.809+0000 7fc9dcc00640 1 --2- 172.21.15.90:0/3138583216 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9d80cf4d0 0x7fc9d81600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:42.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.809+0000 7fc9dcc00640 1 --2- 172.21.15.90:0/3138583216 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9d80722e0 0x7fc9d806ded0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:42.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.809+0000 7fc9dcc00640 1 -- 172.21.15.90:0/3138583216 >> 172.21.15.90:0/3138583216 conn(0x7fc9d806b720 msgr2=0x7fc9d80b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:42.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.809+0000 7fc9dcc00640 1 -- 172.21.15.90:0/3138583216 shutdown_connections 2024-04-25T01:17:42.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.809+0000 7fc9dcc00640 1 -- 172.21.15.90:0/3138583216 wait complete. 2024-04-25T01:17:42.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.810+0000 7fc9dcc00640 1 Processor -- start 2024-04-25T01:17:42.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.811+0000 7fc9dcc00640 1 -- start start 2024-04-25T01:17:42.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.811+0000 7fc9dcc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9d80722e0 0x7fc9d8141370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:42.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.811+0000 7fc9dcc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9d80cf4d0 0x7fc9d80aed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:42.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.812+0000 7fc9dcc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9d815fc40 0x7fc9d815d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:42.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.812+0000 7fc9dcc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d806d6d0 con 0x7fc9d80722e0 2024-04-25T01:17:42.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.812+0000 7fc9dcc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d806cec0 con 0x7fc9d80cf4d0 2024-04-25T01:17:42.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.812+0000 7fc9dcc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc9d8070b50 con 0x7fc9d815fc40 2024-04-25T01:17:42.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.812+0000 7fc9d5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9d80722e0 0x7fc9d8141370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:42.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.812+0000 7fc9d5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9d80722e0 0x7fc9d8141370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57212/0 (socket says 172.21.15.90:57212) 2024-04-25T01:17:42.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.812+0000 7fc9d5e00640 1 -- 172.21.15.90:0/1873392400 learned_addr learned my addr 172.21.15.90:0/1873392400 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:42.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.812+0000 7fc9d5e00640 1 -- 172.21.15.90:0/1873392400 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9d815fc40 msgr2=0x7fc9d815d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:42.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.812+0000 7fc9d5e00640 1 --2- 172.21.15.90:0/1873392400 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9d815fc40 0x7fc9d815d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:42.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.812+0000 7fc9d5e00640 1 -- 172.21.15.90:0/1873392400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9d80cf4d0 msgr2=0x7fc9d80aed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:42.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.812+0000 7fc9d5e00640 1 --2- 172.21.15.90:0/1873392400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9d80cf4d0 0x7fc9d80aed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:42.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.812+0000 7fc9d5e00640 1 -- 172.21.15.90:0/1873392400 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d80af250 con 0x7fc9d80722e0 2024-04-25T01:17:42.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.812+0000 7fc9d5e00640 1 --2- 172.21.15.90:0/1873392400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9d80722e0 0x7fc9d8141370 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8000c00 tx=0x7fc9c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:42.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.813+0000 7fc9ce000640 1 -- 172.21.15.90:0/1873392400 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c8037040 con 0x7fc9d80722e0 2024-04-25T01:17:42.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.813+0000 7fc9ce000640 1 -- 172.21.15.90:0/1873392400 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9c8042070 con 0x7fc9d80722e0 2024-04-25T01:17:42.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.813+0000 7fc9ce000640 1 -- 172.21.15.90:0/1873392400 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c8030bb0 con 0x7fc9d80722e0 2024-04-25T01:17:42.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.813+0000 7fc9dcc00640 1 -- 172.21.15.90:0/1873392400 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9d8158630 con 0x7fc9d80722e0 2024-04-25T01:17:42.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.813+0000 7fc9dcc00640 1 -- 172.21.15.90:0/1873392400 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9d8158bf0 con 0x7fc9d80722e0 2024-04-25T01:17:42.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.814+0000 7fc9dcc00640 1 -- 172.21.15.90:0/1873392400 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9d8068fb0 con 0x7fc9d80722e0 2024-04-25T01:17:42.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.815+0000 7fc9ce000640 1 -- 172.21.15.90:0/1873392400 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc9c8047460 con 0x7fc9d80722e0 2024-04-25T01:17:42.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.816+0000 7fc9ce000640 1 --2- 172.21.15.90:0/1873392400 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc9b806be60 0x7fc9b806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:42.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.816+0000 7fc9ce000640 1 -- 172.21.15.90:0/1873392400 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7fc9c80abe20 con 0x7fc9d80722e0 2024-04-25T01:17:42.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.816+0000 7fc9d5400640 1 --2- 172.21.15.90:0/1873392400 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc9b806be60 0x7fc9b806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:42.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.817+0000 7fc9d5400640 1 --2- 172.21.15.90:0/1873392400 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc9b806be60 0x7fc9b806e320 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4006fd0 tx=0x7fc9c4006bc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:42.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:42.821+0000 7fc9ce000640 1 -- 172.21.15.90:0/1873392400 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc9c807a450 con 0x7fc9d80722e0 2024-04-25T01:17:43.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.019+0000 7fc9dcc00640 1 -- 172.21.15.90:0/1873392400 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc9d806e190 con 0x7fc9d80722e0 2024-04-25T01:17:43.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.020+0000 7fc9ce000640 1 -- 172.21.15.90:0/1873392400 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) ==== 104+0+11 (secure 0 0 0) 0x7fc9c8079d50 con 0x7fc9d80722e0 2024-04-25T01:17:43.003 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:17:43.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.022+0000 7fc9dcc00640 1 -- 172.21.15.90:0/1873392400 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc9b806be60 msgr2=0x7fc9b806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:43.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.022+0000 7fc9dcc00640 1 --2- 172.21.15.90:0/1873392400 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc9b806be60 0x7fc9b806e320 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4006fd0 tx=0x7fc9c4006bc0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.022+0000 7fc9dcc00640 1 -- 172.21.15.90:0/1873392400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9d80722e0 msgr2=0x7fc9d8141370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:43.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.022+0000 7fc9dcc00640 1 --2- 172.21.15.90:0/1873392400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9d80722e0 0x7fc9d8141370 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8000c00 tx=0x7fc9c80046c0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.022+0000 7fc9dcc00640 1 -- 172.21.15.90:0/1873392400 shutdown_connections 2024-04-25T01:17:43.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.022+0000 7fc9dcc00640 1 --2- 172.21.15.90:0/1873392400 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc9b806be60 0x7fc9b806e320 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.022+0000 7fc9dcc00640 1 --2- 172.21.15.90:0/1873392400 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc9d815fc40 0x7fc9d815d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.022+0000 7fc9dcc00640 1 --2- 172.21.15.90:0/1873392400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9d80cf4d0 0x7fc9d80aed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.022+0000 7fc9dcc00640 1 --2- 172.21.15.90:0/1873392400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9d80722e0 0x7fc9d8141370 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.022+0000 7fc9dcc00640 1 -- 172.21.15.90:0/1873392400 >> 172.21.15.90:0/1873392400 conn(0x7fc9d806b720 msgr2=0x7fc9d815ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:43.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.022+0000 7fc9dcc00640 1 -- 172.21.15.90:0/1873392400 shutdown_connections 2024-04-25T01:17:43.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.022+0000 7fc9dcc00640 1 -- 172.21.15.90:0/1873392400 wait complete. 2024-04-25T01:17:43.025 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:17:43.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.245+0000 7ff5dbe00640 1 Processor -- start 2024-04-25T01:17:43.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.245+0000 7ff5dbe00640 1 -- start start 2024-04-25T01:17:43.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.246+0000 7ff5dbe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5d4158680 0x7ff5d406df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:43.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.246+0000 7ff5dbe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff5d4158a60 0x7ff5d406e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:43.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.246+0000 7ff5dbe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5d406d140 con 0x7ff5d415f4c0 2024-04-25T01:17:43.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.246+0000 7ff5dbe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff5d406d420 con 0x7ff5d4158a60 2024-04-25T01:17:43.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.246+0000 7ff5dbe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff5d406d700 con 0x7ff5d4158680 2024-04-25T01:17:43.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.247+0000 7ff5d9400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5d4158680 0x7ff5d406df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:43.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.247+0000 7ff5d8a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff5d4158a60 0x7ff5d406e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:43.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.247+0000 7ff5d9400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5d4158680 0x7ff5d406df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.90:52032/0 (socket says 172.21.15.90:52032) 2024-04-25T01:17:43.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.247+0000 7ff5d9400640 1 -- 172.21.15.90:0/1662706128 learned_addr learned my addr 172.21.15.90:0/1662706128 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:43.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.248+0000 7ff5d9400640 1 -- 172.21.15.90:0/1662706128 >> v2:172.21.15.103:3300/0 conn(0x7ff5d4158a60 msgr2=0x7ff5d406e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:43.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.248+0000 7ff5d9400640 1 --2- 172.21.15.90:0/1662706128 >> v2:172.21.15.103:3300/0 conn(0x7ff5d4158a60 0x7ff5d406e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.248+0000 7ff5d9400640 1 -- 172.21.15.90:0/1662706128 >> v1:172.21.15.103:6789/0 conn(0x7ff5d415f4c0 legacy=0x7ff5d40b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:43.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.248+0000 7ff5d9400640 1 -- 172.21.15.90:0/1662706128 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5d406d9e0 con 0x7ff5d4158680 2024-04-25T01:17:43.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.249+0000 7ff5d9400640 1 --2- 172.21.15.90:0/1662706128 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5d4158680 0x7ff5d406df00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff5c8009870 tx=0x7ff5c802a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b70c03ba0f8b055 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:43.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.250+0000 7ff5d3e00640 1 -- 172.21.15.90:0/1662706128 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c8037070 con 0x7ff5d4158680 2024-04-25T01:17:43.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.250+0000 7ff5d3e00640 1 -- 172.21.15.90:0/1662706128 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5c8032440 con 0x7ff5d4158680 2024-04-25T01:17:43.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.250+0000 7ff5d3e00640 1 -- 172.21.15.90:0/1662706128 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c8032810 con 0x7ff5d4158680 2024-04-25T01:17:43.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.251+0000 7ff5dbe00640 1 -- 172.21.15.90:0/1662706128 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5d4158680 msgr2=0x7ff5d406df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:43.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.251+0000 7ff5dbe00640 1 --2- 172.21.15.90:0/1662706128 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5d4158680 0x7ff5d406df00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff5c8009870 tx=0x7ff5c802a1c0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.251+0000 7ff5dbe00640 1 -- 172.21.15.90:0/1662706128 shutdown_connections 2024-04-25T01:17:43.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.251+0000 7ff5dbe00640 1 --2- 172.21.15.90:0/1662706128 >> v2:172.21.15.103:3300/0 conn(0x7ff5d4158a60 0x7ff5d406e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.251+0000 7ff5dbe00640 1 --2- 172.21.15.90:0/1662706128 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5d4158680 0x7ff5d406df00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.251+0000 7ff5dbe00640 1 -- 172.21.15.90:0/1662706128 >> 172.21.15.90:0/1662706128 conn(0x7ff5d406b720 msgr2=0x7ff5d40b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:43.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.251+0000 7ff5dbe00640 1 -- 172.21.15.90:0/1662706128 shutdown_connections 2024-04-25T01:17:43.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.251+0000 7ff5dbe00640 1 -- 172.21.15.90:0/1662706128 wait complete. 2024-04-25T01:17:43.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.253+0000 7ff5dbe00640 1 Processor -- start 2024-04-25T01:17:43.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.253+0000 7ff5dbe00640 1 -- start start 2024-04-25T01:17:43.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.254+0000 7ff5dbe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5d4158680 0x7ff5d41515d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:43.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.254+0000 7ff5dbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5d4158a60 0x7ff5d413c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:43.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.254+0000 7ff5dbe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5d415f4c0 0x7ff5d41e73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:43.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.254+0000 7ff5dbe00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d406edb0 con 0x7ff5d4158a60 2024-04-25T01:17:43.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.254+0000 7ff5dbe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d406ec30 con 0x7ff5d415f4c0 2024-04-25T01:17:43.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.254+0000 7ff5dbe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff5d40b3c60 con 0x7ff5d4158680 2024-04-25T01:17:43.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.254+0000 7ff5d8a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5d4158a60 0x7ff5d413c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:43.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.254+0000 7ff5d8a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5d4158a60 0x7ff5d413c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57220/0 (socket says 172.21.15.90:57220) 2024-04-25T01:17:43.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.254+0000 7ff5d8a00640 1 -- 172.21.15.90:0/491808823 learned_addr learned my addr 172.21.15.90:0/491808823 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:43.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.255+0000 7ff5d9400640 1 --2- 172.21.15.90:0/491808823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5d4158680 0x7ff5d41515d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:43.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.255+0000 7ff5d9e00640 1 --2- 172.21.15.90:0/491808823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5d415f4c0 0x7ff5d41e73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:43.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.255+0000 7ff5d8a00640 1 -- 172.21.15.90:0/491808823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5d4158680 msgr2=0x7ff5d41515d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:43.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.255+0000 7ff5d8a00640 1 --2- 172.21.15.90:0/491808823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5d4158680 0x7ff5d41515d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.255+0000 7ff5d8a00640 1 -- 172.21.15.90:0/491808823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5d415f4c0 msgr2=0x7ff5d41e73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:43.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.255+0000 7ff5d8a00640 1 --2- 172.21.15.90:0/491808823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5d415f4c0 0x7ff5d41e73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.255+0000 7ff5d8a00640 1 -- 172.21.15.90:0/491808823 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5d413cc20 con 0x7ff5d4158a60 2024-04-25T01:17:43.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.255+0000 7ff5d8a00640 1 --2- 172.21.15.90:0/491808823 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5d4158a60 0x7ff5d413c6e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff5c000e870 tx=0x7ff5c000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:43.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.255+0000 7ff5d1600640 1 -- 172.21.15.90:0/491808823 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c0017070 con 0x7ff5d4158a60 2024-04-25T01:17:43.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.255+0000 7ff5d1600640 1 -- 172.21.15.90:0/491808823 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5c000c940 con 0x7ff5d4158a60 2024-04-25T01:17:43.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.255+0000 7ff5dbe00640 1 -- 172.21.15.90:0/491808823 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5d4140430 con 0x7ff5d4158a60 2024-04-25T01:17:43.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.255+0000 7ff5d1600640 1 -- 172.21.15.90:0/491808823 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c000cd10 con 0x7ff5d4158a60 2024-04-25T01:17:43.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.256+0000 7ff5dbe00640 1 -- 172.21.15.90:0/491808823 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5d41409f0 con 0x7ff5d4158a60 2024-04-25T01:17:43.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.257+0000 7ff5dbe00640 1 -- 172.21.15.90:0/491808823 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5d4068fb0 con 0x7ff5d4158a60 2024-04-25T01:17:43.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.258+0000 7ff5d1600640 1 -- 172.21.15.90:0/491808823 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff5c001a070 con 0x7ff5d4158a60 2024-04-25T01:17:43.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.259+0000 7ff5d1600640 1 --2- 172.21.15.90:0/491808823 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff5b406be60 0x7ff5b406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:43.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.259+0000 7ff5d1600640 1 -- 172.21.15.90:0/491808823 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7ff5c008b080 con 0x7ff5d4158a60 2024-04-25T01:17:43.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.259+0000 7ff5d9400640 1 --2- 172.21.15.90:0/491808823 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff5b406be60 0x7ff5b406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:43.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.260+0000 7ff5d9400640 1 --2- 172.21.15.90:0/491808823 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff5b406be60 0x7ff5b406e320 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff5c8034000 tx=0x7ff5c802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:43.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.263+0000 7ff5d1600640 1 -- 172.21.15.90:0/491808823 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff5c0059620 con 0x7ff5d4158a60 2024-04-25T01:17:43.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.461+0000 7ff5dbe00640 1 -- 172.21.15.90:0/491808823 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff5d40af750 con 0x7ff5d4158a60 2024-04-25T01:17:43.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.461+0000 7ff5d1600640 1 -- 172.21.15.90:0/491808823 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) ==== 100+0+11 (secure 0 0 0) 0x7ff5c0058f20 con 0x7ff5d4158a60 2024-04-25T01:17:43.445 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-25T01:17:43.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.464+0000 7ff5dbe00640 1 -- 172.21.15.90:0/491808823 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff5b406be60 msgr2=0x7ff5b406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:43.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.464+0000 7ff5dbe00640 1 --2- 172.21.15.90:0/491808823 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff5b406be60 0x7ff5b406e320 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff5c8034000 tx=0x7ff5c802e000 comp rx=0 tx=0).stop 2024-04-25T01:17:43.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.464+0000 7ff5dbe00640 1 -- 172.21.15.90:0/491808823 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5d4158a60 msgr2=0x7ff5d413c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:43.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.464+0000 7ff5dbe00640 1 --2- 172.21.15.90:0/491808823 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5d4158a60 0x7ff5d413c6e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff5c000e870 tx=0x7ff5c000ed30 comp rx=0 tx=0).stop 2024-04-25T01:17:43.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.464+0000 7ff5dbe00640 1 -- 172.21.15.90:0/491808823 shutdown_connections 2024-04-25T01:17:43.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.464+0000 7ff5dbe00640 1 --2- 172.21.15.90:0/491808823 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff5b406be60 0x7ff5b406e320 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.464+0000 7ff5dbe00640 1 --2- 172.21.15.90:0/491808823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5d415f4c0 0x7ff5d41e73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.464+0000 7ff5dbe00640 1 --2- 172.21.15.90:0/491808823 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5d4158a60 0x7ff5d413c6e0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.464+0000 7ff5dbe00640 1 --2- 172.21.15.90:0/491808823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff5d4158680 0x7ff5d41515d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.464+0000 7ff5dbe00640 1 -- 172.21.15.90:0/491808823 >> 172.21.15.90:0/491808823 conn(0x7ff5d406b720 msgr2=0x7ff5d40afdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:43.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.464+0000 7ff5dbe00640 1 -- 172.21.15.90:0/491808823 shutdown_connections 2024-04-25T01:17:43.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.464+0000 7ff5dbe00640 1 -- 172.21.15.90:0/491808823 wait complete. 2024-04-25T01:17:43.466 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:17:43.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.684+0000 7f94f1600640 1 Processor -- start 2024-04-25T01:17:43.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.684+0000 7f94f1600640 1 -- start start 2024-04-25T01:17:43.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.686+0000 7f94f1600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f94ec0722e0 0x7f94ec06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:43.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.686+0000 7f94f1600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f94ec15fc40 0x7f94ec1600c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:43.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.686+0000 7f94f1600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94ec0cedc0 0x7f94ec0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:43.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.686+0000 7f94f1600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f94ec160aa0 con 0x7f94ec15fc40 2024-04-25T01:17:43.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.686+0000 7f94f1600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f94ec06da40 con 0x7f94ec0722e0 2024-04-25T01:17:43.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.686+0000 7f94f1600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f94ec06cec0 con 0x7f94ec0cedc0 2024-04-25T01:17:43.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.686+0000 7f94ea000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f94ec15fc40 0x7f94ec1600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:43.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.686+0000 7f94ea000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f94ec15fc40 0x7f94ec1600c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57234/0 (socket says 172.21.15.90:57234) 2024-04-25T01:17:43.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.686+0000 7f94ea000640 1 -- 172.21.15.90:0/1566734877 learned_addr learned my addr 172.21.15.90:0/1566734877 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:43.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.686+0000 7f94eaa00640 1 --2- 172.21.15.90:0/1566734877 >> v2:172.21.15.103:3300/0 conn(0x7f94ec0722e0 0x7f94ec06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:43.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.686+0000 7f94eb400640 1 --2- 172.21.15.90:0/1566734877 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94ec0cedc0 0x7f94ec0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:43.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.687+0000 7f94ea000640 1 -- 172.21.15.90:0/1566734877 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94ec0cedc0 msgr2=0x7f94ec0cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:43.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.687+0000 7f94ea000640 1 --2- 172.21.15.90:0/1566734877 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94ec0cedc0 0x7f94ec0cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.687+0000 7f94ea000640 1 -- 172.21.15.90:0/1566734877 >> v2:172.21.15.103:3300/0 conn(0x7f94ec0722e0 msgr2=0x7f94ec06ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:43.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.687+0000 7f94ea000640 1 --2- 172.21.15.90:0/1566734877 >> v2:172.21.15.103:3300/0 conn(0x7f94ec0722e0 0x7f94ec06ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.687+0000 7f94ea000640 1 -- 172.21.15.90:0/1566734877 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94ec06e410 con 0x7f94ec15fc40 2024-04-25T01:17:43.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.687+0000 7f94eaa00640 1 --2- 172.21.15.90:0/1566734877 >> v2:172.21.15.103:3300/0 conn(0x7f94ec0722e0 0x7f94ec06ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:43.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.687+0000 7f94eb400640 1 --2- 172.21.15.90:0/1566734877 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94ec0cedc0 0x7f94ec0cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:43.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.688+0000 7f94ea000640 1 --2- 172.21.15.90:0/1566734877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94ec15fc40 0x7f94ec1600c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f94ec0727b0 tx=0x7f94d802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dccafe03b2cd3be server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:43.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.688+0000 7f94e9600640 1 -- 172.21.15.90:0/1566734877 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d8037070 con 0x7f94ec15fc40 2024-04-25T01:17:43.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.688+0000 7f94e9600640 1 -- 172.21.15.90:0/1566734877 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94d8032440 con 0x7f94ec15fc40 2024-04-25T01:17:43.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.688+0000 7f94e9600640 1 -- 172.21.15.90:0/1566734877 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d8032810 con 0x7f94ec15fc40 2024-04-25T01:17:43.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.689+0000 7f94f1600640 1 -- 172.21.15.90:0/1566734877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94ec15fc40 msgr2=0x7f94ec1600c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:43.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.689+0000 7f94f1600640 1 --2- 172.21.15.90:0/1566734877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94ec15fc40 0x7f94ec1600c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f94ec0727b0 tx=0x7f94d802a140 comp rx=0 tx=0).stop 2024-04-25T01:17:43.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.689+0000 7f94f1600640 1 -- 172.21.15.90:0/1566734877 shutdown_connections 2024-04-25T01:17:43.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.689+0000 7f94f1600640 1 --2- 172.21.15.90:0/1566734877 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94ec0cedc0 0x7f94ec0cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.689+0000 7f94f1600640 1 --2- 172.21.15.90:0/1566734877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94ec15fc40 0x7f94ec1600c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.689+0000 7f94f1600640 1 --2- 172.21.15.90:0/1566734877 >> v2:172.21.15.103:3300/0 conn(0x7f94ec0722e0 0x7f94ec06ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.689+0000 7f94f1600640 1 -- 172.21.15.90:0/1566734877 >> 172.21.15.90:0/1566734877 conn(0x7f94ec06b720 msgr2=0x7f94ec0b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:43.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.689+0000 7f94f1600640 1 -- 172.21.15.90:0/1566734877 shutdown_connections 2024-04-25T01:17:43.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.689+0000 7f94f1600640 1 -- 172.21.15.90:0/1566734877 wait complete. 2024-04-25T01:17:43.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.691+0000 7f94f1600640 1 Processor -- start 2024-04-25T01:17:43.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.691+0000 7f94f1600640 1 -- start start 2024-04-25T01:17:43.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.692+0000 7f94f1600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94ec0722e0 0x7f94ec158bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:43.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.692+0000 7f94f1600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94ec0cedc0 0x7f94ec14c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:43.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.692+0000 7f94f1600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94ec15fc40 0x7f94ec0adb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:43.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.692+0000 7f94f1600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f94ec06d3f0 con 0x7f94ec15fc40 2024-04-25T01:17:43.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.692+0000 7f94f1600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f94ec0cf770 con 0x7f94ec0cedc0 2024-04-25T01:17:43.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.692+0000 7f94f1600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f94ec06d6d0 con 0x7f94ec0722e0 2024-04-25T01:17:43.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.692+0000 7f94eb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94ec15fc40 0x7f94ec0adb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:43.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.692+0000 7f94eb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94ec15fc40 0x7f94ec0adb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57248/0 (socket says 172.21.15.90:57248) 2024-04-25T01:17:43.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.692+0000 7f94eb400640 1 -- 172.21.15.90:0/2664046881 learned_addr learned my addr 172.21.15.90:0/2664046881 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:43.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.692+0000 7f94eaa00640 1 --2- 172.21.15.90:0/2664046881 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94ec0722e0 0x7f94ec158bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:43.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.692+0000 7f94ea000640 1 --2- 172.21.15.90:0/2664046881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94ec0cedc0 0x7f94ec14c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:43.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.693+0000 7f94eb400640 1 -- 172.21.15.90:0/2664046881 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94ec0722e0 msgr2=0x7f94ec158bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:43.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.693+0000 7f94eb400640 1 --2- 172.21.15.90:0/2664046881 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94ec0722e0 0x7f94ec158bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.693+0000 7f94eb400640 1 -- 172.21.15.90:0/2664046881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94ec0cedc0 msgr2=0x7f94ec14c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:43.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.693+0000 7f94eb400640 1 --2- 172.21.15.90:0/2664046881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94ec0cedc0 0x7f94ec14c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.693+0000 7f94eb400640 1 -- 172.21.15.90:0/2664046881 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94ec14cb70 con 0x7f94ec15fc40 2024-04-25T01:17:43.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.693+0000 7f94eb400640 1 --2- 172.21.15.90:0/2664046881 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94ec15fc40 0x7f94ec0adb80 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f94dc00cd80 tx=0x7f94dc00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:43.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.693+0000 7f94e2a00640 1 -- 172.21.15.90:0/2664046881 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94dc00f680 con 0x7f94ec15fc40 2024-04-25T01:17:43.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.693+0000 7f94e2a00640 1 -- 172.21.15.90:0/2664046881 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94dc00f840 con 0x7f94ec15fc40 2024-04-25T01:17:43.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.693+0000 7f94e2a00640 1 -- 172.21.15.90:0/2664046881 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94dc00fc30 con 0x7f94ec15fc40 2024-04-25T01:17:43.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.693+0000 7f94f1600640 1 -- 172.21.15.90:0/2664046881 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94ec156f60 con 0x7f94ec15fc40 2024-04-25T01:17:43.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.693+0000 7f94f1600640 1 -- 172.21.15.90:0/2664046881 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94ec157520 con 0x7f94ec15fc40 2024-04-25T01:17:43.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.695+0000 7f94f1600640 1 -- 172.21.15.90:0/2664046881 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94ec06cc00 con 0x7f94ec15fc40 2024-04-25T01:17:43.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.695+0000 7f94e2a00640 1 -- 172.21.15.90:0/2664046881 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f94dc0213f0 con 0x7f94ec15fc40 2024-04-25T01:17:43.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.696+0000 7f94e2a00640 1 --2- 172.21.15.90:0/2664046881 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f94cc06be30 0x7f94cc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:43.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.696+0000 7f94e2a00640 1 -- 172.21.15.90:0/2664046881 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f94dc08bf80 con 0x7f94ec15fc40 2024-04-25T01:17:43.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.697+0000 7f94eaa00640 1 --2- 172.21.15.90:0/2664046881 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f94cc06be30 0x7f94cc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:43.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.697+0000 7f94eaa00640 1 --2- 172.21.15.90:0/2664046881 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f94cc06be30 0x7f94cc06e2f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f94ec072920 tx=0x7f94e4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:43.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.701+0000 7f94e2a00640 1 -- 172.21.15.90:0/2664046881 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f94ec06cc00 con 0x7f94ec15fc40 2024-04-25T01:17:43.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.898+0000 7f94f1600640 1 -- 172.21.15.90:0/2664046881 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f94ec068fb0 con 0x7f94ec15fc40 2024-04-25T01:17:43.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.899+0000 7f94e2a00640 1 -- 172.21.15.90:0/2664046881 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) ==== 103+0+11 (secure 0 0 0) 0x7f94dc014030 con 0x7f94ec15fc40 2024-04-25T01:17:43.882 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:17:43.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.901+0000 7f94f1600640 1 -- 172.21.15.90:0/2664046881 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f94cc06be30 msgr2=0x7f94cc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:43.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.901+0000 7f94f1600640 1 --2- 172.21.15.90:0/2664046881 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f94cc06be30 0x7f94cc06e2f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f94ec072920 tx=0x7f94e4008040 comp rx=0 tx=0).stop 2024-04-25T01:17:43.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.901+0000 7f94f1600640 1 -- 172.21.15.90:0/2664046881 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94ec15fc40 msgr2=0x7f94ec0adb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:43.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.901+0000 7f94f1600640 1 --2- 172.21.15.90:0/2664046881 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94ec15fc40 0x7f94ec0adb80 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f94dc00cd80 tx=0x7f94dc00a650 comp rx=0 tx=0).stop 2024-04-25T01:17:43.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.902+0000 7f94f1600640 1 -- 172.21.15.90:0/2664046881 shutdown_connections 2024-04-25T01:17:43.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.902+0000 7f94f1600640 1 --2- 172.21.15.90:0/2664046881 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f94cc06be30 0x7f94cc06e2f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.902+0000 7f94f1600640 1 --2- 172.21.15.90:0/2664046881 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94ec15fc40 0x7f94ec0adb80 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.902+0000 7f94f1600640 1 --2- 172.21.15.90:0/2664046881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94ec0cedc0 0x7f94ec14c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.902+0000 7f94f1600640 1 --2- 172.21.15.90:0/2664046881 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94ec0722e0 0x7f94ec158bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:43.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.902+0000 7f94f1600640 1 -- 172.21.15.90:0/2664046881 >> 172.21.15.90:0/2664046881 conn(0x7f94ec06b720 msgr2=0x7f94ec0b1040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:43.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.902+0000 7f94f1600640 1 -- 172.21.15.90:0/2664046881 shutdown_connections 2024-04-25T01:17:43.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:43.902+0000 7f94f1600640 1 -- 172.21.15.90:0/2664046881 wait complete. 2024-04-25T01:17:43.905 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T01:17:44.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.119+0000 7fe033200640 1 Processor -- start 2024-04-25T01:17:44.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.119+0000 7fe033200640 1 -- start start 2024-04-25T01:17:44.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.121+0000 7fe033200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe03406b830 0x7fe0340ae730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:44.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.121+0000 7fe033200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0340af400 con 0x7fe0341633d0 2024-04-25T01:17:44.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.121+0000 7fe033200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0340aee40 con 0x7fe03415c8a0 2024-04-25T01:17:44.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.121+0000 7fe033200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fe0340cc800 con 0x7fe03406b830 2024-04-25T01:17:44.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.121+0000 7fe032800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe03406b830 0x7fe0340ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:44.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.121+0000 7fe032800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe03406b830 0x7fe0340ae730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57252/0 (socket says 172.21.15.90:57252) 2024-04-25T01:17:44.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.121+0000 7fe032800640 1 -- 172.21.15.90:0/1807132957 learned_addr learned my addr 172.21.15.90:0/1807132957 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:44.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.122+0000 7fe032800640 1 -- 172.21.15.90:0/1807132957 >> v1:172.21.15.103:6789/0 conn(0x7fe03415c8a0 legacy=0x7fe0340ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:44.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.122+0000 7fe032800640 1 -- 172.21.15.90:0/1807132957 >> v1:172.21.15.90:6789/0 conn(0x7fe0341633d0 legacy=0x7fe03406b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:44.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.122+0000 7fe032800640 1 -- 172.21.15.90:0/1807132957 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0340af120 con 0x7fe03406b830 2024-04-25T01:17:44.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.122+0000 7fe030a00640 1 -- 172.21.15.90:0/1807132957 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1242661410 0 0) 0x7fe0340af400 con 0x7fe0341633d0 2024-04-25T01:17:44.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.123+0000 7fe032800640 1 --2- 172.21.15.90:0/1807132957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe03406b830 0x7fe0340ae730 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe02c009870 tx=0x7fe02c02a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=590ede18d9f4602d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:44.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.123+0000 7fe030a00640 1 -- 172.21.15.90:0/1807132957 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe02c037070 con 0x7fe03406b830 2024-04-25T01:17:44.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.123+0000 7fe030a00640 1 -- 172.21.15.90:0/1807132957 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe02c032440 con 0x7fe03406b830 2024-04-25T01:17:44.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.123+0000 7fe030a00640 1 -- 172.21.15.90:0/1807132957 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe02c032810 con 0x7fe03406b830 2024-04-25T01:17:44.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.124+0000 7fe033200640 1 -- 172.21.15.90:0/1807132957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe03406b830 msgr2=0x7fe0340ae730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:44.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.124+0000 7fe033200640 1 --2- 172.21.15.90:0/1807132957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe03406b830 0x7fe0340ae730 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe02c009870 tx=0x7fe02c02a2a0 comp rx=0 tx=0).stop 2024-04-25T01:17:44.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.124+0000 7fe033200640 1 -- 172.21.15.90:0/1807132957 shutdown_connections 2024-04-25T01:17:44.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.124+0000 7fe033200640 1 --2- 172.21.15.90:0/1807132957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe03406b830 0x7fe0340ae730 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:44.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.124+0000 7fe033200640 1 -- 172.21.15.90:0/1807132957 >> 172.21.15.90:0/1807132957 conn(0x7fe0340b1660 msgr2=0x7fe0340b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:44.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.124+0000 7fe033200640 1 -- 172.21.15.90:0/1807132957 shutdown_connections 2024-04-25T01:17:44.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.124+0000 7fe033200640 1 -- 172.21.15.90:0/1807132957 wait complete. 2024-04-25T01:17:44.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.126+0000 7fe033200640 1 Processor -- start 2024-04-25T01:17:44.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.126+0000 7fe033200640 1 -- start start 2024-04-25T01:17:44.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.127+0000 7fe033200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe03406b830 0x7fe03415f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:44.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.127+0000 7fe033200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe03415c8a0 0x7fe034153cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:44.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.127+0000 7fe033200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0341633d0 0x7fe0341f91d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:44.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.127+0000 7fe033200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe03415cc80 con 0x7fe03415c8a0 2024-04-25T01:17:44.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.127+0000 7fe033200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe034163860 con 0x7fe0341633d0 2024-04-25T01:17:44.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.127+0000 7fe033200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe0340cdf40 con 0x7fe03406b830 2024-04-25T01:17:44.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.127+0000 7fe031400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe03415c8a0 0x7fe034153cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:44.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.127+0000 7fe031400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe03415c8a0 0x7fe034153cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57264/0 (socket says 172.21.15.90:57264) 2024-04-25T01:17:44.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.127+0000 7fe031400640 1 -- 172.21.15.90:0/3628423153 learned_addr learned my addr 172.21.15.90:0/3628423153 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:44.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.127+0000 7fe031400640 1 -- 172.21.15.90:0/3628423153 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe03406b830 msgr2=0x7fe03415f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:44.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.127+0000 7fe031400640 1 --2- 172.21.15.90:0/3628423153 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe03406b830 0x7fe03415f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:44.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.127+0000 7fe031400640 1 -- 172.21.15.90:0/3628423153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0341633d0 msgr2=0x7fe0341f91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:44.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.127+0000 7fe031400640 1 --2- 172.21.15.90:0/3628423153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0341633d0 0x7fe0341f91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:44.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.127+0000 7fe031400640 1 -- 172.21.15.90:0/3628423153 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0341f9710 con 0x7fe03415c8a0 2024-04-25T01:17:44.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.128+0000 7fe031400640 1 --2- 172.21.15.90:0/3628423153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe03415c8a0 0x7fe034153cf0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe01c00c440 tx=0x7fe01c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:44.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.128+0000 7fe02a000640 1 -- 172.21.15.90:0/3628423153 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe01c017070 con 0x7fe03415c8a0 2024-04-25T01:17:44.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.128+0000 7fe02a000640 1 -- 172.21.15.90:0/3628423153 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe01c005480 con 0x7fe03415c8a0 2024-04-25T01:17:44.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.128+0000 7fe02a000640 1 -- 172.21.15.90:0/3628423153 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe01c003830 con 0x7fe03415c8a0 2024-04-25T01:17:44.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.128+0000 7fe033200640 1 -- 172.21.15.90:0/3628423153 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe03415e350 con 0x7fe03415c8a0 2024-04-25T01:17:44.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.128+0000 7fe033200640 1 -- 172.21.15.90:0/3628423153 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe03415e910 con 0x7fe03415c8a0 2024-04-25T01:17:44.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.129+0000 7fe033200640 1 -- 172.21.15.90:0/3628423153 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe034075350 con 0x7fe03415c8a0 2024-04-25T01:17:44.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.130+0000 7fe02a000640 1 -- 172.21.15.90:0/3628423153 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe01c01a070 con 0x7fe03415c8a0 2024-04-25T01:17:44.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.131+0000 7fe02a000640 1 --2- 172.21.15.90:0/3628423153 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe01006be30 0x7fe01006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:44.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.131+0000 7fe02a000640 1 -- 172.21.15.90:0/3628423153 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7fe01c08a490 con 0x7fe03415c8a0 2024-04-25T01:17:44.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.131+0000 7fe031e00640 1 --2- 172.21.15.90:0/3628423153 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe01006be30 0x7fe01006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:44.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.132+0000 7fe031e00640 1 --2- 172.21.15.90:0/3628423153 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe01006be30 0x7fe01006e2f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe0240043c0 tx=0x7fe024007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:44.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.136+0000 7fe02a000640 1 -- 172.21.15.90:0/3628423153 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe0341f9710 con 0x7fe03415c8a0 2024-04-25T01:17:44.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.345+0000 7fe033200640 1 -- 172.21.15.90:0/3628423153 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe034160230 con 0x7fe03415c8a0 2024-04-25T01:17:44.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.346+0000 7fe02a000640 1 -- 172.21.15.90:0/3628423153 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v28) ==== 80+0+0 (secure 0 0 0) 0x7fe01c058ac0 con 0x7fe03415c8a0 2024-04-25T01:17:44.329 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-04-25T01:17:44.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.348+0000 7fe033200640 1 -- 172.21.15.90:0/3628423153 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe01006be30 msgr2=0x7fe01006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:44.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.348+0000 7fe033200640 1 --2- 172.21.15.90:0/3628423153 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe01006be30 0x7fe01006e2f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe0240043c0 tx=0x7fe024007d30 comp rx=0 tx=0).stop 2024-04-25T01:17:44.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.348+0000 7fe033200640 1 -- 172.21.15.90:0/3628423153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe03415c8a0 msgr2=0x7fe034153cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:44.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.348+0000 7fe033200640 1 --2- 172.21.15.90:0/3628423153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe03415c8a0 0x7fe034153cf0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe01c00c440 tx=0x7fe01c00c900 comp rx=0 tx=0).stop 2024-04-25T01:17:44.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.349+0000 7fe033200640 1 -- 172.21.15.90:0/3628423153 shutdown_connections 2024-04-25T01:17:44.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.349+0000 7fe033200640 1 --2- 172.21.15.90:0/3628423153 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe01006be30 0x7fe01006e2f0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:44.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.349+0000 7fe033200640 1 --2- 172.21.15.90:0/3628423153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0341633d0 0x7fe0341f91d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:44.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.349+0000 7fe033200640 1 --2- 172.21.15.90:0/3628423153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe03415c8a0 0x7fe034153cf0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:44.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.349+0000 7fe033200640 1 --2- 172.21.15.90:0/3628423153 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe03406b830 0x7fe03415f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:44.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.349+0000 7fe033200640 1 -- 172.21.15.90:0/3628423153 >> 172.21.15.90:0/3628423153 conn(0x7fe0340b1660 msgr2=0x7fe034070a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:44.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.349+0000 7fe033200640 1 -- 172.21.15.90:0/3628423153 shutdown_connections 2024-04-25T01:17:44.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.349+0000 7fe033200640 1 -- 172.21.15.90:0/3628423153 wait complete. 2024-04-25T01:17:44.351 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-04-25T01:17:44.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:44.351 DEBUG:teuthology.orchestra.run.smithi103:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-04-25T01:17:44.445 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi103.front.sepia.ceph.com 2024-04-25T01:17:44.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:17:44.445 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-25T01:17:44.497 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-25T01:17:44.497 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:17:44.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.730+0000 7f0deca00640 1 Processor -- start 2024-04-25T01:17:44.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.730+0000 7f0deca00640 1 -- start start 2024-04-25T01:17:44.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.731+0000 7f0deca00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0de80b41f0 0x7f0de806b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:44.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.731+0000 7f0deca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0de80ae9d0 0x7f0de806b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:44.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.731+0000 7f0deca00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0de80adc10 con 0x7f0de815e6f0 2024-04-25T01:17:44.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.731+0000 7f0deca00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f0de80adef0 con 0x7f0de80b41f0 2024-04-25T01:17:44.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.731+0000 7f0deca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0de80ae1d0 con 0x7f0de80ae9d0 2024-04-25T01:17:44.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.732+0000 7f0de7e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f0de815e6f0 0x7f0de815eb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:35240/0 (socket says 172.21.15.90:35240) 2024-04-25T01:17:44.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.732+0000 7f0de7e00640 1 -- 172.21.15.90:0/1942560807 learned_addr learned my addr 172.21.15.90:0/1942560807 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:44.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.732+0000 7f0de6a00640 1 -- 172.21.15.90:0/1942560807 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1730526483 0 0) 0x7f0de80adc10 con 0x7f0de815e6f0 2024-04-25T01:17:44.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.732+0000 7f0de6a00640 1 -- 172.21.15.90:0/1942560807 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dcc003520 con 0x7f0de815e6f0 2024-04-25T01:17:44.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.732+0000 7f0ddea00640 1 --2- 172.21.15.90:0/1942560807 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0de80ae9d0 0x7f0de806b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:44.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.732+0000 7f0de7400640 1 --2- 172.21.15.90:0/1942560807 >> v2:172.21.15.103:3300/0 conn(0x7f0de80b41f0 0x7f0de806b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:44.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.733+0000 7f0de6a00640 1 -- 172.21.15.90:0/1942560807 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3162030818 0 0) 0x7f0dcc003520 con 0x7f0de815e6f0 2024-04-25T01:17:44.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.733+0000 7f0de6a00640 1 -- 172.21.15.90:0/1942560807 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0de80ae9d0 msgr2=0x7f0de806b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:44.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.733+0000 7f0de6a00640 1 --2- 172.21.15.90:0/1942560807 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0de80ae9d0 0x7f0de806b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:44.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.733+0000 7f0de6a00640 1 -- 172.21.15.90:0/1942560807 >> v2:172.21.15.103:3300/0 conn(0x7f0de80b41f0 msgr2=0x7f0de806b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:44.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.733+0000 7f0de6a00640 1 --2- 172.21.15.90:0/1942560807 >> v2:172.21.15.103:3300/0 conn(0x7f0de80b41f0 0x7f0de806b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:44.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.733+0000 7f0de6a00640 1 -- 172.21.15.90:0/1942560807 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0de80ae4b0 con 0x7f0de815e6f0 2024-04-25T01:17:44.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.733+0000 7f0de6a00640 1 -- 172.21.15.90:0/1942560807 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f0de00019f0 con 0x7f0de815e6f0 2024-04-25T01:17:44.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.733+0000 7f0de6a00640 1 -- 172.21.15.90:0/1942560807 >> v1:172.21.15.90:6789/0 conn(0x7f0de815e6f0 legacy=0x7f0de815eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:44.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.733+0000 7f0de6a00640 1 --2- 172.21.15.90:0/1942560807 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0dcc0039b0 0x7f0dcc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:44.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.733+0000 7f0de6a00640 1 --2- 172.21.15.90:0/1942560807 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dcc004800 0x7f0dcc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:44.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.733+0000 7f0de6a00640 1 --2- 172.21.15.90:0/1942560807 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0de80ae9d0 0x7f0dcc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:44.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.733+0000 7f0de6a00640 1 -- 172.21.15.90:0/1942560807 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0de80adef0 con 0x7f0dcc004800 2024-04-25T01:17:44.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.733+0000 7f0de6a00640 1 -- 172.21.15.90:0/1942560807 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0de80ae1d0 con 0x7f0dcc0039b0 2024-04-25T01:17:44.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.733+0000 7f0de6a00640 1 -- 172.21.15.90:0/1942560807 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0dcc00bac0 con 0x7f0de80ae9d0 2024-04-25T01:17:44.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.733+0000 7f0ddea00640 1 --2- 172.21.15.90:0/1942560807 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dcc004800 0x7f0dcc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:44.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.733+0000 7f0ddea00640 1 -- 172.21.15.90:0/1942560807 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0de80ae9d0 msgr2=0x7f0dcc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:44.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.733+0000 7f0ddea00640 1 --2- 172.21.15.90:0/1942560807 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0de80ae9d0 0x7f0dcc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:44.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.733+0000 7f0ddea00640 1 -- 172.21.15.90:0/1942560807 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0dcc0039b0 msgr2=0x7f0dcc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:44.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.733+0000 7f0ddea00640 1 --2- 172.21.15.90:0/1942560807 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0dcc0039b0 0x7f0dcc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:44.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.733+0000 7f0ddea00640 1 -- 172.21.15.90:0/1942560807 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dcc003520 con 0x7f0dcc004800 2024-04-25T01:17:44.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.734+0000 7f0ddea00640 1 --2- 172.21.15.90:0/1942560807 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dcc004800 0x7f0dcc004c80 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0dd000ba00 tx=0x7f0dcc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=600cdb50d7f34189 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:44.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.735+0000 7f0de6a00640 1 -- 172.21.15.90:0/1942560807 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd0030070 con 0x7f0dcc004800 2024-04-25T01:17:44.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.735+0000 7f0de6a00640 1 -- 172.21.15.90:0/1942560807 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dd002adb0 con 0x7f0dcc004800 2024-04-25T01:17:44.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.735+0000 7f0de6a00640 1 -- 172.21.15.90:0/1942560807 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd000b600 con 0x7f0dcc004800 2024-04-25T01:17:44.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.735+0000 7f0deca00640 1 -- 172.21.15.90:0/1942560807 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dcc004800 msgr2=0x7f0dcc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:44.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.735+0000 7f0deca00640 1 --2- 172.21.15.90:0/1942560807 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dcc004800 0x7f0dcc004c80 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0dd000ba00 tx=0x7f0dcc005830 comp rx=0 tx=0).stop 2024-04-25T01:17:44.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.735+0000 7f0de7e00640 1 -- 172.21.15.90:0/1942560807 reap_dead start 2024-04-25T01:17:44.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.735+0000 7f0deca00640 1 -- 172.21.15.90:0/1942560807 shutdown_connections 2024-04-25T01:17:44.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.735+0000 7f0deca00640 1 -- 172.21.15.90:0/1942560807 >> 172.21.15.90:0/1942560807 conn(0x7f0de80b1660 msgr2=0x7f0de80b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:44.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.735+0000 7f0deca00640 1 -- 172.21.15.90:0/1942560807 shutdown_connections 2024-04-25T01:17:44.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.736+0000 7f0deca00640 1 -- 172.21.15.90:0/1942560807 wait complete. 2024-04-25T01:17:44.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.737+0000 7f0deca00640 1 Processor -- start 2024-04-25T01:17:44.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.737+0000 7f0deca00640 1 -- start start 2024-04-25T01:17:44.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.738+0000 7f0deca00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0dcc004800 0x7f0de81f7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:44.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.738+0000 7f0deca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0de81418d0 0x7f0de8141d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:44.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.738+0000 7f0deca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0de8156110 0x7f0de81565c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:44.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.738+0000 7f0deca00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0de806b9f0 con 0x7f0de81418d0 2024-04-25T01:17:44.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.738+0000 7f0deca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0de806b870 con 0x7f0de8156110 2024-04-25T01:17:44.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.738+0000 7f0deca00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0de8157c30 con 0x7f0dcc004800 2024-04-25T01:17:44.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.739+0000 7f0ddea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0de81418d0 0x7f0de8141d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:44.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.739+0000 7f0ddea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0de81418d0 0x7f0de8141d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57282/0 (socket says 172.21.15.90:57282) 2024-04-25T01:17:44.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.739+0000 7f0ddea00640 1 -- 172.21.15.90:0/4033412915 learned_addr learned my addr 172.21.15.90:0/4033412915 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:44.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.739+0000 7f0de7400640 1 --2- 172.21.15.90:0/4033412915 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0dcc004800 0x7f0de81f7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:44.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.739+0000 7f0ddea00640 1 -- 172.21.15.90:0/4033412915 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0dcc004800 msgr2=0x7f0de81f7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:44.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.739+0000 7f0ddea00640 1 --2- 172.21.15.90:0/4033412915 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0dcc004800 0x7f0de81f7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:44.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.739+0000 7f0ddea00640 1 -- 172.21.15.90:0/4033412915 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0de8156110 msgr2=0x7f0de81565c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:44.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.739+0000 7f0ddea00640 1 --2- 172.21.15.90:0/4033412915 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0de8156110 0x7f0de81565c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:44.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.739+0000 7f0ddea00640 1 -- 172.21.15.90:0/4033412915 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0de81f76e0 con 0x7f0de81418d0 2024-04-25T01:17:44.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.739+0000 7f0ddea00640 1 --2- 172.21.15.90:0/4033412915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0de81418d0 0x7f0de8141d50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0dd000bb20 tx=0x7f0dd0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:44.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.740+0000 7f0ddfe00640 1 -- 172.21.15.90:0/4033412915 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd0043020 con 0x7f0de81418d0 2024-04-25T01:17:44.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.740+0000 7f0ddfe00640 1 -- 172.21.15.90:0/4033412915 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dd003f680 con 0x7f0de81418d0 2024-04-25T01:17:44.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.740+0000 7f0ddfe00640 1 -- 172.21.15.90:0/4033412915 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd000b600 con 0x7f0de81418d0 2024-04-25T01:17:44.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.740+0000 7f0deca00640 1 -- 172.21.15.90:0/4033412915 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0de81f79c0 con 0x7f0de81418d0 2024-04-25T01:17:44.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.740+0000 7f0deca00640 1 -- 172.21.15.90:0/4033412915 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0de81413b0 con 0x7f0de81418d0 2024-04-25T01:17:44.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.741+0000 7f0deca00640 1 -- 172.21.15.90:0/4033412915 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0de8075350 con 0x7f0de81418d0 2024-04-25T01:17:44.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.742+0000 7f0ddfe00640 1 -- 172.21.15.90:0/4033412915 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0dd0046020 con 0x7f0de81418d0 2024-04-25T01:17:44.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.743+0000 7f0ddfe00640 1 --2- 172.21.15.90:0/4033412915 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0dc406be60 0x7f0dc406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:44.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.743+0000 7f0ddfe00640 1 -- 172.21.15.90:0/4033412915 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f0dd00abda0 con 0x7f0de81418d0 2024-04-25T01:17:44.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.743+0000 7f0de7400640 1 --2- 172.21.15.90:0/4033412915 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0dc406be60 0x7f0dc406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:44.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.744+0000 7f0de7400640 1 --2- 172.21.15.90:0/4033412915 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0dc406be60 0x7f0dc406e320 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0dd8006a90 tx=0x7f0dd8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:44.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.747+0000 7f0ddfe00640 1 -- 172.21.15.90:0/4033412915 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0dd004d030 con 0x7f0de81418d0 2024-04-25T01:17:44.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.986+0000 7f0deca00640 1 -- 172.21.15.90:0/4033412915 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0de81f7ee0 con 0x7f0de81418d0 2024-04-25T01:17:44.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.987+0000 7f0ddfe00640 1 -- 172.21.15.90:0/4033412915 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5180 (secure 0 0 0) 0x7f0dd007a2b0 con 0x7f0de81418d0 2024-04-25T01:17:44.971 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:17:44.971 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6834/2605945289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":2605945289},{"type":"v1","addr":"172.21.15.103:6834","nonce":2605945289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6836/704639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":704639426},{"type":"v1","addr":"172.21.15.90:6836","nonce":704639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4245,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3477867755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6835","nonce":3477867755},{"type":"v1","addr":"172.21.15.90:6837","nonce":3477867755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:16:22.060018+0000","modified":"2024-04-25T01:16:50.405431+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6835/4291578198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6833","nonce":4291578198},{"type":"v1","addr":"172.21.15.103:6835","nonce":4291578198}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1}]} 2024-04-25T01:17:44.971 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 9 2024-04-25T01:17:44.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.990+0000 7f0deca00640 1 -- 172.21.15.90:0/4033412915 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0dc406be60 msgr2=0x7f0dc406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:44.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.990+0000 7f0deca00640 1 --2- 172.21.15.90:0/4033412915 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0dc406be60 0x7f0dc406e320 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0dd8006a90 tx=0x7f0dd8008040 comp rx=0 tx=0).stop 2024-04-25T01:17:44.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.990+0000 7f0deca00640 1 -- 172.21.15.90:0/4033412915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0de81418d0 msgr2=0x7f0de8141d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:44.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.990+0000 7f0deca00640 1 --2- 172.21.15.90:0/4033412915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0de81418d0 0x7f0de8141d50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0dd000bb20 tx=0x7f0dd0004730 comp rx=0 tx=0).stop 2024-04-25T01:17:44.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.990+0000 7f0deca00640 1 -- 172.21.15.90:0/4033412915 shutdown_connections 2024-04-25T01:17:44.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.990+0000 7f0deca00640 1 --2- 172.21.15.90:0/4033412915 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0dc406be60 0x7f0dc406e320 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:44.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.990+0000 7f0deca00640 1 --2- 172.21.15.90:0/4033412915 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0de8156110 0x7f0de81565c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:44.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.990+0000 7f0deca00640 1 --2- 172.21.15.90:0/4033412915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0de81418d0 0x7f0de8141d50 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:44.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.990+0000 7f0deca00640 1 --2- 172.21.15.90:0/4033412915 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0dcc004800 0x7f0de81f7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:44.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.990+0000 7f0deca00640 1 -- 172.21.15.90:0/4033412915 >> 172.21.15.90:0/4033412915 conn(0x7f0de80b1660 msgr2=0x7f0de80b2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:44.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.990+0000 7f0deca00640 1 -- 172.21.15.90:0/4033412915 shutdown_connections 2024-04-25T01:17:44.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:44.990+0000 7f0deca00640 1 -- 172.21.15.90:0/4033412915 wait complete. 2024-04-25T01:17:44.993 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:17:45.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.211+0000 7f0cfbe00640 1 Processor -- start 2024-04-25T01:17:45.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.211+0000 7f0cfbe00640 1 -- start start 2024-04-25T01:17:45.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.212+0000 7f0cfbe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0cfc15ad40 0x7f0cfc06b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:45.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.212+0000 7f0cfbe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0cfc0cf520 0x7f0cfc0ae730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:45.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.212+0000 7f0cfbe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cfc06ba90 con 0x7f0cfc06d910 2024-04-25T01:17:45.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.212+0000 7f0cfbe00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f0cfc06cb50 con 0x7f0cfc0cf520 2024-04-25T01:17:45.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.212+0000 7f0cfbe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0cfc06ce30 con 0x7f0cfc15ad40 2024-04-25T01:17:45.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.213+0000 7f0cf3e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0cfc0cf520 0x7f0cfc0ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:45.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.213+0000 7f0cf3e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0cfc0cf520 0x7f0cfc0ae730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46106/0 (socket says 172.21.15.90:46106) 2024-04-25T01:17:45.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.213+0000 7f0cf3e00640 1 -- 172.21.15.90:0/2280219377 learned_addr learned my addr 172.21.15.90:0/2280219377 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:45.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.213+0000 7f0cfaa00640 1 --2- 172.21.15.90:0/2280219377 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0cfc15ad40 0x7f0cfc06b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:45.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.213+0000 7f0cf3e00640 1 -- 172.21.15.90:0/2280219377 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0cfc15ad40 msgr2=0x7f0cfc06b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:45.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.213+0000 7f0cf3e00640 1 --2- 172.21.15.90:0/2280219377 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0cfc15ad40 0x7f0cfc06b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:45.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.213+0000 7f0cf3e00640 1 -- 172.21.15.90:0/2280219377 >> v1:172.21.15.103:6789/0 conn(0x7f0cfc06d910 legacy=0x7f0cfc15e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:45.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.213+0000 7f0cf3e00640 1 -- 172.21.15.90:0/2280219377 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cfc06d110 con 0x7f0cfc0cf520 2024-04-25T01:17:45.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.215+0000 7f0cf3e00640 1 --2- 172.21.15.90:0/2280219377 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0cfc0cf520 0x7f0cfc0ae730 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4009870 tx=0x7f0ce402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=512d62d130b0dd47 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:45.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.215+0000 7f0cfa000640 1 -- 172.21.15.90:0/2280219377 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4037070 con 0x7f0cfc0cf520 2024-04-25T01:17:45.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.215+0000 7f0cfa000640 1 -- 172.21.15.90:0/2280219377 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ce4032440 con 0x7f0cfc0cf520 2024-04-25T01:17:45.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.215+0000 7f0cfa000640 1 -- 172.21.15.90:0/2280219377 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4032810 con 0x7f0cfc0cf520 2024-04-25T01:17:45.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.216+0000 7f0cfbe00640 1 -- 172.21.15.90:0/2280219377 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0cfc0cf520 msgr2=0x7f0cfc0ae730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:45.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.216+0000 7f0cfbe00640 1 --2- 172.21.15.90:0/2280219377 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0cfc0cf520 0x7f0cfc0ae730 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4009870 tx=0x7f0ce402a270 comp rx=0 tx=0).stop 2024-04-25T01:17:45.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.216+0000 7f0cfbe00640 1 -- 172.21.15.90:0/2280219377 shutdown_connections 2024-04-25T01:17:45.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.216+0000 7f0cfbe00640 1 --2- 172.21.15.90:0/2280219377 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0cfc0cf520 0x7f0cfc0ae730 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:45.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.216+0000 7f0cfbe00640 1 --2- 172.21.15.90:0/2280219377 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0cfc15ad40 0x7f0cfc06b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:45.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.216+0000 7f0cfbe00640 1 -- 172.21.15.90:0/2280219377 >> 172.21.15.90:0/2280219377 conn(0x7f0cfc0b1660 msgr2=0x7f0cfc0b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:45.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.216+0000 7f0cfbe00640 1 -- 172.21.15.90:0/2280219377 shutdown_connections 2024-04-25T01:17:45.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.216+0000 7f0cfbe00640 1 -- 172.21.15.90:0/2280219377 wait complete. 2024-04-25T01:17:45.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.218+0000 7f0cfbe00640 1 Processor -- start 2024-04-25T01:17:45.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.218+0000 7f0cfbe00640 1 -- start start 2024-04-25T01:17:45.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.219+0000 7f0cfbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0cfc06d910 0x7f0cfc1477d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:45.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.219+0000 7f0cfbe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0cfc0cf520 0x7f0cfc0cc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:45.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.219+0000 7f0cfbe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0cfc15ad40 0x7f0cfc0cc930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:45.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.219+0000 7f0cfbe00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0cfc06b7e0 con 0x7f0cfc06d910 2024-04-25T01:17:45.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.219+0000 7f0cfbe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0cfc06b660 con 0x7f0cfc15ad40 2024-04-25T01:17:45.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.219+0000 7f0cfbe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0cfc0aec70 con 0x7f0cfc0cf520 2024-04-25T01:17:45.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.219+0000 7f0cfaa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0cfc06d910 0x7f0cfc1477d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:45.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.219+0000 7f0cfaa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0cfc06d910 0x7f0cfc1477d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46108/0 (socket says 172.21.15.90:46108) 2024-04-25T01:17:45.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.219+0000 7f0cfaa00640 1 -- 172.21.15.90:0/1025186609 learned_addr learned my addr 172.21.15.90:0/1025186609 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:45.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.219+0000 7f0cfaa00640 1 -- 172.21.15.90:0/1025186609 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0cfc0cf520 msgr2=0x7f0cfc0cc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:45.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.219+0000 7f0cfaa00640 1 --2- 172.21.15.90:0/1025186609 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0cfc0cf520 0x7f0cfc0cc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:45.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.220+0000 7f0cfaa00640 1 -- 172.21.15.90:0/1025186609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0cfc15ad40 msgr2=0x7f0cfc0cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:45.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.220+0000 7f0cfaa00640 1 --2- 172.21.15.90:0/1025186609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0cfc15ad40 0x7f0cfc0cc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:45.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.220+0000 7f0cfaa00640 1 -- 172.21.15.90:0/1025186609 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cfc143db0 con 0x7f0cfc06d910 2024-04-25T01:17:45.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.220+0000 7f0cfaa00640 1 --2- 172.21.15.90:0/1025186609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0cfc06d910 0x7f0cfc1477d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f0cec00d5f0 tx=0x7f0cec00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:45.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.220+0000 7f0cf2a00640 1 -- 172.21.15.90:0/1025186609 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cec0125f0 con 0x7f0cfc06d910 2024-04-25T01:17:45.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.220+0000 7f0cf2a00640 1 -- 172.21.15.90:0/1025186609 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cec019070 con 0x7f0cfc06d910 2024-04-25T01:17:45.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.220+0000 7f0cf2a00640 1 -- 172.21.15.90:0/1025186609 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cec0129d0 con 0x7f0cfc06d910 2024-04-25T01:17:45.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.220+0000 7f0cfbe00640 1 -- 172.21.15.90:0/1025186609 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cfc144090 con 0x7f0cfc06d910 2024-04-25T01:17:45.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.220+0000 7f0cfbe00640 1 -- 172.21.15.90:0/1025186609 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cfc144650 con 0x7f0cfc06d910 2024-04-25T01:17:45.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.221+0000 7f0cfbe00640 1 -- 172.21.15.90:0/1025186609 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cfc075350 con 0x7f0cfc06d910 2024-04-25T01:17:45.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.222+0000 7f0cf2a00640 1 -- 172.21.15.90:0/1025186609 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0cec012b70 con 0x7f0cfc06d910 2024-04-25T01:17:45.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.223+0000 7f0cf2a00640 1 --2- 172.21.15.90:0/1025186609 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0cd806be30 0x7f0cd806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:45.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.223+0000 7f0cf2a00640 1 -- 172.21.15.90:0/1025186609 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f0cec08b660 con 0x7f0cfc06d910 2024-04-25T01:17:45.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.224+0000 7f0cf3e00640 1 --2- 172.21.15.90:0/1025186609 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0cd806be30 0x7f0cd806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:45.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.224+0000 7f0cf3e00640 1 --2- 172.21.15.90:0/1025186609 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0cd806be30 0x7f0cd806e2f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4034000 tx=0x7f0ce402c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:45.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.228+0000 7f0cf2a00640 1 -- 172.21.15.90:0/1025186609 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0cec059c00 con 0x7f0cfc06d910 2024-04-25T01:17:45.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.425+0000 7f0cfbe00640 1 -- 172.21.15.90:0/1025186609 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0cfc06b120 con 0x7f0cfc06d910 2024-04-25T01:17:45.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.428+0000 7f0cf2a00640 1 -- 172.21.15.90:0/1025186609 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18576 (secure 0 0 0) 0x7f0cec059500 con 0x7f0cfc06d910 2024-04-25T01:17:45.412 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:17:45.412 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:17:09.346008+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:16:20.094512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:16:21.078268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:16:24.670642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.e","mappings":[{"from":0,"to":7}]},{"pgid":"4.f","mappings":[{"from":7,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:17:45.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.431+0000 7f0cfbe00640 1 -- 172.21.15.90:0/1025186609 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0cd806be30 msgr2=0x7f0cd806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:45.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.431+0000 7f0cfbe00640 1 --2- 172.21.15.90:0/1025186609 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0cd806be30 0x7f0cd806e2f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4034000 tx=0x7f0ce402c000 comp rx=0 tx=0).stop 2024-04-25T01:17:45.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.431+0000 7f0cfbe00640 1 -- 172.21.15.90:0/1025186609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0cfc06d910 msgr2=0x7f0cfc1477d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:45.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.431+0000 7f0cfbe00640 1 --2- 172.21.15.90:0/1025186609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0cfc06d910 0x7f0cfc1477d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f0cec00d5f0 tx=0x7f0cec00dab0 comp rx=0 tx=0).stop 2024-04-25T01:17:45.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.431+0000 7f0cfbe00640 1 -- 172.21.15.90:0/1025186609 shutdown_connections 2024-04-25T01:17:45.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.431+0000 7f0cfbe00640 1 --2- 172.21.15.90:0/1025186609 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0cd806be30 0x7f0cd806e2f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:45.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.431+0000 7f0cfbe00640 1 --2- 172.21.15.90:0/1025186609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0cfc15ad40 0x7f0cfc0cc930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:45.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.431+0000 7f0cfbe00640 1 --2- 172.21.15.90:0/1025186609 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0cfc0cf520 0x7f0cfc0cc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:45.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.431+0000 7f0cfbe00640 1 --2- 172.21.15.90:0/1025186609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0cfc06d910 0x7f0cfc1477d0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:45.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.431+0000 7f0cfbe00640 1 -- 172.21.15.90:0/1025186609 >> 172.21.15.90:0/1025186609 conn(0x7f0cfc0b1660 msgr2=0x7f0cfc070dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:45.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.432+0000 7f0cfbe00640 1 -- 172.21.15.90:0/1025186609 shutdown_connections 2024-04-25T01:17:45.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.432+0000 7f0cfbe00640 1 -- 172.21.15.90:0/1025186609 wait complete. 2024-04-25T01:17:45.433 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:16:15.894126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:16:20.094512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:16:21.078268+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:16:24.670642+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:17:45.434 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:17:45.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.655+0000 7f572d000640 1 Processor -- start 2024-04-25T01:17:45.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.655+0000 7f572d000640 1 -- start start 2024-04-25T01:17:45.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.656+0000 7f572d000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f57280b37c0 0x7f572815bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:45.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.656+0000 7f572d000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f572815c210 con 0x7f5728158a70 2024-04-25T01:17:45.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.656+0000 7f572d000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57280b2ce0 con 0x7f57280b47d0 2024-04-25T01:17:45.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.656+0000 7f572d000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f57280b2fc0 con 0x7f57280b37c0 2024-04-25T01:17:45.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.657+0000 7f5725e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f5728158a70 0x7f57280b40c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47888/0 (socket says 172.21.15.90:47888) 2024-04-25T01:17:45.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.657+0000 7f5725e00640 1 -- 172.21.15.90:0/2072009370 learned_addr learned my addr 172.21.15.90:0/2072009370 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:45.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.657+0000 7f5726800640 1 --2- 172.21.15.90:0/2072009370 >> v2:172.21.15.90:3300/0 conn(0x7f57280b37c0 0x7f572815bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:45.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.657+0000 7f5726800640 1 -- 172.21.15.90:0/2072009370 >> v1:172.21.15.103:6789/0 conn(0x7f57280b47d0 legacy=0x7f572815b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:45.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.657+0000 7f5726800640 1 -- 172.21.15.90:0/2072009370 >> v1:172.21.15.90:6789/0 conn(0x7f5728158a70 legacy=0x7f57280b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:45.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.657+0000 7f5726800640 1 -- 172.21.15.90:0/2072009370 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57280b32a0 con 0x7f57280b37c0 2024-04-25T01:17:45.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.657+0000 7f5725400640 1 -- 172.21.15.90:0/2072009370 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3670363797 0 0) 0x7f572815c210 con 0x7f5728158a70 2024-04-25T01:17:45.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.658+0000 7f5726800640 1 --2- 172.21.15.90:0/2072009370 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57280b37c0 0x7f572815bc30 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f5720009dc0 tx=0x7f572002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29e05cf16a6b2090 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:45.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.659+0000 7f5725400640 1 -- 172.21.15.90:0/2072009370 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5720007740 con 0x7f57280b37c0 2024-04-25T01:17:45.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.659+0000 7f5725400640 1 -- 172.21.15.90:0/2072009370 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57200078e0 con 0x7f57280b37c0 2024-04-25T01:17:45.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.659+0000 7f5725400640 1 -- 172.21.15.90:0/2072009370 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5720007cb0 con 0x7f57280b37c0 2024-04-25T01:17:45.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.659+0000 7f572d000640 1 -- 172.21.15.90:0/2072009370 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57280b37c0 msgr2=0x7f572815bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:45.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.659+0000 7f572d000640 1 --2- 172.21.15.90:0/2072009370 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57280b37c0 0x7f572815bc30 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f5720009dc0 tx=0x7f572002a730 comp rx=0 tx=0).stop 2024-04-25T01:17:45.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.659+0000 7f572d000640 1 -- 172.21.15.90:0/2072009370 shutdown_connections 2024-04-25T01:17:45.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.659+0000 7f572d000640 1 --2- 172.21.15.90:0/2072009370 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57280b37c0 0x7f572815bc30 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:45.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.659+0000 7f572d000640 1 -- 172.21.15.90:0/2072009370 >> 172.21.15.90:0/2072009370 conn(0x7f572806b720 msgr2=0x7f57280b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:45.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.660+0000 7f572d000640 1 -- 172.21.15.90:0/2072009370 shutdown_connections 2024-04-25T01:17:45.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.660+0000 7f572d000640 1 -- 172.21.15.90:0/2072009370 wait complete. 2024-04-25T01:17:45.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.661+0000 7f572d000640 1 Processor -- start 2024-04-25T01:17:45.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.661+0000 7f572d000640 1 -- start start 2024-04-25T01:17:45.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.662+0000 7f572d000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57280b37c0 0x7f572812eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:45.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.662+0000 7f572d000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57280b47d0 0x7f57280cd220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:45.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.662+0000 7f572d000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5728158a70 0x7f57280ae660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:45.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.662+0000 7f572d000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5728166020 con 0x7f5728158a70 2024-04-25T01:17:45.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.662+0000 7f572d000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5728165ea0 con 0x7f57280b47d0 2024-04-25T01:17:45.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.662+0000 7f572d000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f57281661a0 con 0x7f57280b37c0 2024-04-25T01:17:45.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.663+0000 7f5726800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5728158a70 0x7f57280ae660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:45.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.663+0000 7f5726800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5728158a70 0x7f57280ae660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46128/0 (socket says 172.21.15.90:46128) 2024-04-25T01:17:45.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.663+0000 7f5726800640 1 -- 172.21.15.90:0/3919167723 learned_addr learned my addr 172.21.15.90:0/3919167723 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:45.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.663+0000 7f5725e00640 1 --2- 172.21.15.90:0/3919167723 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57280b37c0 0x7f572812eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:45.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.663+0000 7f571fe00640 1 --2- 172.21.15.90:0/3919167723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57280b47d0 0x7f57280cd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:45.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.663+0000 7f5726800640 1 -- 172.21.15.90:0/3919167723 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57280b37c0 msgr2=0x7f572812eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:45.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.663+0000 7f5726800640 1 --2- 172.21.15.90:0/3919167723 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57280b37c0 0x7f572812eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:45.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.663+0000 7f5726800640 1 -- 172.21.15.90:0/3919167723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57280b47d0 msgr2=0x7f57280cd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:45.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.663+0000 7f5726800640 1 --2- 172.21.15.90:0/3919167723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57280b47d0 0x7f57280cd220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:45.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.663+0000 7f5726800640 1 -- 172.21.15.90:0/3919167723 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f572812f0a0 con 0x7f5728158a70 2024-04-25T01:17:45.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.663+0000 7f571fe00640 1 --2- 172.21.15.90:0/3919167723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57280b47d0 0x7f57280cd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:45.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.664+0000 7f5726800640 1 --2- 172.21.15.90:0/3919167723 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5728158a70 0x7f57280ae660 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f5720009ee0 tx=0x7f5720004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:45.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.664+0000 7f571e000640 1 -- 172.21.15.90:0/3919167723 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5720040070 con 0x7f5728158a70 2024-04-25T01:17:45.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.664+0000 7f571e000640 1 -- 172.21.15.90:0/3919167723 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5720031bc0 con 0x7f5728158a70 2024-04-25T01:17:45.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.664+0000 7f571e000640 1 -- 172.21.15.90:0/3919167723 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5720045460 con 0x7f5728158a70 2024-04-25T01:17:45.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.664+0000 7f572d000640 1 -- 172.21.15.90:0/3919167723 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f572812f320 con 0x7f5728158a70 2024-04-25T01:17:45.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.664+0000 7f572d000640 1 -- 172.21.15.90:0/3919167723 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f572812f8e0 con 0x7f5728158a70 2024-04-25T01:17:45.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.665+0000 7f572d000640 1 -- 172.21.15.90:0/3919167723 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5728068180 con 0x7f5728158a70 2024-04-25T01:17:45.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.666+0000 7f571e000640 1 -- 172.21.15.90:0/3919167723 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5720039040 con 0x7f5728158a70 2024-04-25T01:17:45.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.667+0000 7f571e000640 1 --2- 172.21.15.90:0/3919167723 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f570406be30 0x7f570406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:45.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.667+0000 7f571e000640 1 -- 172.21.15.90:0/3919167723 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f57200acbc0 con 0x7f5728158a70 2024-04-25T01:17:45.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.667+0000 7f5725e00640 1 --2- 172.21.15.90:0/3919167723 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f570406be30 0x7f570406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:45.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.668+0000 7f5725e00640 1 --2- 172.21.15.90:0/3919167723 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f570406be30 0x7f570406e2f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5718000f60 tx=0x7f5718007d00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:45.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.672+0000 7f571e000640 1 -- 172.21.15.90:0/3919167723 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f5728068180 con 0x7f5728158a70 2024-04-25T01:17:45.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.872+0000 7f572d000640 1 -- 172.21.15.90:0/3919167723 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5728068fb0 con 0x7f5728158a70 2024-04-25T01:17:45.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.873+0000 7f571e000640 1 -- 172.21.15.90:0/3919167723 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) ==== 93+0+10 (secure 0 0 0) 0x7f572812f0a0 con 0x7f5728158a70 2024-04-25T01:17:45.857 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T01:17:45.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.875+0000 7f572d000640 1 -- 172.21.15.90:0/3919167723 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f570406be30 msgr2=0x7f570406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:45.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.875+0000 7f572d000640 1 --2- 172.21.15.90:0/3919167723 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f570406be30 0x7f570406e2f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5718000f60 tx=0x7f5718007d00 comp rx=0 tx=0).stop 2024-04-25T01:17:45.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.875+0000 7f572d000640 1 -- 172.21.15.90:0/3919167723 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5728158a70 msgr2=0x7f57280ae660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:45.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.875+0000 7f572d000640 1 --2- 172.21.15.90:0/3919167723 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5728158a70 0x7f57280ae660 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f5720009ee0 tx=0x7f5720004a40 comp rx=0 tx=0).stop 2024-04-25T01:17:45.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.876+0000 7f572d000640 1 -- 172.21.15.90:0/3919167723 shutdown_connections 2024-04-25T01:17:45.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.876+0000 7f572d000640 1 --2- 172.21.15.90:0/3919167723 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f570406be30 0x7f570406e2f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:45.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.876+0000 7f572d000640 1 --2- 172.21.15.90:0/3919167723 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5728158a70 0x7f57280ae660 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:45.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.876+0000 7f572d000640 1 --2- 172.21.15.90:0/3919167723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57280b47d0 0x7f57280cd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:45.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.876+0000 7f572d000640 1 --2- 172.21.15.90:0/3919167723 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f57280b37c0 0x7f572812eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:45.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.876+0000 7f572d000640 1 -- 172.21.15.90:0/3919167723 >> 172.21.15.90:0/3919167723 conn(0x7f572806b720 msgr2=0x7f57280ad9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:45.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.876+0000 7f572d000640 1 -- 172.21.15.90:0/3919167723 shutdown_connections 2024-04-25T01:17:45.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:45.876+0000 7f572d000640 1 -- 172.21.15.90:0/3919167723 wait complete. 2024-04-25T01:17:45.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:17:46.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.097+0000 7fbc7a600640 1 Processor -- start 2024-04-25T01:17:46.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.097+0000 7fbc7a600640 1 -- start start 2024-04-25T01:17:46.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.099+0000 7fbc7a600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbc740722e0 0x7fbc7406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:46.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.099+0000 7fbc7a600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc7415fc40 0x7fbc741600c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:46.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.099+0000 7fbc7a600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc7406e410 con 0x7fbc740cedc0 2024-04-25T01:17:46.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.099+0000 7fbc7a600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fbc74160aa0 con 0x7fbc740722e0 2024-04-25T01:17:46.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.099+0000 7fbc7a600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbc7406da40 con 0x7fbc7415fc40 2024-04-25T01:17:46.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.100+0000 7fbc73400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbc740722e0 0x7fbc7406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:46.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.100+0000 7fbc6be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc7415fc40 0x7fbc741600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:46.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.100+0000 7fbc73400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbc740722e0 0x7fbc7406ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.90:58324/0 (socket says 172.21.15.90:58324) 2024-04-25T01:17:46.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.100+0000 7fbc73400640 1 -- 172.21.15.90:0/604994999 learned_addr learned my addr 172.21.15.90:0/604994999 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:46.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.101+0000 7fbc73400640 1 -- 172.21.15.90:0/604994999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc7415fc40 msgr2=0x7fbc741600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:46.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.101+0000 7fbc73400640 1 --2- 172.21.15.90:0/604994999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc7415fc40 0x7fbc741600c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.101+0000 7fbc73400640 1 -- 172.21.15.90:0/604994999 >> v1:172.21.15.103:6789/0 conn(0x7fbc740cedc0 legacy=0x7fbc740cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:46.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.101+0000 7fbc73400640 1 -- 172.21.15.90:0/604994999 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc7406d110 con 0x7fbc740722e0 2024-04-25T01:17:46.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.102+0000 7fbc73400640 1 --2- 172.21.15.90:0/604994999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc740722e0 0x7fbc7406ded0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbc64009870 tx=0x7fbc6402a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7e7bc423e553c8e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:46.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.103+0000 7fbc72a00640 1 -- 172.21.15.90:0/604994999 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc64037070 con 0x7fbc740722e0 2024-04-25T01:17:46.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.103+0000 7fbc72a00640 1 -- 172.21.15.90:0/604994999 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc64032440 con 0x7fbc740722e0 2024-04-25T01:17:46.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.103+0000 7fbc72a00640 1 -- 172.21.15.90:0/604994999 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc64032810 con 0x7fbc740722e0 2024-04-25T01:17:46.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.103+0000 7fbc7a600640 1 -- 172.21.15.90:0/604994999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc740722e0 msgr2=0x7fbc7406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:46.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.103+0000 7fbc7a600640 1 --2- 172.21.15.90:0/604994999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc740722e0 0x7fbc7406ded0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbc64009870 tx=0x7fbc6402a1c0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.104+0000 7fbc7a600640 1 -- 172.21.15.90:0/604994999 shutdown_connections 2024-04-25T01:17:46.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.104+0000 7fbc7a600640 1 --2- 172.21.15.90:0/604994999 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc7415fc40 0x7fbc741600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.104+0000 7fbc7a600640 1 --2- 172.21.15.90:0/604994999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc740722e0 0x7fbc7406ded0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.104+0000 7fbc7a600640 1 -- 172.21.15.90:0/604994999 >> 172.21.15.90:0/604994999 conn(0x7fbc7406b720 msgr2=0x7fbc740b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:46.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.104+0000 7fbc7a600640 1 -- 172.21.15.90:0/604994999 shutdown_connections 2024-04-25T01:17:46.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.104+0000 7fbc7a600640 1 -- 172.21.15.90:0/604994999 wait complete. 2024-04-25T01:17:46.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.106+0000 7fbc7a600640 1 Processor -- start 2024-04-25T01:17:46.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.106+0000 7fbc7a600640 1 -- start start 2024-04-25T01:17:46.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.108+0000 7fbc7a600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc740722e0 0x7fbc7414c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:46.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.108+0000 7fbc7a600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc740cedc0 0x7fbc74154e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:46.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.108+0000 7fbc7a600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc7415fc40 0x7fbc741553b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:46.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.108+0000 7fbc7a600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7406d6d0 con 0x7fbc740cedc0 2024-04-25T01:17:46.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.108+0000 7fbc7a600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7406cec0 con 0x7fbc7415fc40 2024-04-25T01:17:46.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.108+0000 7fbc7a600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbc74070b50 con 0x7fbc740722e0 2024-04-25T01:17:46.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.108+0000 7fbc6be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc740cedc0 0x7fbc74154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:46.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.108+0000 7fbc6be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc740cedc0 0x7fbc74154e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46136/0 (socket says 172.21.15.90:46136) 2024-04-25T01:17:46.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.108+0000 7fbc6be00640 1 -- 172.21.15.90:0/3462059059 learned_addr learned my addr 172.21.15.90:0/3462059059 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:46.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.108+0000 7fbc73400640 1 --2- 172.21.15.90:0/3462059059 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc740722e0 0x7fbc7414c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:46.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.108+0000 7fbc73e00640 1 --2- 172.21.15.90:0/3462059059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc7415fc40 0x7fbc741553b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:46.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.109+0000 7fbc6be00640 1 -- 172.21.15.90:0/3462059059 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc740722e0 msgr2=0x7fbc7414c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:46.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.109+0000 7fbc6be00640 1 --2- 172.21.15.90:0/3462059059 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc740722e0 0x7fbc7414c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.109+0000 7fbc6be00640 1 -- 172.21.15.90:0/3462059059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc7415fc40 msgr2=0x7fbc741553b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:46.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.109+0000 7fbc6be00640 1 --2- 172.21.15.90:0/3462059059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc7415fc40 0x7fbc741553b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.109+0000 7fbc6be00640 1 -- 172.21.15.90:0/3462059059 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc740ae600 con 0x7fbc740cedc0 2024-04-25T01:17:46.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.109+0000 7fbc73e00640 1 --2- 172.21.15.90:0/3462059059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc7415fc40 0x7fbc741553b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:17:46.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.109+0000 7fbc6be00640 1 --2- 172.21.15.90:0/3462059059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc740cedc0 0x7fbc74154e70 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c00e870 tx=0x7fbc5c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:46.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.109+0000 7fbc6b400640 1 -- 172.21.15.90:0/3462059059 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c017070 con 0x7fbc740cedc0 2024-04-25T01:17:46.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.109+0000 7fbc6b400640 1 -- 172.21.15.90:0/3462059059 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc5c00c940 con 0x7fbc740cedc0 2024-04-25T01:17:46.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.109+0000 7fbc6b400640 1 -- 172.21.15.90:0/3462059059 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c00cd30 con 0x7fbc740cedc0 2024-04-25T01:17:46.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.109+0000 7fbc7a600640 1 -- 172.21.15.90:0/3462059059 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc740ae8e0 con 0x7fbc740cedc0 2024-04-25T01:17:46.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.110+0000 7fbc7a600640 1 -- 172.21.15.90:0/3462059059 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc740aeea0 con 0x7fbc740cedc0 2024-04-25T01:17:46.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.111+0000 7fbc7a600640 1 -- 172.21.15.90:0/3462059059 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc7415a070 con 0x7fbc740cedc0 2024-04-25T01:17:46.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.112+0000 7fbc6b400640 1 -- 172.21.15.90:0/3462059059 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbc5c01a070 con 0x7fbc740cedc0 2024-04-25T01:17:46.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.112+0000 7fbc6b400640 1 --2- 172.21.15.90:0/3462059059 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbc5006be30 0x7fbc5006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:46.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.113+0000 7fbc6b400640 1 -- 172.21.15.90:0/3462059059 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7fbc5c08b390 con 0x7fbc740cedc0 2024-04-25T01:17:46.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.113+0000 7fbc73400640 1 --2- 172.21.15.90:0/3462059059 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbc5006be30 0x7fbc5006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:46.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.114+0000 7fbc73400640 1 --2- 172.21.15.90:0/3462059059 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbc5006be30 0x7fbc5006e2f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbc64034000 tx=0x7fbc6402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:46.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.117+0000 7fbc6b400640 1 -- 172.21.15.90:0/3462059059 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fbc7415a070 con 0x7fbc740cedc0 2024-04-25T01:17:46.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.310+0000 7fbc7a600640 1 -- 172.21.15.90:0/3462059059 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbc74068fb0 con 0x7fbc740cedc0 2024-04-25T01:17:46.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.311+0000 7fbc6b400640 1 -- 172.21.15.90:0/3462059059 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) ==== 104+0+11 (secure 0 0 0) 0x7fbc5c059370 con 0x7fbc740cedc0 2024-04-25T01:17:46.294 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:17:46.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.313+0000 7fbc7a600640 1 -- 172.21.15.90:0/3462059059 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbc5006be30 msgr2=0x7fbc5006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:46.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.313+0000 7fbc7a600640 1 --2- 172.21.15.90:0/3462059059 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbc5006be30 0x7fbc5006e2f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbc64034000 tx=0x7fbc6402e000 comp rx=0 tx=0).stop 2024-04-25T01:17:46.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.313+0000 7fbc7a600640 1 -- 172.21.15.90:0/3462059059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc740cedc0 msgr2=0x7fbc74154e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:46.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.313+0000 7fbc7a600640 1 --2- 172.21.15.90:0/3462059059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc740cedc0 0x7fbc74154e70 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c00e870 tx=0x7fbc5c00ed30 comp rx=0 tx=0).stop 2024-04-25T01:17:46.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.314+0000 7fbc7a600640 1 -- 172.21.15.90:0/3462059059 shutdown_connections 2024-04-25T01:17:46.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.314+0000 7fbc7a600640 1 --2- 172.21.15.90:0/3462059059 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbc5006be30 0x7fbc5006e2f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.314+0000 7fbc7a600640 1 --2- 172.21.15.90:0/3462059059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc7415fc40 0x7fbc741553b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.314+0000 7fbc7a600640 1 --2- 172.21.15.90:0/3462059059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc740cedc0 0x7fbc74154e70 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.314+0000 7fbc7a600640 1 --2- 172.21.15.90:0/3462059059 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbc740722e0 0x7fbc7414c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.314+0000 7fbc7a600640 1 -- 172.21.15.90:0/3462059059 >> 172.21.15.90:0/3462059059 conn(0x7fbc7406b720 msgr2=0x7fbc740b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:46.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.314+0000 7fbc7a600640 1 -- 172.21.15.90:0/3462059059 shutdown_connections 2024-04-25T01:17:46.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.314+0000 7fbc7a600640 1 -- 172.21.15.90:0/3462059059 wait complete. 2024-04-25T01:17:46.316 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:17:46.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.523+0000 7f4674a00640 1 Processor -- start 2024-04-25T01:17:46.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.523+0000 7f4674a00640 1 -- start start 2024-04-25T01:17:46.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.524+0000 7f4674a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f467006e280 0x7f467015b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:46.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.524+0000 7f4674a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f467015b780 0x7f46700ae620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:46.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.524+0000 7f4674a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4670155510 con 0x7f4670155130 2024-04-25T01:17:46.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.525+0000 7f4674a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f467015bfe0 con 0x7f467006e280 2024-04-25T01:17:46.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.525+0000 7f4674a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f467015d1d0 con 0x7f467015b780 2024-04-25T01:17:46.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.525+0000 7f4667e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f467006e280 0x7f467015b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:46.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.525+0000 7f466fe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f467015b780 0x7f46700ae620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:46.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.525+0000 7f4667e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f467006e280 0x7f467015b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.90:58342/0 (socket says 172.21.15.90:58342) 2024-04-25T01:17:46.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.525+0000 7f4667e00640 1 -- 172.21.15.90:0/848714290 learned_addr learned my addr 172.21.15.90:0/848714290 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:46.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.526+0000 7f466fe00640 1 -- 172.21.15.90:0/848714290 >> v2:172.21.15.103:3300/0 conn(0x7f467006e280 msgr2=0x7f467015b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:46.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.526+0000 7f466fe00640 1 --2- 172.21.15.90:0/848714290 >> v2:172.21.15.103:3300/0 conn(0x7f467006e280 0x7f467015b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.526+0000 7f466fe00640 1 -- 172.21.15.90:0/848714290 >> v1:172.21.15.103:6789/0 conn(0x7f4670155130 legacy=0x7f467006db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:46.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.526+0000 7f466fe00640 1 -- 172.21.15.90:0/848714290 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4670071af0 con 0x7f467015b780 2024-04-25T01:17:46.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.526+0000 7f4667e00640 1 --2- 172.21.15.90:0/848714290 >> v2:172.21.15.103:3300/0 conn(0x7f467006e280 0x7f467015b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:17:46.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.527+0000 7f466fe00640 1 --2- 172.21.15.90:0/848714290 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f467015b780 0x7f46700ae620 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4668009d90 tx=0x7f466802a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73a34100f7760c26 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:46.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.528+0000 7f466ea00640 1 -- 172.21.15.90:0/848714290 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4668031600 con 0x7f467015b780 2024-04-25T01:17:46.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.528+0000 7f466ea00640 1 -- 172.21.15.90:0/848714290 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46680317a0 con 0x7f467015b780 2024-04-25T01:17:46.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.528+0000 7f466ea00640 1 -- 172.21.15.90:0/848714290 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4668031b70 con 0x7f467015b780 2024-04-25T01:17:46.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.528+0000 7f4674a00640 1 -- 172.21.15.90:0/848714290 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f467015b780 msgr2=0x7f46700ae620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:46.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.528+0000 7f4674a00640 1 --2- 172.21.15.90:0/848714290 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f467015b780 0x7f46700ae620 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4668009d90 tx=0x7f466802a5d0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.528+0000 7f4674a00640 1 -- 172.21.15.90:0/848714290 shutdown_connections 2024-04-25T01:17:46.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.528+0000 7f4674a00640 1 --2- 172.21.15.90:0/848714290 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f467015b780 0x7f46700ae620 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.528+0000 7f4674a00640 1 --2- 172.21.15.90:0/848714290 >> v2:172.21.15.103:3300/0 conn(0x7f467006e280 0x7f467015b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.528+0000 7f4674a00640 1 -- 172.21.15.90:0/848714290 >> 172.21.15.90:0/848714290 conn(0x7f467006b120 msgr2=0x7f467006b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:46.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.528+0000 7f4674a00640 1 -- 172.21.15.90:0/848714290 shutdown_connections 2024-04-25T01:17:46.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.529+0000 7f4674a00640 1 -- 172.21.15.90:0/848714290 wait complete. 2024-04-25T01:17:46.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.530+0000 7f4674a00640 1 Processor -- start 2024-04-25T01:17:46.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.530+0000 7f4674a00640 1 -- start start 2024-04-25T01:17:46.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.531+0000 7f4674a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f467006e280 0x7f46701527f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:46.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.531+0000 7f4674a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4670155130 0x7f46700cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:46.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.531+0000 7f4674a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f467015b780 0x7f46700c9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:46.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.531+0000 7f4674a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f46700728b0 con 0x7f467006e280 2024-04-25T01:17:46.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.531+0000 7f4674a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f46700720b0 con 0x7f467015b780 2024-04-25T01:17:46.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.531+0000 7f4674a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4670072a30 con 0x7f4670155130 2024-04-25T01:17:46.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.532+0000 7f466f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f467006e280 0x7f46701527f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:46.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.532+0000 7f466f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f467006e280 0x7f46701527f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46138/0 (socket says 172.21.15.90:46138) 2024-04-25T01:17:46.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.532+0000 7f466f400640 1 -- 172.21.15.90:0/1651312409 learned_addr learned my addr 172.21.15.90:0/1651312409 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:46.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.532+0000 7f4667e00640 1 --2- 172.21.15.90:0/1651312409 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4670155130 0x7f46700cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:46.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.532+0000 7f466fe00640 1 --2- 172.21.15.90:0/1651312409 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f467015b780 0x7f46700c9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:46.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.532+0000 7f466f400640 1 -- 172.21.15.90:0/1651312409 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4670155130 msgr2=0x7f46700cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:46.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.532+0000 7f466f400640 1 --2- 172.21.15.90:0/1651312409 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4670155130 0x7f46700cfa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.532+0000 7f466f400640 1 -- 172.21.15.90:0/1651312409 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f467015b780 msgr2=0x7f46700c9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:46.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.532+0000 7f466f400640 1 --2- 172.21.15.90:0/1651312409 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f467015b780 0x7f46700c9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.532+0000 7f466f400640 1 -- 172.21.15.90:0/1651312409 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46700cff60 con 0x7f467006e280 2024-04-25T01:17:46.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.533+0000 7f466f400640 1 --2- 172.21.15.90:0/1651312409 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f467006e280 0x7f46701527f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f466000c500 tx=0x7f466000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:46.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.533+0000 7f4667400640 1 -- 172.21.15.90:0/1651312409 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46600052d0 con 0x7f467006e280 2024-04-25T01:17:46.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.533+0000 7f4667400640 1 -- 172.21.15.90:0/1651312409 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4660005470 con 0x7f467006e280 2024-04-25T01:17:46.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.533+0000 7f4667400640 1 -- 172.21.15.90:0/1651312409 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4660012850 con 0x7f467006e280 2024-04-25T01:17:46.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.533+0000 7f4674a00640 1 -- 172.21.15.90:0/1651312409 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46700c9e70 con 0x7f467006e280 2024-04-25T01:17:46.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.533+0000 7f4674a00640 1 -- 172.21.15.90:0/1651312409 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46700c8e50 con 0x7f467006e280 2024-04-25T01:17:46.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.534+0000 7f4674a00640 1 -- 172.21.15.90:0/1651312409 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4670075350 con 0x7f467006e280 2024-04-25T01:17:46.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.535+0000 7f4667400640 1 -- 172.21.15.90:0/1651312409 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f46600129f0 con 0x7f467006e280 2024-04-25T01:17:46.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.536+0000 7f4667400640 1 --2- 172.21.15.90:0/1651312409 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f464c06be30 0x7f464c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:46.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.536+0000 7f4667400640 1 -- 172.21.15.90:0/1651312409 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f466008ae90 con 0x7f467006e280 2024-04-25T01:17:46.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.536+0000 7f4667e00640 1 --2- 172.21.15.90:0/1651312409 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f464c06be30 0x7f464c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:46.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.537+0000 7f4667e00640 1 --2- 172.21.15.90:0/1651312409 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f464c06be30 0x7f464c06e2f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f46701f7530 tx=0x7f4658008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:46.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.541+0000 7f4667400640 1 -- 172.21.15.90:0/1651312409 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f46700cff60 con 0x7f467006e280 2024-04-25T01:17:46.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.741+0000 7f4674a00640 1 -- 172.21.15.90:0/1651312409 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f467015b560 con 0x7f467006e280 2024-04-25T01:17:46.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.742+0000 7f4667400640 1 -- 172.21.15.90:0/1651312409 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) ==== 100+0+11 (secure 0 0 0) 0x7f4660059440 con 0x7f467006e280 2024-04-25T01:17:46.726 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-25T01:17:46.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.745+0000 7f4674a00640 1 -- 172.21.15.90:0/1651312409 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f464c06be30 msgr2=0x7f464c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:46.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.745+0000 7f4674a00640 1 --2- 172.21.15.90:0/1651312409 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f464c06be30 0x7f464c06e2f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f46701f7530 tx=0x7f4658008040 comp rx=0 tx=0).stop 2024-04-25T01:17:46.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.745+0000 7f4674a00640 1 -- 172.21.15.90:0/1651312409 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f467006e280 msgr2=0x7f46701527f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:46.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.745+0000 7f4674a00640 1 --2- 172.21.15.90:0/1651312409 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f467006e280 0x7f46701527f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f466000c500 tx=0x7f466000c9c0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.745+0000 7f4674a00640 1 -- 172.21.15.90:0/1651312409 shutdown_connections 2024-04-25T01:17:46.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.745+0000 7f4674a00640 1 --2- 172.21.15.90:0/1651312409 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f464c06be30 0x7f464c06e2f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.745+0000 7f4674a00640 1 --2- 172.21.15.90:0/1651312409 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f467015b780 0x7f46700c9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.745+0000 7f4674a00640 1 --2- 172.21.15.90:0/1651312409 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4670155130 0x7f46700cfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.745+0000 7f4674a00640 1 --2- 172.21.15.90:0/1651312409 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f467006e280 0x7f46701527f0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.745+0000 7f4674a00640 1 -- 172.21.15.90:0/1651312409 >> 172.21.15.90:0/1651312409 conn(0x7f467006b120 msgr2=0x7f467015c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:46.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.745+0000 7f4674a00640 1 -- 172.21.15.90:0/1651312409 shutdown_connections 2024-04-25T01:17:46.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.745+0000 7f4674a00640 1 -- 172.21.15.90:0/1651312409 wait complete. 2024-04-25T01:17:46.746 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:17:46.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.963+0000 7fe3ebe00640 1 Processor -- start 2024-04-25T01:17:46.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.963+0000 7fe3ebe00640 1 -- start start 2024-04-25T01:17:46.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.964+0000 7fe3ebe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe3e40722e0 0x7fe3e406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:46.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.964+0000 7fe3ebe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe3e415fc40 0x7fe3e41600c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:46.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.964+0000 7fe3ebe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe3e40cedc0 0x7fe3e40cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:46.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.964+0000 7fe3ebe00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fe3e4160aa0 con 0x7fe3e40cedc0 2024-04-25T01:17:46.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.964+0000 7fe3ebe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe3e406da40 con 0x7fe3e40722e0 2024-04-25T01:17:46.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.964+0000 7fe3ebe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe3e406cec0 con 0x7fe3e415fc40 2024-04-25T01:17:46.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.965+0000 7fe3e9e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe3e40cedc0 0x7fe3e40cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:46.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.965+0000 7fe3e9e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe3e40cedc0 0x7fe3e40cf250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46152/0 (socket says 172.21.15.90:46152) 2024-04-25T01:17:46.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.965+0000 7fe3e9e00640 1 -- 172.21.15.90:0/3806913036 learned_addr learned my addr 172.21.15.90:0/3806913036 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:46.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.965+0000 7fe3e9400640 1 --2- 172.21.15.90:0/3806913036 >> v2:172.21.15.103:3300/0 conn(0x7fe3e40722e0 0x7fe3e406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:46.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.965+0000 7fe3e8a00640 1 --2- 172.21.15.90:0/3806913036 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe3e415fc40 0x7fe3e41600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:46.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.965+0000 7fe3e9e00640 1 -- 172.21.15.90:0/3806913036 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe3e415fc40 msgr2=0x7fe3e41600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:46.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.965+0000 7fe3e9e00640 1 --2- 172.21.15.90:0/3806913036 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe3e415fc40 0x7fe3e41600c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.965+0000 7fe3e9e00640 1 -- 172.21.15.90:0/3806913036 >> v2:172.21.15.103:3300/0 conn(0x7fe3e40722e0 msgr2=0x7fe3e406ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:46.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.965+0000 7fe3e9e00640 1 --2- 172.21.15.90:0/3806913036 >> v2:172.21.15.103:3300/0 conn(0x7fe3e40722e0 0x7fe3e406ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.965+0000 7fe3e9e00640 1 -- 172.21.15.90:0/3806913036 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3e40cf790 con 0x7fe3e40cedc0 2024-04-25T01:17:46.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.965+0000 7fe3e8a00640 1 --2- 172.21.15.90:0/3806913036 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe3e415fc40 0x7fe3e41600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:46.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.966+0000 7fe3e9e00640 1 --2- 172.21.15.90:0/3806913036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3e40cedc0 0x7fe3e40cf250 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe3e40727b0 tx=0x7fe3dc02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3a3273c4a40c6fa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:46.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.967+0000 7fe3e3e00640 1 -- 172.21.15.90:0/3806913036 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc036070 con 0x7fe3e40cedc0 2024-04-25T01:17:46.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.967+0000 7fe3e3e00640 1 -- 172.21.15.90:0/3806913036 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3dc031600 con 0x7fe3e40cedc0 2024-04-25T01:17:46.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.967+0000 7fe3e3e00640 1 -- 172.21.15.90:0/3806913036 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc0319d0 con 0x7fe3e40cedc0 2024-04-25T01:17:46.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.967+0000 7fe3ebe00640 1 -- 172.21.15.90:0/3806913036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3e40cedc0 msgr2=0x7fe3e40cf250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:46.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.967+0000 7fe3ebe00640 1 --2- 172.21.15.90:0/3806913036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3e40cedc0 0x7fe3e40cf250 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe3e40727b0 tx=0x7fe3dc02a610 comp rx=0 tx=0).stop 2024-04-25T01:17:46.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.968+0000 7fe3ebe00640 1 -- 172.21.15.90:0/3806913036 shutdown_connections 2024-04-25T01:17:46.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.968+0000 7fe3ebe00640 1 --2- 172.21.15.90:0/3806913036 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe3e415fc40 0x7fe3e41600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.968+0000 7fe3ebe00640 1 --2- 172.21.15.90:0/3806913036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3e40cedc0 0x7fe3e40cf250 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.968+0000 7fe3ebe00640 1 --2- 172.21.15.90:0/3806913036 >> v2:172.21.15.103:3300/0 conn(0x7fe3e40722e0 0x7fe3e406ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.968+0000 7fe3ebe00640 1 -- 172.21.15.90:0/3806913036 >> 172.21.15.90:0/3806913036 conn(0x7fe3e406b720 msgr2=0x7fe3e40b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:46.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.968+0000 7fe3ebe00640 1 -- 172.21.15.90:0/3806913036 shutdown_connections 2024-04-25T01:17:46.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.968+0000 7fe3ebe00640 1 -- 172.21.15.90:0/3806913036 wait complete. 2024-04-25T01:17:46.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.969+0000 7fe3ebe00640 1 Processor -- start 2024-04-25T01:17:46.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.970+0000 7fe3ebe00640 1 -- start start 2024-04-25T01:17:46.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.970+0000 7fe3ebe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3e40722e0 0x7fe3e40b35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:46.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.970+0000 7fe3ebe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3e40cedc0 0x7fe3e4156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:46.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.970+0000 7fe3ebe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe3e415fc40 0x7fe3e40cc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:46.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.970+0000 7fe3ebe00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e406d3f0 con 0x7fe3e40722e0 2024-04-25T01:17:46.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.970+0000 7fe3ebe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e406e410 con 0x7fe3e40cedc0 2024-04-25T01:17:46.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.970+0000 7fe3ebe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe3e406d6d0 con 0x7fe3e415fc40 2024-04-25T01:17:46.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.971+0000 7fe3e9400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3e40722e0 0x7fe3e40b35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:46.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.971+0000 7fe3e9400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3e40722e0 0x7fe3e40b35f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46156/0 (socket says 172.21.15.90:46156) 2024-04-25T01:17:46.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.971+0000 7fe3e9400640 1 -- 172.21.15.90:0/1720504406 learned_addr learned my addr 172.21.15.90:0/1720504406 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:46.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.971+0000 7fe3e8a00640 1 --2- 172.21.15.90:0/1720504406 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3e40cedc0 0x7fe3e4156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:46.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.971+0000 7fe3e9400640 1 -- 172.21.15.90:0/1720504406 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe3e415fc40 msgr2=0x7fe3e40cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:46.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.971+0000 7fe3e9400640 1 --2- 172.21.15.90:0/1720504406 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe3e415fc40 0x7fe3e40cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.971+0000 7fe3e9400640 1 -- 172.21.15.90:0/1720504406 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3e40cedc0 msgr2=0x7fe3e4156d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:46.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.971+0000 7fe3e9400640 1 --2- 172.21.15.90:0/1720504406 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3e40cedc0 0x7fe3e4156d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:46.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.971+0000 7fe3e9400640 1 -- 172.21.15.90:0/1720504406 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3e4157560 con 0x7fe3e40722e0 2024-04-25T01:17:46.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.971+0000 7fe3e9400640 1 --2- 172.21.15.90:0/1720504406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3e40722e0 0x7fe3e40b35f0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe3d800d800 tx=0x7fe3d800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:46.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.972+0000 7fe3e1600640 1 -- 172.21.15.90:0/1720504406 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d8018030 con 0x7fe3e40722e0 2024-04-25T01:17:46.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.972+0000 7fe3e1600640 1 -- 172.21.15.90:0/1720504406 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3d80096a0 con 0x7fe3e40722e0 2024-04-25T01:17:46.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.972+0000 7fe3e1600640 1 -- 172.21.15.90:0/1720504406 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d8009870 con 0x7fe3e40722e0 2024-04-25T01:17:46.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.972+0000 7fe3ebe00640 1 -- 172.21.15.90:0/1720504406 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3e4157840 con 0x7fe3e40722e0 2024-04-25T01:17:46.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.972+0000 7fe3ebe00640 1 -- 172.21.15.90:0/1720504406 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3e41580e0 con 0x7fe3e40722e0 2024-04-25T01:17:46.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.973+0000 7fe3ebe00640 1 -- 172.21.15.90:0/1720504406 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3e4157a10 con 0x7fe3e40722e0 2024-04-25T01:17:46.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.974+0000 7fe3e1600640 1 -- 172.21.15.90:0/1720504406 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe3d801b070 con 0x7fe3e40722e0 2024-04-25T01:17:46.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.975+0000 7fe3e1600640 1 --2- 172.21.15.90:0/1720504406 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe3c406be30 0x7fe3c406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:46.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.975+0000 7fe3e1600640 1 -- 172.21.15.90:0/1720504406 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7fe3d808b4d0 con 0x7fe3e40722e0 2024-04-25T01:17:46.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.975+0000 7fe3e8a00640 1 --2- 172.21.15.90:0/1720504406 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe3c406be30 0x7fe3c406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:46.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.976+0000 7fe3e8a00640 1 --2- 172.21.15.90:0/1720504406 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe3c406be30 0x7fe3c406e2f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4159d10 tx=0x7fe3d0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:46.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:46.979+0000 7fe3e1600640 1 -- 172.21.15.90:0/1720504406 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe3d8059b00 con 0x7fe3e40722e0 2024-04-25T01:17:47.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.179+0000 7fe3ebe00640 1 -- 172.21.15.90:0/1720504406 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe3e4160020 con 0x7fe3e40722e0 2024-04-25T01:17:47.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.180+0000 7fe3e1600640 1 -- 172.21.15.90:0/1720504406 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) ==== 103+0+11 (secure 0 0 0) 0x7fe3e4160020 con 0x7fe3e40722e0 2024-04-25T01:17:47.164 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:17:47.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.183+0000 7fe3ebe00640 1 -- 172.21.15.90:0/1720504406 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe3c406be30 msgr2=0x7fe3c406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:47.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.183+0000 7fe3ebe00640 1 --2- 172.21.15.90:0/1720504406 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe3c406be30 0x7fe3c406e2f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4159d10 tx=0x7fe3d0008040 comp rx=0 tx=0).stop 2024-04-25T01:17:47.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.183+0000 7fe3ebe00640 1 -- 172.21.15.90:0/1720504406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3e40722e0 msgr2=0x7fe3e40b35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:47.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.183+0000 7fe3ebe00640 1 --2- 172.21.15.90:0/1720504406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3e40722e0 0x7fe3e40b35f0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe3d800d800 tx=0x7fe3d800dcc0 comp rx=0 tx=0).stop 2024-04-25T01:17:47.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.183+0000 7fe3ebe00640 1 -- 172.21.15.90:0/1720504406 shutdown_connections 2024-04-25T01:17:47.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.183+0000 7fe3ebe00640 1 --2- 172.21.15.90:0/1720504406 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe3c406be30 0x7fe3c406e2f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:47.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.183+0000 7fe3ebe00640 1 --2- 172.21.15.90:0/1720504406 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe3e415fc40 0x7fe3e40cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:47.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.183+0000 7fe3ebe00640 1 --2- 172.21.15.90:0/1720504406 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3e40cedc0 0x7fe3e4156d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:47.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.183+0000 7fe3ebe00640 1 --2- 172.21.15.90:0/1720504406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3e40722e0 0x7fe3e40b35f0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:47.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.183+0000 7fe3ebe00640 1 -- 172.21.15.90:0/1720504406 >> 172.21.15.90:0/1720504406 conn(0x7fe3e406b720 msgr2=0x7fe3e40adc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:47.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.183+0000 7fe3ebe00640 1 -- 172.21.15.90:0/1720504406 shutdown_connections 2024-04-25T01:17:47.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.184+0000 7fe3ebe00640 1 -- 172.21.15.90:0/1720504406 wait complete. 2024-04-25T01:17:47.185 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:17:47.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.403+0000 7f3ea2000640 1 Processor -- start 2024-04-25T01:17:47.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.403+0000 7f3ea2000640 1 -- start start 2024-04-25T01:17:47.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.404+0000 7f3ea2000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3e9c143940 0x7f3e9c06e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:47.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.405+0000 7f3ea2000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f3e9c15f400 0x7f3e9c06eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:47.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.405+0000 7f3ea2000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e9c070e60 con 0x7f3e9c143d20 2024-04-25T01:17:47.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.405+0000 7f3ea2000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f3e9c071140 con 0x7f3e9c143940 2024-04-25T01:17:47.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.405+0000 7f3ea2000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f3e9c071420 con 0x7f3e9c15f400 2024-04-25T01:17:47.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.405+0000 7f3e9b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3e9c143940 0x7f3e9c06e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:47.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.405+0000 7f3e9b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3e9c143940 0x7f3e9c06e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46168/0 (socket says 172.21.15.90:46168) 2024-04-25T01:17:47.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.405+0000 7f3e9b400640 1 -- 172.21.15.90:0/2025746611 learned_addr learned my addr 172.21.15.90:0/2025746611 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:47.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.405+0000 7f3e9b400640 1 -- 172.21.15.90:0/2025746611 >> v2:172.21.15.103:3300/0 conn(0x7f3e9c15f400 msgr2=0x7f3e9c06eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:47.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.405+0000 7f3e9b400640 1 --2- 172.21.15.90:0/2025746611 >> v2:172.21.15.103:3300/0 conn(0x7f3e9c15f400 0x7f3e9c06eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:47.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.405+0000 7f3e9b400640 1 -- 172.21.15.90:0/2025746611 >> v1:172.21.15.103:6789/0 conn(0x7f3e9c143d20 legacy=0x7f3e9c15ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:47.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.405+0000 7f3e9b400640 1 -- 172.21.15.90:0/2025746611 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e9c071700 con 0x7f3e9c143940 2024-04-25T01:17:47.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.406+0000 7f3e9b400640 1 --2- 172.21.15.90:0/2025746611 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e9c143940 0x7f3e9c06e650 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c009870 tx=0x7f3e8c02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3571207dd654988a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:47.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.407+0000 7f3e9a000640 1 -- 172.21.15.90:0/2025746611 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c037070 con 0x7f3e9c143940 2024-04-25T01:17:47.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.407+0000 7f3e9a000640 1 -- 172.21.15.90:0/2025746611 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e8c032440 con 0x7f3e9c143940 2024-04-25T01:17:47.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.407+0000 7f3e9a000640 1 -- 172.21.15.90:0/2025746611 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c032810 con 0x7f3e9c143940 2024-04-25T01:17:47.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.407+0000 7f3ea2000640 1 -- 172.21.15.90:0/2025746611 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e9c143940 msgr2=0x7f3e9c06e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:47.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.407+0000 7f3ea2000640 1 --2- 172.21.15.90:0/2025746611 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e9c143940 0x7f3e9c06e650 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c009870 tx=0x7f3e8c02a250 comp rx=0 tx=0).stop 2024-04-25T01:17:47.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.407+0000 7f3ea2000640 1 -- 172.21.15.90:0/2025746611 shutdown_connections 2024-04-25T01:17:47.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.407+0000 7f3ea2000640 1 --2- 172.21.15.90:0/2025746611 >> v2:172.21.15.103:3300/0 conn(0x7f3e9c15f400 0x7f3e9c06eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:47.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.407+0000 7f3ea2000640 1 --2- 172.21.15.90:0/2025746611 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e9c143940 0x7f3e9c06e650 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:47.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.407+0000 7f3ea2000640 1 -- 172.21.15.90:0/2025746611 >> 172.21.15.90:0/2025746611 conn(0x7f3e9c0b1820 msgr2=0x7f3e9c0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:47.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.407+0000 7f3ea2000640 1 -- 172.21.15.90:0/2025746611 shutdown_connections 2024-04-25T01:17:47.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.407+0000 7f3ea2000640 1 -- 172.21.15.90:0/2025746611 wait complete. 2024-04-25T01:17:47.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.409+0000 7f3ea2000640 1 Processor -- start 2024-04-25T01:17:47.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.409+0000 7f3ea2000640 1 -- start start 2024-04-25T01:17:47.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.410+0000 7f3ea2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e9c143940 0x7f3e9c12e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:47.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.410+0000 7f3ea2000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e9c143d20 0x7f3e9c15cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:47.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.410+0000 7f3ea2000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e9c15f400 0x7f3e9c13c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:47.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.410+0000 7f3ea2000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9c06d8d0 con 0x7f3e9c143940 2024-04-25T01:17:47.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.410+0000 7f3ea2000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9c06d750 con 0x7f3e9c143d20 2024-04-25T01:17:47.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.410+0000 7f3ea2000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3e9c06da50 con 0x7f3e9c15f400 2024-04-25T01:17:47.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.410+0000 7f3e9b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e9c143940 0x7f3e9c12e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:47.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.410+0000 7f3e9b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e9c143940 0x7f3e9c12e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46178/0 (socket says 172.21.15.90:46178) 2024-04-25T01:17:47.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.410+0000 7f3e9b400640 1 -- 172.21.15.90:0/336123524 learned_addr learned my addr 172.21.15.90:0/336123524 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:47.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.411+0000 7f3e9aa00640 1 --2- 172.21.15.90:0/336123524 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e9c143d20 0x7f3e9c15cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:47.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.411+0000 7f3e9b400640 1 -- 172.21.15.90:0/336123524 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e9c15f400 msgr2=0x7f3e9c13c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:47.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.411+0000 7f3e9b400640 1 --2- 172.21.15.90:0/336123524 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e9c15f400 0x7f3e9c13c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:47.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.411+0000 7f3e9b400640 1 -- 172.21.15.90:0/336123524 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e9c143d20 msgr2=0x7f3e9c15cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:47.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.411+0000 7f3e9b400640 1 --2- 172.21.15.90:0/336123524 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e9c143d20 0x7f3e9c15cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:47.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.411+0000 7f3e9b400640 1 -- 172.21.15.90:0/336123524 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e9c0ce570 con 0x7f3e9c143940 2024-04-25T01:17:47.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.411+0000 7f3e9aa00640 1 --2- 172.21.15.90:0/336123524 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e9c143d20 0x7f3e9c15cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:47.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.411+0000 7f3e9b400640 1 --2- 172.21.15.90:0/336123524 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e9c143940 0x7f3e9c12e340 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c009870 tx=0x7f3e8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:47.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.411+0000 7f3e93400640 1 -- 172.21.15.90:0/336123524 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c037040 con 0x7f3e9c143940 2024-04-25T01:17:47.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.411+0000 7f3e93400640 1 -- 172.21.15.90:0/336123524 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e8c004230 con 0x7f3e9c143940 2024-04-25T01:17:47.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.411+0000 7f3e93400640 1 -- 172.21.15.90:0/336123524 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c03c670 con 0x7f3e9c143940 2024-04-25T01:17:47.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.412+0000 7f3ea2000640 1 -- 172.21.15.90:0/336123524 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e9c0ce850 con 0x7f3e9c143940 2024-04-25T01:17:47.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.412+0000 7f3ea2000640 1 -- 172.21.15.90:0/336123524 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e9c13f0d0 con 0x7f3e9c143940 2024-04-25T01:17:47.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.413+0000 7f3ea2000640 1 -- 172.21.15.90:0/336123524 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e9c068fb0 con 0x7f3e9c143940 2024-04-25T01:17:47.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.414+0000 7f3e93400640 1 -- 172.21.15.90:0/336123524 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3e8c03a070 con 0x7f3e9c143940 2024-04-25T01:17:47.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.415+0000 7f3e93400640 1 --2- 172.21.15.90:0/336123524 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f3e7c06be30 0x7f3e7c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:47.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.415+0000 7f3e93400640 1 -- 172.21.15.90:0/336123524 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f3e8c0aba60 con 0x7f3e9c143940 2024-04-25T01:17:47.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.415+0000 7f3e9aa00640 1 --2- 172.21.15.90:0/336123524 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f3e7c06be30 0x7f3e7c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:47.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.416+0000 7f3e9aa00640 1 --2- 172.21.15.90:0/336123524 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f3e7c06be30 0x7f3e7c06e2f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3e88007fd0 tx=0x7f3e88009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:47.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.419+0000 7f3e93400640 1 -- 172.21.15.90:0/336123524 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f3e8c041020 con 0x7f3e9c143940 2024-04-25T01:17:47.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.648+0000 7f3ea2000640 1 -- 172.21.15.90:0/336123524 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e9c0728d0 con 0x7f3e9c143940 2024-04-25T01:17:47.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.649+0000 7f3e93400640 1 -- 172.21.15.90:0/336123524 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5180 (secure 0 0 0) 0x7f3e8c07a1b0 con 0x7f3e9c143940 2024-04-25T01:17:47.633 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:17:47.633 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6834/2605945289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":2605945289},{"type":"v1","addr":"172.21.15.103:6834","nonce":2605945289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6836/704639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":704639426},{"type":"v1","addr":"172.21.15.90:6836","nonce":704639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4245,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3477867755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6835","nonce":3477867755},{"type":"v1","addr":"172.21.15.90:6837","nonce":3477867755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:16:22.060018+0000","modified":"2024-04-25T01:16:50.405431+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6835/4291578198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6833","nonce":4291578198},{"type":"v1","addr":"172.21.15.103:6835","nonce":4291578198}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1}]} 2024-04-25T01:17:47.633 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 9 2024-04-25T01:17:47.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.651+0000 7f3ea2000640 1 -- 172.21.15.90:0/336123524 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f3e7c06be30 msgr2=0x7f3e7c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:47.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.651+0000 7f3ea2000640 1 --2- 172.21.15.90:0/336123524 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f3e7c06be30 0x7f3e7c06e2f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3e88007fd0 tx=0x7f3e88009040 comp rx=0 tx=0).stop 2024-04-25T01:17:47.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.651+0000 7f3ea2000640 1 -- 172.21.15.90:0/336123524 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e9c143940 msgr2=0x7f3e9c12e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:47.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.651+0000 7f3ea2000640 1 --2- 172.21.15.90:0/336123524 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e9c143940 0x7f3e9c12e340 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c009870 tx=0x7f3e8c0046c0 comp rx=0 tx=0).stop 2024-04-25T01:17:47.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.652+0000 7f3ea2000640 1 -- 172.21.15.90:0/336123524 shutdown_connections 2024-04-25T01:17:47.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.652+0000 7f3ea2000640 1 --2- 172.21.15.90:0/336123524 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f3e7c06be30 0x7f3e7c06e2f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:47.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.652+0000 7f3ea2000640 1 --2- 172.21.15.90:0/336123524 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3e9c15f400 0x7f3e9c13c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:47.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.652+0000 7f3ea2000640 1 --2- 172.21.15.90:0/336123524 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e9c143d20 0x7f3e9c15cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:47.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.652+0000 7f3ea2000640 1 --2- 172.21.15.90:0/336123524 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e9c143940 0x7f3e9c12e340 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:47.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.652+0000 7f3ea2000640 1 -- 172.21.15.90:0/336123524 >> 172.21.15.90:0/336123524 conn(0x7f3e9c0b1820 msgr2=0x7f3e9c0ae980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:47.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.652+0000 7f3ea2000640 1 -- 172.21.15.90:0/336123524 shutdown_connections 2024-04-25T01:17:47.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.652+0000 7f3ea2000640 1 -- 172.21.15.90:0/336123524 wait complete. 2024-04-25T01:17:47.652 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:17:47.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.866+0000 7fdf6b400640 1 Processor -- start 2024-04-25T01:17:47.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.866+0000 7fdf6b400640 1 -- start start 2024-04-25T01:17:47.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.867+0000 7fdf6b400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf64143d20 0x7fdf6415ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:47.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.867+0000 7fdf6b400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdf6406ed60 0x7fdf6415f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:47.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.867+0000 7fdf6b400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf6415f770 con 0x7fdf64143940 2024-04-25T01:17:47.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.867+0000 7fdf6b400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fdf64070e60 con 0x7fdf6406ed60 2024-04-25T01:17:47.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.867+0000 7fdf6b400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdf64071140 con 0x7fdf64143d20 2024-04-25T01:17:47.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.868+0000 7fdf68a00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fdf64143940 0x7fdf6406e650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.90:37378/0 (socket says 172.21.15.90:37378) 2024-04-25T01:17:47.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.868+0000 7fdf68a00640 1 -- 172.21.15.90:0/217097330 learned_addr learned my addr 172.21.15.90:0/217097330 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:47.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.868+0000 7fdf5be00640 1 --2- 172.21.15.90:0/217097330 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf64143d20 0x7fdf6415ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:47.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.868+0000 7fdf69400640 1 --2- 172.21.15.90:0/217097330 >> v2:172.21.15.103:3300/0 conn(0x7fdf6406ed60 0x7fdf6415f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:47.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.869+0000 7fdf63e00640 1 -- 172.21.15.90:0/217097330 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2991118935 0 0) 0x7fdf6415f770 con 0x7fdf64143940 2024-04-25T01:17:47.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.869+0000 7fdf63e00640 1 -- 172.21.15.90:0/217097330 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf48005730 con 0x7fdf64143940 2024-04-25T01:17:47.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.869+0000 7fdf5be00640 1 -- 172.21.15.90:0/217097330 >> v2:172.21.15.103:3300/0 conn(0x7fdf6406ed60 msgr2=0x7fdf6415f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:47.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.869+0000 7fdf5be00640 1 --2- 172.21.15.90:0/217097330 >> v2:172.21.15.103:3300/0 conn(0x7fdf6406ed60 0x7fdf6415f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:47.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.869+0000 7fdf5be00640 1 -- 172.21.15.90:0/217097330 >> v1:172.21.15.103:6789/0 conn(0x7fdf64143940 legacy=0x7fdf6406e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:47.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.869+0000 7fdf5be00640 1 -- 172.21.15.90:0/217097330 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf64071420 con 0x7fdf64143d20 2024-04-25T01:17:47.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.869+0000 7fdf69400640 1 --2- 172.21.15.90:0/217097330 >> v2:172.21.15.103:3300/0 conn(0x7fdf6406ed60 0x7fdf6415f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:17:47.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.870+0000 7fdf5be00640 1 --2- 172.21.15.90:0/217097330 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf64143d20 0x7fdf6415ecf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c0097d0 tx=0x7fdf4c02a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bfa3ef4a48df02c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:47.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.871+0000 7fdf63e00640 1 -- 172.21.15.90:0/217097330 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c037070 con 0x7fdf64143d20 2024-04-25T01:17:47.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.871+0000 7fdf63e00640 1 -- 172.21.15.90:0/217097330 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf4c032440 con 0x7fdf64143d20 2024-04-25T01:17:47.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.871+0000 7fdf63e00640 1 -- 172.21.15.90:0/217097330 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c0327f0 con 0x7fdf64143d20 2024-04-25T01:17:47.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.872+0000 7fdf6b400640 1 -- 172.21.15.90:0/217097330 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf64143d20 msgr2=0x7fdf6415ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:47.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.872+0000 7fdf6b400640 1 --2- 172.21.15.90:0/217097330 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf64143d20 0x7fdf6415ecf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c0097d0 tx=0x7fdf4c02a1c0 comp rx=0 tx=0).stop 2024-04-25T01:17:47.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.872+0000 7fdf6b400640 1 -- 172.21.15.90:0/217097330 shutdown_connections 2024-04-25T01:17:47.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.872+0000 7fdf6b400640 1 --2- 172.21.15.90:0/217097330 >> v2:172.21.15.103:3300/0 conn(0x7fdf6406ed60 0x7fdf6415f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:47.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.872+0000 7fdf6b400640 1 --2- 172.21.15.90:0/217097330 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf64143d20 0x7fdf6415ecf0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:47.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.872+0000 7fdf6b400640 1 -- 172.21.15.90:0/217097330 >> 172.21.15.90:0/217097330 conn(0x7fdf640b1820 msgr2=0x7fdf640b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:47.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.872+0000 7fdf6b400640 1 -- 172.21.15.90:0/217097330 shutdown_connections 2024-04-25T01:17:47.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.872+0000 7fdf6b400640 1 -- 172.21.15.90:0/217097330 wait complete. 2024-04-25T01:17:47.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.874+0000 7fdf6b400640 1 Processor -- start 2024-04-25T01:17:47.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.874+0000 7fdf6b400640 1 -- start start 2024-04-25T01:17:47.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.875+0000 7fdf6b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf6406ed60 0x7fdf6415a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:47.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.875+0000 7fdf6b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf64143940 0x7fdf640d0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:47.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.875+0000 7fdf6b400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf64143d20 0x7fdf6413f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:47.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.875+0000 7fdf6b400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6406d6c0 con 0x7fdf64143940 2024-04-25T01:17:47.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.875+0000 7fdf6b400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6406d540 con 0x7fdf6406ed60 2024-04-25T01:17:47.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.875+0000 7fdf6b400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdf6406d840 con 0x7fdf64143d20 2024-04-25T01:17:47.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.875+0000 7fdf5be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf64143940 0x7fdf640d0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:47.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.875+0000 7fdf5be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf64143940 0x7fdf640d0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46182/0 (socket says 172.21.15.90:46182) 2024-04-25T01:17:47.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.875+0000 7fdf5be00640 1 -- 172.21.15.90:0/3014759833 learned_addr learned my addr 172.21.15.90:0/3014759833 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:47.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.876+0000 7fdf68a00640 1 --2- 172.21.15.90:0/3014759833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf6406ed60 0x7fdf6415a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:47.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.876+0000 7fdf69400640 1 --2- 172.21.15.90:0/3014759833 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf64143d20 0x7fdf6413f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:47.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.876+0000 7fdf5be00640 1 -- 172.21.15.90:0/3014759833 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf64143d20 msgr2=0x7fdf6413f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:47.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.876+0000 7fdf5be00640 1 --2- 172.21.15.90:0/3014759833 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf64143d20 0x7fdf6413f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:47.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.876+0000 7fdf5be00640 1 -- 172.21.15.90:0/3014759833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf6406ed60 msgr2=0x7fdf6415a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:47.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.876+0000 7fdf5be00640 1 --2- 172.21.15.90:0/3014759833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf6406ed60 0x7fdf6415a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:47.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.876+0000 7fdf5be00640 1 -- 172.21.15.90:0/3014759833 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf64157790 con 0x7fdf64143940 2024-04-25T01:17:47.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.876+0000 7fdf69400640 1 --2- 172.21.15.90:0/3014759833 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf64143d20 0x7fdf6413f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:47.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.876+0000 7fdf5be00640 1 --2- 172.21.15.90:0/3014759833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf64143940 0x7fdf640d0200 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c000c00 tx=0x7fdf4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:47.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.877+0000 7fdf61600640 1 -- 172.21.15.90:0/3014759833 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c041020 con 0x7fdf64143940 2024-04-25T01:17:47.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.877+0000 7fdf61600640 1 -- 172.21.15.90:0/3014759833 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf4c03d6e0 con 0x7fdf64143940 2024-04-25T01:17:47.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.877+0000 7fdf61600640 1 -- 172.21.15.90:0/3014759833 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c037040 con 0x7fdf64143940 2024-04-25T01:17:47.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.877+0000 7fdf6b400640 1 -- 172.21.15.90:0/3014759833 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf64141430 con 0x7fdf64143940 2024-04-25T01:17:47.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.877+0000 7fdf6b400640 1 -- 172.21.15.90:0/3014759833 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf641419f0 con 0x7fdf64143940 2024-04-25T01:17:47.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.878+0000 7fdf6b400640 1 -- 172.21.15.90:0/3014759833 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf64068fb0 con 0x7fdf64143940 2024-04-25T01:17:47.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.879+0000 7fdf61600640 1 -- 172.21.15.90:0/3014759833 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fdf4c044020 con 0x7fdf64143940 2024-04-25T01:17:47.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.880+0000 7fdf61600640 1 --2- 172.21.15.90:0/3014759833 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdf4006be60 0x7fdf4006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:47.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.880+0000 7fdf61600640 1 -- 172.21.15.90:0/3014759833 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7fdf4c0ab630 con 0x7fdf64143940 2024-04-25T01:17:47.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.880+0000 7fdf68a00640 1 --2- 172.21.15.90:0/3014759833 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdf4006be60 0x7fdf4006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:47.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.881+0000 7fdf68a00640 1 --2- 172.21.15.90:0/3014759833 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdf4006be60 0x7fdf4006e320 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fdf540011e0 tx=0x7fdf54009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:47.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:47.885+0000 7fdf61600640 1 -- 172.21.15.90:0/3014759833 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fdf4c04b070 con 0x7fdf64143940 2024-04-25T01:17:48.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.083+0000 7fdf6b400640 1 -- 172.21.15.90:0/3014759833 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdf64141d90 con 0x7fdf64143940 2024-04-25T01:17:48.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.085+0000 7fdf61600640 1 -- 172.21.15.90:0/3014759833 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18576 (secure 0 0 0) 0x7fdf4c079be0 con 0x7fdf64143940 2024-04-25T01:17:48.069 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:17:48.069 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:17:09.346008+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:16:20.094512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:16:21.078268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:16:24.670642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.e","mappings":[{"from":0,"to":7}]},{"pgid":"4.f","mappings":[{"from":7,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:17:48.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.088+0000 7fdf6b400640 1 -- 172.21.15.90:0/3014759833 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdf4006be60 msgr2=0x7fdf4006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:48.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.088+0000 7fdf6b400640 1 --2- 172.21.15.90:0/3014759833 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdf4006be60 0x7fdf4006e320 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fdf540011e0 tx=0x7fdf54009040 comp rx=0 tx=0).stop 2024-04-25T01:17:48.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.088+0000 7fdf6b400640 1 -- 172.21.15.90:0/3014759833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf64143940 msgr2=0x7fdf640d0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:48.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.088+0000 7fdf6b400640 1 --2- 172.21.15.90:0/3014759833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf64143940 0x7fdf640d0200 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c000c00 tx=0x7fdf4c0046c0 comp rx=0 tx=0).stop 2024-04-25T01:17:48.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.088+0000 7fdf6b400640 1 -- 172.21.15.90:0/3014759833 shutdown_connections 2024-04-25T01:17:48.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.088+0000 7fdf6b400640 1 --2- 172.21.15.90:0/3014759833 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdf4006be60 0x7fdf4006e320 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:48.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.088+0000 7fdf6b400640 1 --2- 172.21.15.90:0/3014759833 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdf64143d20 0x7fdf6413f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:48.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.088+0000 7fdf6b400640 1 --2- 172.21.15.90:0/3014759833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf64143940 0x7fdf640d0200 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:48.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.088+0000 7fdf6b400640 1 --2- 172.21.15.90:0/3014759833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf6406ed60 0x7fdf6415a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:48.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.088+0000 7fdf6b400640 1 -- 172.21.15.90:0/3014759833 >> 172.21.15.90:0/3014759833 conn(0x7fdf640b1820 msgr2=0x7fdf640b3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:48.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.088+0000 7fdf6b400640 1 -- 172.21.15.90:0/3014759833 shutdown_connections 2024-04-25T01:17:48.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.088+0000 7fdf6b400640 1 -- 172.21.15.90:0/3014759833 wait complete. 2024-04-25T01:17:48.091 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T01:17:48.091 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:17:48.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.303+0000 7fe2bb200640 1 Processor -- start 2024-04-25T01:17:48.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.303+0000 7fe2bb200640 1 -- start start 2024-04-25T01:17:48.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.305+0000 7fe2bb200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2b406b120 0x7fe2b406b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:48.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.305+0000 7fe2bb200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe2b40ae730 0x7fe2b40cd900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:48.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.305+0000 7fe2bb200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe2b415ca10 0x7fe2b415cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:48.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.305+0000 7fe2bb200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fe2b40cef40 con 0x7fe2b415ca10 2024-04-25T01:17:48.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.305+0000 7fe2bb200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe2b415c580 con 0x7fe2b40ae730 2024-04-25T01:17:48.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.305+0000 7fe2bb200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe2b406ba40 con 0x7fe2b406b120 2024-04-25T01:17:48.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.305+0000 7fe2ba800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe2b415ca10 0x7fe2b415cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:48.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.305+0000 7fe2ba800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe2b415ca10 0x7fe2b415cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46190/0 (socket says 172.21.15.90:46190) 2024-04-25T01:17:48.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.305+0000 7fe2ba800640 1 -- 172.21.15.90:0/2733798512 learned_addr learned my addr 172.21.15.90:0/2733798512 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:48.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.305+0000 7fe2b9400640 1 --2- 172.21.15.90:0/2733798512 >> v2:172.21.15.103:3300/0 conn(0x7fe2b40ae730 0x7fe2b40cd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:48.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.305+0000 7fe2b9e00640 1 --2- 172.21.15.90:0/2733798512 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2b406b120 0x7fe2b406b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:48.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.306+0000 7fe2ba800640 1 -- 172.21.15.90:0/2733798512 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2b406b120 msgr2=0x7fe2b406b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:48.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.306+0000 7fe2ba800640 1 --2- 172.21.15.90:0/2733798512 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2b406b120 0x7fe2b406b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:48.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.306+0000 7fe2ba800640 1 -- 172.21.15.90:0/2733798512 >> v2:172.21.15.103:3300/0 conn(0x7fe2b40ae730 msgr2=0x7fe2b40cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:48.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.306+0000 7fe2ba800640 1 --2- 172.21.15.90:0/2733798512 >> v2:172.21.15.103:3300/0 conn(0x7fe2b40ae730 0x7fe2b40cd900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:48.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.306+0000 7fe2ba800640 1 -- 172.21.15.90:0/2733798512 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2b415c220 con 0x7fe2b415ca10 2024-04-25T01:17:48.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.307+0000 7fe2ba800640 1 --2- 172.21.15.90:0/2733798512 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe2b415ca10 0x7fe2b415cea0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fe2b40b2590 tx=0x7fe2b002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8326b806395e95ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:48.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.307+0000 7fe2b8a00640 1 -- 172.21.15.90:0/2733798512 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0036070 con 0x7fe2b415ca10 2024-04-25T01:17:48.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.307+0000 7fe2b8a00640 1 -- 172.21.15.90:0/2733798512 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2b0031600 con 0x7fe2b415ca10 2024-04-25T01:17:48.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.307+0000 7fe2b8a00640 1 -- 172.21.15.90:0/2733798512 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b00319d0 con 0x7fe2b415ca10 2024-04-25T01:17:48.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.308+0000 7fe2bb200640 1 -- 172.21.15.90:0/2733798512 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe2b415ca10 msgr2=0x7fe2b415cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:48.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.308+0000 7fe2bb200640 1 --2- 172.21.15.90:0/2733798512 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe2b415ca10 0x7fe2b415cea0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fe2b40b2590 tx=0x7fe2b002a6a0 comp rx=0 tx=0).stop 2024-04-25T01:17:48.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.308+0000 7fe2bb200640 1 -- 172.21.15.90:0/2733798512 shutdown_connections 2024-04-25T01:17:48.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.308+0000 7fe2bb200640 1 --2- 172.21.15.90:0/2733798512 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe2b415ca10 0x7fe2b415cea0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:48.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.308+0000 7fe2bb200640 1 --2- 172.21.15.90:0/2733798512 >> v2:172.21.15.103:3300/0 conn(0x7fe2b40ae730 0x7fe2b40cd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:48.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.308+0000 7fe2bb200640 1 --2- 172.21.15.90:0/2733798512 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2b406b120 0x7fe2b406b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:48.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.308+0000 7fe2bb200640 1 -- 172.21.15.90:0/2733798512 >> 172.21.15.90:0/2733798512 conn(0x7fe2b40b1660 msgr2=0x7fe2b40b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:48.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.308+0000 7fe2bb200640 1 -- 172.21.15.90:0/2733798512 shutdown_connections 2024-04-25T01:17:48.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.308+0000 7fe2bb200640 1 -- 172.21.15.90:0/2733798512 wait complete. 2024-04-25T01:17:48.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.310+0000 7fe2bb200640 1 Processor -- start 2024-04-25T01:17:48.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.310+0000 7fe2bb200640 1 -- start start 2024-04-25T01:17:48.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.310+0000 7fe2bb200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2b406b120 0x7fe2b4150dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:48.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.311+0000 7fe2bb200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe2b40ae730 0x7fe2b40adb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:48.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.311+0000 7fe2bb200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2b415ca10 0x7fe2b40ae0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:48.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.311+0000 7fe2bb200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b40af240 con 0x7fe2b40ae730 2024-04-25T01:17:48.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.311+0000 7fe2bb200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b40aef60 con 0x7fe2b415ca10 2024-04-25T01:17:48.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.311+0000 7fe2bb200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe2b4153080 con 0x7fe2b406b120 2024-04-25T01:17:48.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.311+0000 7fe2b9400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe2b40ae730 0x7fe2b40adb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:48.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.311+0000 7fe2b9400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe2b40ae730 0x7fe2b40adb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46204/0 (socket says 172.21.15.90:46204) 2024-04-25T01:17:48.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.311+0000 7fe2b9400640 1 -- 172.21.15.90:0/4219487052 learned_addr learned my addr 172.21.15.90:0/4219487052 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:48.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.311+0000 7fe2b9e00640 1 --2- 172.21.15.90:0/4219487052 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2b406b120 0x7fe2b4150dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:48.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.311+0000 7fe2b9400640 1 -- 172.21.15.90:0/4219487052 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2b406b120 msgr2=0x7fe2b4150dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:48.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.311+0000 7fe2b9400640 1 --2- 172.21.15.90:0/4219487052 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2b406b120 0x7fe2b4150dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:48.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.311+0000 7fe2b9400640 1 -- 172.21.15.90:0/4219487052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2b415ca10 msgr2=0x7fe2b40ae0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:48.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.311+0000 7fe2b9400640 1 --2- 172.21.15.90:0/4219487052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2b415ca10 0x7fe2b40ae0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:48.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.311+0000 7fe2b9400640 1 -- 172.21.15.90:0/4219487052 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2b4141dd0 con 0x7fe2b40ae730 2024-04-25T01:17:48.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.312+0000 7fe2b9400640 1 --2- 172.21.15.90:0/4219487052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe2b40ae730 0x7fe2b40adb90 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe2a400d800 tx=0x7fe2a400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:48.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.312+0000 7fe2ae000640 1 -- 172.21.15.90:0/4219487052 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a400b6c0 con 0x7fe2b40ae730 2024-04-25T01:17:48.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.312+0000 7fe2ae000640 1 -- 172.21.15.90:0/4219487052 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2a4019070 con 0x7fe2b40ae730 2024-04-25T01:17:48.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.312+0000 7fe2ae000640 1 -- 172.21.15.90:0/4219487052 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a400ba80 con 0x7fe2b40ae730 2024-04-25T01:17:48.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.312+0000 7fe2bb200640 1 -- 172.21.15.90:0/4219487052 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2b41f55b0 con 0x7fe2b40ae730 2024-04-25T01:17:48.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.312+0000 7fe2bb200640 1 -- 172.21.15.90:0/4219487052 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2b4151300 con 0x7fe2b40ae730 2024-04-25T01:17:48.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.313+0000 7fe2bb200640 1 -- 172.21.15.90:0/4219487052 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2b4074540 con 0x7fe2b40ae730 2024-04-25T01:17:48.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.314+0000 7fe2ae000640 1 -- 172.21.15.90:0/4219487052 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe2a400bc20 con 0x7fe2b40ae730 2024-04-25T01:17:48.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.315+0000 7fe2ae000640 1 --2- 172.21.15.90:0/4219487052 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe29806be30 0x7fe29806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:48.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.315+0000 7fe2ae000640 1 -- 172.21.15.90:0/4219487052 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7fe2a408b6b0 con 0x7fe2b40ae730 2024-04-25T01:17:48.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.315+0000 7fe2b9e00640 1 --2- 172.21.15.90:0/4219487052 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe29806be30 0x7fe29806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:48.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.316+0000 7fe2b9e00640 1 --2- 172.21.15.90:0/4219487052 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe29806be30 0x7fe29806e2f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe2a8006fd0 tx=0x7fe2a8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:48.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.319+0000 7fe2ae000640 1 -- 172.21.15.90:0/4219487052 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe2b4074540 con 0x7fe2b40ae730 2024-04-25T01:17:48.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.559+0000 7fe2bb200640 1 -- 172.21.15.90:0/4219487052 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2b4075350 con 0x7fe2b40ae730 2024-04-25T01:17:48.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.560+0000 7fe2ae000640 1 -- 172.21.15.90:0/4219487052 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5180 (secure 0 0 0) 0x7fe2a4059c50 con 0x7fe2b40ae730 2024-04-25T01:17:48.544 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:17:48.544 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6834/2605945289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":2605945289},{"type":"v1","addr":"172.21.15.103:6834","nonce":2605945289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6836/704639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":704639426},{"type":"v1","addr":"172.21.15.90:6836","nonce":704639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4245,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3477867755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6835","nonce":3477867755},{"type":"v1","addr":"172.21.15.90:6837","nonce":3477867755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:16:22.060018+0000","modified":"2024-04-25T01:16:50.405431+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6835/4291578198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6833","nonce":4291578198},{"type":"v1","addr":"172.21.15.103:6835","nonce":4291578198}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1}]} 2024-04-25T01:17:48.544 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 9 2024-04-25T01:17:48.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.563+0000 7fe2bb200640 1 -- 172.21.15.90:0/4219487052 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe29806be30 msgr2=0x7fe29806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:48.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.563+0000 7fe2bb200640 1 --2- 172.21.15.90:0/4219487052 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe29806be30 0x7fe29806e2f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe2a8006fd0 tx=0x7fe2a8008040 comp rx=0 tx=0).stop 2024-04-25T01:17:48.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.563+0000 7fe2bb200640 1 -- 172.21.15.90:0/4219487052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe2b40ae730 msgr2=0x7fe2b40adb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:48.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.563+0000 7fe2bb200640 1 --2- 172.21.15.90:0/4219487052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe2b40ae730 0x7fe2b40adb90 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe2a400d800 tx=0x7fe2a400dcc0 comp rx=0 tx=0).stop 2024-04-25T01:17:48.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.563+0000 7fe2bb200640 1 -- 172.21.15.90:0/4219487052 shutdown_connections 2024-04-25T01:17:48.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.563+0000 7fe2bb200640 1 --2- 172.21.15.90:0/4219487052 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe29806be30 0x7fe29806e2f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:48.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.563+0000 7fe2bb200640 1 --2- 172.21.15.90:0/4219487052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2b415ca10 0x7fe2b40ae0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:48.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.563+0000 7fe2bb200640 1 --2- 172.21.15.90:0/4219487052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe2b40ae730 0x7fe2b40adb90 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:48.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.563+0000 7fe2bb200640 1 --2- 172.21.15.90:0/4219487052 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe2b406b120 0x7fe2b4150dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:48.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.563+0000 7fe2bb200640 1 -- 172.21.15.90:0/4219487052 >> 172.21.15.90:0/4219487052 conn(0x7fe2b40b1660 msgr2=0x7fe2b40cbd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:48.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.563+0000 7fe2bb200640 1 -- 172.21.15.90:0/4219487052 shutdown_connections 2024-04-25T01:17:48.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.563+0000 7fe2bb200640 1 -- 172.21.15.90:0/4219487052 wait complete. 2024-04-25T01:17:48.565 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:17:48.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.781+0000 7f9acce00640 1 Processor -- start 2024-04-25T01:17:48.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.781+0000 7f9acce00640 1 -- start start 2024-04-25T01:17:48.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.782+0000 7f9acce00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9ac815ad40 0x7f9ac806b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:48.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.782+0000 7f9acce00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ac806d910 0x7f9ac80cd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:48.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.782+0000 7f9acce00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ac806ba90 con 0x7f9ac80cf520 2024-04-25T01:17:48.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.782+0000 7f9acce00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9ac806cb50 con 0x7f9ac815ad40 2024-04-25T01:17:48.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.782+0000 7f9acce00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9ac806ce30 con 0x7f9ac806d910 2024-04-25T01:17:48.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.782+0000 7f9abea00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f9ac80cf520 0x7f9ac80ae730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47898/0 (socket says 172.21.15.90:47898) 2024-04-25T01:17:48.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.782+0000 7f9abea00640 1 -- 172.21.15.90:0/2042578719 learned_addr learned my addr 172.21.15.90:0/2042578719 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:48.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.783+0000 7f9ac7400640 1 --2- 172.21.15.90:0/2042578719 >> v2:172.21.15.103:3300/0 conn(0x7f9ac815ad40 0x7f9ac806b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:48.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.783+0000 7f9ac7e00640 1 --2- 172.21.15.90:0/2042578719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ac806d910 0x7f9ac80cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:48.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.783+0000 7f9ac6a00640 1 -- 172.21.15.90:0/2042578719 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4171959556 0 0) 0x7f9ac806ba90 con 0x7f9ac80cf520 2024-04-25T01:17:48.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.783+0000 7f9ac6a00640 1 -- 172.21.15.90:0/2042578719 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9aac003520 con 0x7f9ac80cf520 2024-04-25T01:17:48.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.783+0000 7f9ac6a00640 1 -- 172.21.15.90:0/2042578719 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2398582874 0 0) 0x7f9aac003520 con 0x7f9ac80cf520 2024-04-25T01:17:48.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.783+0000 7f9ac6a00640 1 -- 172.21.15.90:0/2042578719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ac806d910 msgr2=0x7f9ac80cd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:48.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.783+0000 7f9ac6a00640 1 --2- 172.21.15.90:0/2042578719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ac806d910 0x7f9ac80cd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:48.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.783+0000 7f9ac6a00640 1 -- 172.21.15.90:0/2042578719 >> v2:172.21.15.103:3300/0 conn(0x7f9ac815ad40 msgr2=0x7f9ac806b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:48.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.783+0000 7f9ac6a00640 1 --2- 172.21.15.90:0/2042578719 >> v2:172.21.15.103:3300/0 conn(0x7f9ac815ad40 0x7f9ac806b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:48.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.783+0000 7f9ac6a00640 1 -- 172.21.15.90:0/2042578719 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac806d110 con 0x7f9ac80cf520 2024-04-25T01:17:48.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.783+0000 7f9ac6a00640 1 -- 172.21.15.90:0/2042578719 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f9ab0001670 con 0x7f9ac80cf520 2024-04-25T01:17:48.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.783+0000 7f9ac6a00640 1 -- 172.21.15.90:0/2042578719 >> v1:172.21.15.90:6789/0 conn(0x7f9ac80cf520 legacy=0x7f9ac80ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:48.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.783+0000 7f9ac6a00640 1 --2- 172.21.15.90:0/2042578719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9aac0039b0 0x7f9aac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:48.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.783+0000 7f9ac6a00640 1 --2- 172.21.15.90:0/2042578719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9aac0043a0 0x7f9aac004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:48.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.783+0000 7f9ac6a00640 1 --2- 172.21.15.90:0/2042578719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9aac004fa0 0x7f9aac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:48.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.783+0000 7f9ac6a00640 1 -- 172.21.15.90:0/2042578719 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac806cb50 con 0x7f9aac004fa0 2024-04-25T01:17:48.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.783+0000 7f9ac6a00640 1 -- 172.21.15.90:0/2042578719 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac806ce30 con 0x7f9aac0043a0 2024-04-25T01:17:48.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.784+0000 7f9ac6a00640 1 -- 172.21.15.90:0/2042578719 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9aac00bf30 con 0x7f9aac0039b0 2024-04-25T01:17:48.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.784+0000 7f9ac7e00640 1 --2- 172.21.15.90:0/2042578719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ac806d910 0x7f9ac80cd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:17:48.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.784+0000 7f9ac7e00640 1 --2- 172.21.15.90:0/2042578719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9aac004fa0 0x7f9aac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:48.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.784+0000 7f9abea00640 1 --2- 172.21.15.90:0/2042578719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9aac0043a0 0x7f9aac004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:48.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.784+0000 7f9ac7400640 1 --2- 172.21.15.90:0/2042578719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9aac0039b0 0x7f9aac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:48.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.784+0000 7f9ac7e00640 1 -- 172.21.15.90:0/2042578719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9aac0039b0 msgr2=0x7f9aac003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:48.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.784+0000 7f9ac7e00640 1 --2- 172.21.15.90:0/2042578719 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9aac0039b0 0x7f9aac003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:48.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.784+0000 7f9ac7e00640 1 -- 172.21.15.90:0/2042578719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9aac0043a0 msgr2=0x7f9aac004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:48.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.784+0000 7f9ac7e00640 1 --2- 172.21.15.90:0/2042578719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9aac0043a0 0x7f9aac004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:48.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.784+0000 7f9ac7e00640 1 -- 172.21.15.90:0/2042578719 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aac003520 con 0x7f9aac004fa0 2024-04-25T01:17:48.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.785+0000 7f9ac7e00640 1 --2- 172.21.15.90:0/2042578719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9aac004fa0 0x7f9aac00b9c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f9ac806dcf0 tx=0x7f9aac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42e9aad5e9f7ed72 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:48.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.785+0000 7f9ac6a00640 1 -- 172.21.15.90:0/2042578719 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac0030070 con 0x7f9aac004fa0 2024-04-25T01:17:48.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.785+0000 7f9ac6a00640 1 -- 172.21.15.90:0/2042578719 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ac002e070 con 0x7f9aac004fa0 2024-04-25T01:17:48.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.785+0000 7f9ac6a00640 1 -- 172.21.15.90:0/2042578719 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac0009b50 con 0x7f9aac004fa0 2024-04-25T01:17:48.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.786+0000 7f9acce00640 1 -- 172.21.15.90:0/2042578719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9aac004fa0 msgr2=0x7f9aac00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:48.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.786+0000 7f9acce00640 1 --2- 172.21.15.90:0/2042578719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9aac004fa0 0x7f9aac00b9c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f9ac806dcf0 tx=0x7f9aac005830 comp rx=0 tx=0).stop 2024-04-25T01:17:48.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.786+0000 7f9ac7e00640 1 -- 172.21.15.90:0/2042578719 reap_dead start 2024-04-25T01:17:48.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.786+0000 7f9acce00640 1 -- 172.21.15.90:0/2042578719 shutdown_connections 2024-04-25T01:17:48.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.786+0000 7f9acce00640 1 -- 172.21.15.90:0/2042578719 >> 172.21.15.90:0/2042578719 conn(0x7f9ac80b1660 msgr2=0x7f9ac80b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:48.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.786+0000 7f9acce00640 1 -- 172.21.15.90:0/2042578719 shutdown_connections 2024-04-25T01:17:48.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.786+0000 7f9acce00640 1 -- 172.21.15.90:0/2042578719 wait complete. 2024-04-25T01:17:48.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.788+0000 7f9acce00640 1 Processor -- start 2024-04-25T01:17:48.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.788+0000 7f9acce00640 1 -- start start 2024-04-25T01:17:48.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.789+0000 7f9acce00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9aac004fa0 0x7f9ac815d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:48.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.789+0000 7f9acce00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ac80af5b0 0x7f9ac814c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:48.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.789+0000 7f9acce00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ac815b120 0x7f9ac80cf920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:48.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.789+0000 7f9acce00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac806b7e0 con 0x7f9ac815b120 2024-04-25T01:17:48.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.789+0000 7f9acce00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac806b660 con 0x7f9ac80af5b0 2024-04-25T01:17:48.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.789+0000 7f9acce00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9ac80aee40 con 0x7f9aac004fa0 2024-04-25T01:17:48.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.789+0000 7f9ac7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ac815b120 0x7f9ac80cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:48.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.789+0000 7f9ac7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ac815b120 0x7f9ac80cf920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46216/0 (socket says 172.21.15.90:46216) 2024-04-25T01:17:48.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.789+0000 7f9ac7e00640 1 -- 172.21.15.90:0/1846355253 learned_addr learned my addr 172.21.15.90:0/1846355253 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:48.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.789+0000 7f9ac7e00640 1 -- 172.21.15.90:0/1846355253 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9aac004fa0 msgr2=0x7f9ac815d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:48.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.790+0000 7f9ac7e00640 1 --2- 172.21.15.90:0/1846355253 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9aac004fa0 0x7f9ac815d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:48.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.790+0000 7f9ac7e00640 1 -- 172.21.15.90:0/1846355253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ac80af5b0 msgr2=0x7f9ac814c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:48.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.790+0000 7f9abea00640 1 --2- 172.21.15.90:0/1846355253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ac80af5b0 0x7f9ac814c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:48.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.790+0000 7f9ac7e00640 1 --2- 172.21.15.90:0/1846355253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ac80af5b0 0x7f9ac814c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:48.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.790+0000 7f9ac7e00640 1 -- 172.21.15.90:0/1846355253 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac815d780 con 0x7f9ac815b120 2024-04-25T01:17:48.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.790+0000 7f9ac7e00640 1 --2- 172.21.15.90:0/1846355253 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ac815b120 0x7f9ac80cf920 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9ac002adb0 tx=0x7f9ac0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:48.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.790+0000 7f9abfe00640 1 -- 172.21.15.90:0/1846355253 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac0003e10 con 0x7f9ac815b120 2024-04-25T01:17:48.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.790+0000 7f9abfe00640 1 -- 172.21.15.90:0/1846355253 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ac0009490 con 0x7f9ac815b120 2024-04-25T01:17:48.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.790+0000 7f9abfe00640 1 -- 172.21.15.90:0/1846355253 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac0035460 con 0x7f9ac815b120 2024-04-25T01:17:48.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.790+0000 7f9acce00640 1 -- 172.21.15.90:0/1846355253 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ac80afa40 con 0x7f9ac815b120 2024-04-25T01:17:48.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.790+0000 7f9acce00640 1 -- 172.21.15.90:0/1846355253 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ac80b0000 con 0x7f9ac815b120 2024-04-25T01:17:48.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.791+0000 7f9acce00640 1 -- 172.21.15.90:0/1846355253 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ac8075350 con 0x7f9ac815b120 2024-04-25T01:17:48.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.792+0000 7f9abfe00640 1 -- 172.21.15.90:0/1846355253 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9ac0035600 con 0x7f9ac815b120 2024-04-25T01:17:48.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.793+0000 7f9abfe00640 1 --2- 172.21.15.90:0/1846355253 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9aa406be30 0x7f9aa406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:48.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.793+0000 7f9abfe00640 1 -- 172.21.15.90:0/1846355253 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f9ac00aaaf0 con 0x7f9ac815b120 2024-04-25T01:17:48.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.793+0000 7f9ac7400640 1 --2- 172.21.15.90:0/1846355253 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9aa406be30 0x7f9aa406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:48.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.794+0000 7f9ac7400640 1 --2- 172.21.15.90:0/1846355253 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9aa406be30 0x7f9aa406e2f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8009990 tx=0x7f9ab8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:48.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:48.797+0000 7f9abfe00640 1 -- 172.21.15.90:0/1846355253 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9ac0079130 con 0x7f9ac815b120 2024-04-25T01:17:49.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.030+0000 7f9acce00640 1 -- 172.21.15.90:0/1846355253 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ac80af2b0 con 0x7f9ac815b120 2024-04-25T01:17:49.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.031+0000 7f9abfe00640 1 -- 172.21.15.90:0/1846355253 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5180 (secure 0 0 0) 0x7f9ac0078a30 con 0x7f9ac815b120 2024-04-25T01:17:49.015 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:17:49.015 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6834/2605945289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":2605945289},{"type":"v1","addr":"172.21.15.103:6834","nonce":2605945289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6836/704639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":704639426},{"type":"v1","addr":"172.21.15.90:6836","nonce":704639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4245,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3477867755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6835","nonce":3477867755},{"type":"v1","addr":"172.21.15.90:6837","nonce":3477867755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:16:22.060018+0000","modified":"2024-04-25T01:16:50.405431+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6835/4291578198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6833","nonce":4291578198},{"type":"v1","addr":"172.21.15.103:6835","nonce":4291578198}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1}]} 2024-04-25T01:17:49.015 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 9 2024-04-25T01:17:49.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.033+0000 7f9acce00640 1 -- 172.21.15.90:0/1846355253 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9aa406be30 msgr2=0x7f9aa406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:49.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.033+0000 7f9acce00640 1 --2- 172.21.15.90:0/1846355253 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9aa406be30 0x7f9aa406e2f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8009990 tx=0x7f9ab8008040 comp rx=0 tx=0).stop 2024-04-25T01:17:49.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.033+0000 7f9acce00640 1 -- 172.21.15.90:0/1846355253 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ac815b120 msgr2=0x7f9ac80cf920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:49.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.033+0000 7f9acce00640 1 --2- 172.21.15.90:0/1846355253 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ac815b120 0x7f9ac80cf920 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9ac002adb0 tx=0x7f9ac0004ab0 comp rx=0 tx=0).stop 2024-04-25T01:17:49.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.034+0000 7f9acce00640 1 -- 172.21.15.90:0/1846355253 shutdown_connections 2024-04-25T01:17:49.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.034+0000 7f9acce00640 1 --2- 172.21.15.90:0/1846355253 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9aa406be30 0x7f9aa406e2f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:49.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.034+0000 7f9acce00640 1 --2- 172.21.15.90:0/1846355253 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ac815b120 0x7f9ac80cf920 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:49.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.034+0000 7f9acce00640 1 --2- 172.21.15.90:0/1846355253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ac80af5b0 0x7f9ac814c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:49.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.034+0000 7f9acce00640 1 --2- 172.21.15.90:0/1846355253 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9aac004fa0 0x7f9ac815d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:49.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.034+0000 7f9acce00640 1 -- 172.21.15.90:0/1846355253 >> 172.21.15.90:0/1846355253 conn(0x7f9ac80b1660 msgr2=0x7f9ac814fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:49.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.034+0000 7f9acce00640 1 -- 172.21.15.90:0/1846355253 shutdown_connections 2024-04-25T01:17:49.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.034+0000 7f9acce00640 1 -- 172.21.15.90:0/1846355253 wait complete. 2024-04-25T01:17:49.034 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:17:49.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.243+0000 7f7cb7e00640 1 Processor -- start 2024-04-25T01:17:49.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.243+0000 7f7cb7e00640 1 -- start start 2024-04-25T01:17:49.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.245+0000 7f7cb7e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7cb806b120 0x7f7cb806b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:49.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.245+0000 7f7cb7e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cb806ba40 con 0x7f7cb815ca10 2024-04-25T01:17:49.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.245+0000 7f7cb7e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cb815c220 con 0x7f7cb80ae730 2024-04-25T01:17:49.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.245+0000 7f7cb7e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f7cb80cef40 con 0x7f7cb806b120 2024-04-25T01:17:49.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.245+0000 7f7cb6a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7cb806b120 0x7f7cb806b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:49.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.245+0000 7f7cb7400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f7cb815ca10 0x7f7cb80cc030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47902/0 (socket says 172.21.15.90:47902) 2024-04-25T01:17:49.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.245+0000 7f7cb6a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7cb806b120 0x7f7cb806b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46218/0 (socket says 172.21.15.90:46218) 2024-04-25T01:17:49.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.245+0000 7f7cb7400640 1 -- 172.21.15.90:0/2330118294 learned_addr learned my addr 172.21.15.90:0/2330118294 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:49.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.245+0000 7f7cb6a00640 1 -- 172.21.15.90:0/2330118294 >> v1:172.21.15.103:6789/0 conn(0x7f7cb80ae730 legacy=0x7f7cb80cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:49.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.245+0000 7f7cb6a00640 1 -- 172.21.15.90:0/2330118294 >> v1:172.21.15.90:6789/0 conn(0x7f7cb815ca10 legacy=0x7f7cb80cc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:49.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.245+0000 7f7cb6a00640 1 -- 172.21.15.90:0/2330118294 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cb80aec80 con 0x7f7cb806b120 2024-04-25T01:17:49.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.246+0000 7f7cb6a00640 1 --2- 172.21.15.90:0/2330118294 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7cb806b120 0x7f7cb806b500 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f7ca80097a0 tx=0x7f7ca802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2585cf0e51a3313 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:49.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.247+0000 7f7cb6000640 1 -- 172.21.15.90:0/2330118294 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca8037070 con 0x7f7cb806b120 2024-04-25T01:17:49.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.247+0000 7f7cb6000640 1 -- 172.21.15.90:0/2330118294 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ca8032440 con 0x7f7cb806b120 2024-04-25T01:17:49.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.247+0000 7f7cb6000640 1 -- 172.21.15.90:0/2330118294 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca8032810 con 0x7f7cb806b120 2024-04-25T01:17:49.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.247+0000 7f7cb7e00640 1 -- 172.21.15.90:0/2330118294 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7cb806b120 msgr2=0x7f7cb806b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:49.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.247+0000 7f7cb7e00640 1 --2- 172.21.15.90:0/2330118294 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7cb806b120 0x7f7cb806b500 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f7ca80097a0 tx=0x7f7ca802a1e0 comp rx=0 tx=0).stop 2024-04-25T01:17:49.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.247+0000 7f7cb7e00640 1 -- 172.21.15.90:0/2330118294 shutdown_connections 2024-04-25T01:17:49.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.247+0000 7f7cb7e00640 1 --2- 172.21.15.90:0/2330118294 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7cb806b120 0x7f7cb806b500 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:49.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.247+0000 7f7cb7e00640 1 -- 172.21.15.90:0/2330118294 >> 172.21.15.90:0/2330118294 conn(0x7f7cb80b1660 msgr2=0x7f7cb80b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:49.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.248+0000 7f7cb7e00640 1 -- 172.21.15.90:0/2330118294 shutdown_connections 2024-04-25T01:17:49.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.248+0000 7f7cb7e00640 1 -- 172.21.15.90:0/2330118294 wait complete. 2024-04-25T01:17:49.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.249+0000 7f7cb7e00640 1 Processor -- start 2024-04-25T01:17:49.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.249+0000 7f7cb7e00640 1 -- start start 2024-04-25T01:17:49.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.250+0000 7f7cb7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7cb806b120 0x7f7cb81450a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:49.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.250+0000 7f7cb7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7cb80ae730 0x7f7cb8157d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:49.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.250+0000 7f7cb7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7cb815ca10 0x7f7cb814d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:49.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.250+0000 7f7cb7e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb80af240 con 0x7f7cb80ae730 2024-04-25T01:17:49.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.250+0000 7f7cb7e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb815c580 con 0x7f7cb806b120 2024-04-25T01:17:49.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.250+0000 7f7cb7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7cb8153080 con 0x7f7cb815ca10 2024-04-25T01:17:49.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.250+0000 7f7cafe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7cb80ae730 0x7f7cb8157d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:49.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.250+0000 7f7cafe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7cb80ae730 0x7f7cb8157d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46230/0 (socket says 172.21.15.90:46230) 2024-04-25T01:17:49.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.250+0000 7f7cafe00640 1 -- 172.21.15.90:0/4249338893 learned_addr learned my addr 172.21.15.90:0/4249338893 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:49.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.251+0000 7f7cafe00640 1 -- 172.21.15.90:0/4249338893 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7cb815ca10 msgr2=0x7f7cb814d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:49.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.251+0000 7f7cafe00640 1 --2- 172.21.15.90:0/4249338893 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7cb815ca10 0x7f7cb814d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:49.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.251+0000 7f7cafe00640 1 -- 172.21.15.90:0/4249338893 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7cb806b120 msgr2=0x7f7cb81450a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:49.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.251+0000 7f7cafe00640 1 --2- 172.21.15.90:0/4249338893 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7cb806b120 0x7f7cb81450a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:49.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.251+0000 7f7cafe00640 1 -- 172.21.15.90:0/4249338893 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cb8156ca0 con 0x7f7cb80ae730 2024-04-25T01:17:49.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.251+0000 7f7cafe00640 1 --2- 172.21.15.90:0/4249338893 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7cb80ae730 0x7f7cb8157d40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f7ca000b3e0 tx=0x7f7ca000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:49.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.251+0000 7f7caea00640 1 -- 172.21.15.90:0/4249338893 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca0015020 con 0x7f7cb80ae730 2024-04-25T01:17:49.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.251+0000 7f7caea00640 1 -- 172.21.15.90:0/4249338893 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ca0011850 con 0x7f7cb80ae730 2024-04-25T01:17:49.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.251+0000 7f7caea00640 1 -- 172.21.15.90:0/4249338893 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca0011c40 con 0x7f7cb80ae730 2024-04-25T01:17:49.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.251+0000 7f7cb7e00640 1 -- 172.21.15.90:0/4249338893 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cb8156f80 con 0x7f7cb80ae730 2024-04-25T01:17:49.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.251+0000 7f7cb7e00640 1 -- 172.21.15.90:0/4249338893 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cb8157540 con 0x7f7cb80ae730 2024-04-25T01:17:49.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.253+0000 7f7cb7e00640 1 -- 172.21.15.90:0/4249338893 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cb8075350 con 0x7f7cb80ae730 2024-04-25T01:17:49.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.253+0000 7f7caea00640 1 -- 172.21.15.90:0/4249338893 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7ca0018020 con 0x7f7cb80ae730 2024-04-25T01:17:49.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.254+0000 7f7caea00640 1 --2- 172.21.15.90:0/4249338893 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7c9406be60 0x7f7c9406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:49.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.254+0000 7f7caea00640 1 -- 172.21.15.90:0/4249338893 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7f7ca008b1d0 con 0x7f7cb80ae730 2024-04-25T01:17:49.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.254+0000 7f7cb6a00640 1 --2- 172.21.15.90:0/4249338893 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7c9406be60 0x7f7c9406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:49.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.255+0000 7f7cb6a00640 1 --2- 172.21.15.90:0/4249338893 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7c9406be60 0x7f7c9406e320 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8034000 tx=0x7f7ca802c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:49.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.259+0000 7f7caea00640 1 -- 172.21.15.90:0/4249338893 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7ca001f020 con 0x7f7cb80ae730 2024-04-25T01:17:49.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.443+0000 7f7cb7e00640 1 -- 172.21.15.90:0/4249338893 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7cb806ccd0 con 0x7f7cb80ae730 2024-04-25T01:17:49.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.446+0000 7f7caea00640 1 -- 172.21.15.90:0/4249338893 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18576 (secure 0 0 0) 0x7f7ca00172c0 con 0x7f7cb80ae730 2024-04-25T01:17:49.430 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:17:49.430 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:17:09.346008+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:16:20.094512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:16:21.078268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:16:24.670642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.e","mappings":[{"from":0,"to":7}]},{"pgid":"4.f","mappings":[{"from":7,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:17:49.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.449+0000 7f7cb7e00640 1 -- 172.21.15.90:0/4249338893 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7c9406be60 msgr2=0x7f7c9406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:49.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.449+0000 7f7cb7e00640 1 --2- 172.21.15.90:0/4249338893 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7c9406be60 0x7f7c9406e320 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8034000 tx=0x7f7ca802c000 comp rx=0 tx=0).stop 2024-04-25T01:17:49.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.449+0000 7f7cb7e00640 1 -- 172.21.15.90:0/4249338893 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7cb80ae730 msgr2=0x7f7cb8157d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:49.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.449+0000 7f7cb7e00640 1 --2- 172.21.15.90:0/4249338893 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7cb80ae730 0x7f7cb8157d40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f7ca000b3e0 tx=0x7f7ca000b8a0 comp rx=0 tx=0).stop 2024-04-25T01:17:49.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.449+0000 7f7cb7e00640 1 -- 172.21.15.90:0/4249338893 shutdown_connections 2024-04-25T01:17:49.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.449+0000 7f7cb7e00640 1 --2- 172.21.15.90:0/4249338893 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7c9406be60 0x7f7c9406e320 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:49.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.449+0000 7f7cb7e00640 1 --2- 172.21.15.90:0/4249338893 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7cb815ca10 0x7f7cb814d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:49.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.449+0000 7f7cb7e00640 1 --2- 172.21.15.90:0/4249338893 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7cb80ae730 0x7f7cb8157d40 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:49.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.449+0000 7f7cb7e00640 1 --2- 172.21.15.90:0/4249338893 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7cb806b120 0x7f7cb81450a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:49.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.449+0000 7f7cb7e00640 1 -- 172.21.15.90:0/4249338893 >> 172.21.15.90:0/4249338893 conn(0x7f7cb80b1660 msgr2=0x7f7cb814e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:49.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.450+0000 7f7cb7e00640 1 -- 172.21.15.90:0/4249338893 shutdown_connections 2024-04-25T01:17:49.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.450+0000 7f7cb7e00640 1 -- 172.21.15.90:0/4249338893 wait complete. 2024-04-25T01:17:49.450 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T01:17:49.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.657+0000 7ff228800640 1 Processor -- start 2024-04-25T01:17:49.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.657+0000 7ff228800640 1 -- start start 2024-04-25T01:17:49.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.658+0000 7ff228800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff22015f9b0 0x7ff22015fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:49.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.658+0000 7ff228800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2200b4240 con 0x7ff220145590 2024-04-25T01:17:49.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.658+0000 7ff228800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2200b39a0 con 0x7ff22006d270 2024-04-25T01:17:49.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.658+0000 7ff228800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff220160830 con 0x7ff22015f9b0 2024-04-25T01:17:49.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.658+0000 7ff225e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff220145590 0x7ff2200b4a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47908/0 (socket says 172.21.15.90:47908) 2024-04-25T01:17:49.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.658+0000 7ff225e00640 1 -- 172.21.15.90:0/3495713845 learned_addr learned my addr 172.21.15.90:0/3495713845 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:49.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.659+0000 7ff224a00640 1 -- 172.21.15.90:0/3495713845 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1865978422 0 0) 0x7ff2200b4240 con 0x7ff220145590 2024-04-25T01:17:49.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.659+0000 7ff224a00640 1 -- 172.21.15.90:0/3495713845 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff208003520 con 0x7ff220145590 2024-04-25T01:17:49.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.659+0000 7ff225400640 1 --2- 172.21.15.90:0/3495713845 >> v2:172.21.15.103:3300/0 conn(0x7ff22015f9b0 0x7ff22015fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:49.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.659+0000 7ff224a00640 1 -- 172.21.15.90:0/3495713845 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3607415277 0 0) 0x7ff208003520 con 0x7ff220145590 2024-04-25T01:17:49.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.659+0000 7ff224a00640 1 -- 172.21.15.90:0/3495713845 >> v2:172.21.15.103:3300/0 conn(0x7ff22015f9b0 msgr2=0x7ff22015fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:49.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.659+0000 7ff224a00640 1 --2- 172.21.15.90:0/3495713845 >> v2:172.21.15.103:3300/0 conn(0x7ff22015f9b0 0x7ff22015fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:49.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.659+0000 7ff224a00640 1 -- 172.21.15.90:0/3495713845 >> v1:172.21.15.103:6789/0 conn(0x7ff22006d270 legacy=0x7ff22006d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:49.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.659+0000 7ff224a00640 1 -- 172.21.15.90:0/3495713845 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2200b3c80 con 0x7ff220145590 2024-04-25T01:17:49.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.659+0000 7ff224a00640 1 -- 172.21.15.90:0/3495713845 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7ff214001670 con 0x7ff220145590 2024-04-25T01:17:49.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.659+0000 7ff224a00640 1 -- 172.21.15.90:0/3495713845 >> v1:172.21.15.90:6789/0 conn(0x7ff220145590 legacy=0x7ff2200b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:49.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.659+0000 7ff224a00640 1 --2- 172.21.15.90:0/3495713845 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2080039b0 0x7ff208003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:49.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.659+0000 7ff224a00640 1 --2- 172.21.15.90:0/3495713845 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff208004800 0x7ff208004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:49.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.659+0000 7ff224a00640 1 --2- 172.21.15.90:0/3495713845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2080051c0 0x7ff20800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:49.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.659+0000 7ff224a00640 1 -- 172.21.15.90:0/3495713845 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff20800c0c0 con 0x7ff2080051c0 2024-04-25T01:17:49.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.659+0000 7ff224a00640 1 -- 172.21.15.90:0/3495713845 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff20800c270 con 0x7ff2080039b0 2024-04-25T01:17:49.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.659+0000 7ff224a00640 1 -- 172.21.15.90:0/3495713845 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff20800c470 con 0x7ff208004800 2024-04-25T01:17:49.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.660+0000 7ff226800640 1 --2- 172.21.15.90:0/3495713845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2080051c0 0x7ff20800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:49.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.660+0000 7ff225e00640 1 --2- 172.21.15.90:0/3495713845 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2080039b0 0x7ff208003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:49.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.660+0000 7ff226800640 1 -- 172.21.15.90:0/3495713845 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff208004800 msgr2=0x7ff208004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:49.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.660+0000 7ff226800640 1 --2- 172.21.15.90:0/3495713845 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff208004800 0x7ff208004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:49.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.660+0000 7ff226800640 1 -- 172.21.15.90:0/3495713845 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2080039b0 msgr2=0x7ff208003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:49.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.660+0000 7ff226800640 1 --2- 172.21.15.90:0/3495713845 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2080039b0 0x7ff208003db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:49.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.660+0000 7ff226800640 1 -- 172.21.15.90:0/3495713845 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff208003520 con 0x7ff2080051c0 2024-04-25T01:17:49.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.661+0000 7ff226800640 1 -- 172.21.15.90:0/3495713845 reap_dead start 2024-04-25T01:17:49.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.661+0000 7ff226800640 1 --2- 172.21.15.90:0/3495713845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2080051c0 0x7ff20800b9c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff21c004480 tx=0x7ff208005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50fe4ea68bc3494e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:49.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.661+0000 7ff224a00640 1 -- 172.21.15.90:0/3495713845 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21c004080 con 0x7ff2080051c0 2024-04-25T01:17:49.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.661+0000 7ff224a00640 1 -- 172.21.15.90:0/3495713845 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff21c02ad80 con 0x7ff2080051c0 2024-04-25T01:17:49.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.661+0000 7ff224a00640 1 -- 172.21.15.90:0/3495713845 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21c006d90 con 0x7ff2080051c0 2024-04-25T01:17:49.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.662+0000 7ff228800640 1 -- 172.21.15.90:0/3495713845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2080051c0 msgr2=0x7ff20800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:49.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.662+0000 7ff228800640 1 --2- 172.21.15.90:0/3495713845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2080051c0 0x7ff20800b9c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff21c004480 tx=0x7ff208005830 comp rx=0 tx=0).stop 2024-04-25T01:17:49.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.662+0000 7ff228800640 1 -- 172.21.15.90:0/3495713845 shutdown_connections 2024-04-25T01:17:49.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.662+0000 7ff228800640 1 --2- 172.21.15.90:0/3495713845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2080051c0 0x7ff20800b9c0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:49.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.662+0000 7ff228800640 1 -- 172.21.15.90:0/3495713845 >> 172.21.15.90:0/3495713845 conn(0x7ff2200b1820 msgr2=0x7ff2200b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:49.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.662+0000 7ff228800640 1 -- 172.21.15.90:0/3495713845 shutdown_connections 2024-04-25T01:17:49.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.662+0000 7ff228800640 1 -- 172.21.15.90:0/3495713845 wait complete. 2024-04-25T01:17:49.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.664+0000 7ff228800640 1 Processor -- start 2024-04-25T01:17:49.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.664+0000 7ff228800640 1 -- start start 2024-04-25T01:17:49.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.665+0000 7ff228800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2080051c0 0x7ff22015b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:49.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.665+0000 7ff228800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff220142eb0 0x7ff2201e5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:49.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.665+0000 7ff228800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff220143310 0x7ff2201e59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:49.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.665+0000 7ff228800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff220145970 con 0x7ff220142eb0 2024-04-25T01:17:49.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.665+0000 7ff228800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff22006cde0 con 0x7ff220143310 2024-04-25T01:17:49.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.665+0000 7ff228800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff2200b3530 con 0x7ff2080051c0 2024-04-25T01:17:49.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.665+0000 7ff225400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff220142eb0 0x7ff2201e5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:49.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.665+0000 7ff225400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff220142eb0 0x7ff2201e5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46250/0 (socket says 172.21.15.90:46250) 2024-04-25T01:17:49.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.665+0000 7ff225400640 1 -- 172.21.15.90:0/4050256116 learned_addr learned my addr 172.21.15.90:0/4050256116 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:49.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.665+0000 7ff225400640 1 -- 172.21.15.90:0/4050256116 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2080051c0 msgr2=0x7ff22015b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:49.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.665+0000 7ff225400640 1 --2- 172.21.15.90:0/4050256116 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2080051c0 0x7ff22015b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:49.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.665+0000 7ff225400640 1 -- 172.21.15.90:0/4050256116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff220143310 msgr2=0x7ff2201e59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:49.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.665+0000 7ff225400640 1 --2- 172.21.15.90:0/4050256116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff220143310 0x7ff2201e59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:49.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.665+0000 7ff225400640 1 -- 172.21.15.90:0/4050256116 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff220143890 con 0x7ff220142eb0 2024-04-25T01:17:49.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.665+0000 7ff225400640 1 --2- 172.21.15.90:0/4050256116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff220142eb0 0x7ff2201e5490 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ff21000d800 tx=0x7ff21000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:49.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.666+0000 7ff21a000640 1 -- 172.21.15.90:0/4050256116 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff210018020 con 0x7ff220142eb0 2024-04-25T01:17:49.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.666+0000 7ff21a000640 1 -- 172.21.15.90:0/4050256116 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff21000b660 con 0x7ff220142eb0 2024-04-25T01:17:49.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.666+0000 7ff21a000640 1 -- 172.21.15.90:0/4050256116 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21000ba30 con 0x7ff220142eb0 2024-04-25T01:17:49.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.666+0000 7ff228800640 1 -- 172.21.15.90:0/4050256116 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff220143b70 con 0x7ff220142eb0 2024-04-25T01:17:49.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.666+0000 7ff228800640 1 -- 172.21.15.90:0/4050256116 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff220144130 con 0x7ff220142eb0 2024-04-25T01:17:49.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.667+0000 7ff228800640 1 -- 172.21.15.90:0/4050256116 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff220068fb0 con 0x7ff220142eb0 2024-04-25T01:17:49.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.668+0000 7ff21a000640 1 -- 172.21.15.90:0/4050256116 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff21001b020 con 0x7ff220142eb0 2024-04-25T01:17:49.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.669+0000 7ff21a000640 1 --2- 172.21.15.90:0/4050256116 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff20406be30 0x7ff20406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:49.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.669+0000 7ff21a000640 1 -- 172.21.15.90:0/4050256116 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6590+0+0 (secure 0 0 0) 0x7ff21008b360 con 0x7ff220142eb0 2024-04-25T01:17:49.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.669+0000 7ff225e00640 1 --2- 172.21.15.90:0/4050256116 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff20406be30 0x7ff20406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:49.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.670+0000 7ff225e00640 1 --2- 172.21.15.90:0/4050256116 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff20406be30 0x7ff20406e2f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff2140017e0 tx=0x7ff214009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:49.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.673+0000 7ff21a000640 1 -- 172.21.15.90:0/4050256116 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff210022030 con 0x7ff220142eb0 2024-04-25T01:17:49.847 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:17:49.858 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:17:49.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.916+0000 7ff228800640 1 -- 172.21.15.90:0/4050256116 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7ff2200b20f0 con 0x7ff220142eb0 2024-04-25T01:17:49.901 INFO:tasks.ceph.mon.a.smithi090.stderr:2024-04-25T01:17:49.918+0000 7f0aa5400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T01:17:49.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.920+0000 7ff21a000640 1 -- 172.21.15.90:0/4050256116 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v10) ==== 161+0+0 (secure 0 0 0) 0x7ff210059a30 con 0x7ff220142eb0 2024-04-25T01:17:49.903 INFO:teuthology.orchestra.run.smithi090.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T01:17:49.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.923+0000 7ff228800640 1 -- 172.21.15.90:0/4050256116 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff20406be30 msgr2=0x7ff20406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:49.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.923+0000 7ff228800640 1 --2- 172.21.15.90:0/4050256116 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff20406be30 0x7ff20406e2f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff2140017e0 tx=0x7ff214009040 comp rx=0 tx=0).stop 2024-04-25T01:17:49.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.923+0000 7ff228800640 1 -- 172.21.15.90:0/4050256116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff220142eb0 msgr2=0x7ff2201e5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:49.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.923+0000 7ff228800640 1 --2- 172.21.15.90:0/4050256116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff220142eb0 0x7ff2201e5490 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ff21000d800 tx=0x7ff21000dcc0 comp rx=0 tx=0).stop 2024-04-25T01:17:49.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.923+0000 7ff228800640 1 -- 172.21.15.90:0/4050256116 shutdown_connections 2024-04-25T01:17:49.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.923+0000 7ff228800640 1 --2- 172.21.15.90:0/4050256116 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff20406be30 0x7ff20406e2f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:49.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.923+0000 7ff228800640 1 --2- 172.21.15.90:0/4050256116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff220143310 0x7ff2201e59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:49.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.923+0000 7ff228800640 1 --2- 172.21.15.90:0/4050256116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff220142eb0 0x7ff2201e5490 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:49.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.923+0000 7ff228800640 1 --2- 172.21.15.90:0/4050256116 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2080051c0 0x7ff22015b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:49.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.923+0000 7ff228800640 1 -- 172.21.15.90:0/4050256116 >> 172.21.15.90:0/4050256116 conn(0x7ff2200b1820 msgr2=0x7ff2200ce0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:49.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.923+0000 7ff228800640 1 -- 172.21.15.90:0/4050256116 shutdown_connections 2024-04-25T01:17:49.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:49.923+0000 7ff228800640 1 -- 172.21.15.90:0/4050256116 wait complete. 2024-04-25T01:17:49.925 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T01:17:49.939 INFO:tasks.ceph.mds.d.smithi103.stderr: -6667> 2024-04-25T01:16:26.047+0000 7fd4bc400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T01:17:50.009 INFO:tasks.ceph.mds.d.smithi103.stdout:starting mds.d at 2024-04-25T01:17:50.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.134+0000 7f8e68200640 1 Processor -- start 2024-04-25T01:17:50.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.134+0000 7f8e68200640 1 -- start start 2024-04-25T01:17:50.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.136+0000 7f8e68200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e60158a60 0x7f8e6006df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:50.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.136+0000 7f8e68200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e6006d140 con 0x7f8e6015f4c0 2024-04-25T01:17:50.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.136+0000 7f8e68200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e6006d420 con 0x7f8e60158680 2024-04-25T01:17:50.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.136+0000 7f8e68200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8e6006d700 con 0x7f8e60158a60 2024-04-25T01:17:50.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.137+0000 7f8e66200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f8e6015f4c0 0x7f8e600b3550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47918/0 (socket says 172.21.15.90:47918) 2024-04-25T01:17:50.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.137+0000 7f8e66200640 1 -- 172.21.15.90:0/821288303 learned_addr learned my addr 172.21.15.90:0/821288303 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:50.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.137+0000 7f8e64e00640 1 --2- 172.21.15.90:0/821288303 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e60158a60 0x7f8e6006df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:50.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.137+0000 7f8e5fe00640 1 -- 172.21.15.90:0/821288303 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3379594133 0 0) 0x7f8e6006d140 con 0x7f8e6015f4c0 2024-04-25T01:17:50.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.137+0000 7f8e5fe00640 1 -- 172.21.15.90:0/821288303 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e48003520 con 0x7f8e6015f4c0 2024-04-25T01:17:50.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.138+0000 7f8e5fe00640 1 -- 172.21.15.90:0/821288303 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2560318344 0 0) 0x7f8e48003520 con 0x7f8e6015f4c0 2024-04-25T01:17:50.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.138+0000 7f8e5fe00640 1 -- 172.21.15.90:0/821288303 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e60158a60 msgr2=0x7f8e6006df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:50.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.138+0000 7f8e5fe00640 1 --2- 172.21.15.90:0/821288303 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e60158a60 0x7f8e6006df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:50.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.138+0000 7f8e5fe00640 1 -- 172.21.15.90:0/821288303 >> v1:172.21.15.103:6789/0 conn(0x7f8e60158680 legacy=0x7f8e6006e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:50.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.138+0000 7f8e5fe00640 1 -- 172.21.15.90:0/821288303 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e6006d9e0 con 0x7f8e6015f4c0 2024-04-25T01:17:50.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.138+0000 7f8e5fe00640 1 -- 172.21.15.90:0/821288303 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f8e580019f0 con 0x7f8e6015f4c0 2024-04-25T01:17:50.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.138+0000 7f8e5fe00640 1 -- 172.21.15.90:0/821288303 >> v1:172.21.15.90:6789/0 conn(0x7f8e6015f4c0 legacy=0x7f8e600b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:50.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.138+0000 7f8e5fe00640 1 --2- 172.21.15.90:0/821288303 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e60158a60 0x7f8e480039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:50.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.138+0000 7f8e5fe00640 1 --2- 172.21.15.90:0/821288303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e48003f60 0x7f8e48004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:50.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.138+0000 7f8e5fe00640 1 --2- 172.21.15.90:0/821288303 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e48004d40 0x7f8e480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:50.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.138+0000 7f8e5fe00640 1 -- 172.21.15.90:0/821288303 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8e6006d700 con 0x7f8e48003f60 2024-04-25T01:17:50.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.138+0000 7f8e5fe00640 1 -- 172.21.15.90:0/821288303 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4800bb50 con 0x7f8e48004d40 2024-04-25T01:17:50.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.138+0000 7f8e5fe00640 1 -- 172.21.15.90:0/821288303 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8e4800bd50 con 0x7f8e60158a60 2024-04-25T01:17:50.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.138+0000 7f8e64e00640 1 --2- 172.21.15.90:0/821288303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e48003f60 0x7f8e48004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:50.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.138+0000 7f8e65800640 1 --2- 172.21.15.90:0/821288303 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e60158a60 0x7f8e480039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:50.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.138+0000 7f8e64e00640 1 -- 172.21.15.90:0/821288303 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e60158a60 msgr2=0x7f8e480039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:50.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.138+0000 7f8e64e00640 1 --2- 172.21.15.90:0/821288303 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e60158a60 0x7f8e480039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:50.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.138+0000 7f8e66200640 1 --2- 172.21.15.90:0/821288303 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e48004d40 0x7f8e480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:50.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.138+0000 7f8e64e00640 1 -- 172.21.15.90:0/821288303 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e48004d40 msgr2=0x7f8e480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:50.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.139+0000 7f8e64e00640 1 --2- 172.21.15.90:0/821288303 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e48004d40 0x7f8e480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:50.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.139+0000 7f8e64e00640 1 -- 172.21.15.90:0/821288303 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e48003520 con 0x7f8e48003f60 2024-04-25T01:17:50.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.139+0000 7f8e66200640 1 --2- 172.21.15.90:0/821288303 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e48004d40 0x7f8e480051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:50.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.140+0000 7f8e64e00640 1 --2- 172.21.15.90:0/821288303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e48003f60 0x7f8e48004800 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c009a90 tx=0x7f8e48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e83050cfd7554ac8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:50.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.140+0000 7f8e5fe00640 1 -- 172.21.15.90:0/821288303 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c036070 con 0x7f8e48003f60 2024-04-25T01:17:50.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.140+0000 7f8e5fe00640 1 -- 172.21.15.90:0/821288303 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e4c031400 con 0x7f8e48003f60 2024-04-25T01:17:50.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.140+0000 7f8e5fe00640 1 -- 172.21.15.90:0/821288303 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c009690 con 0x7f8e48003f60 2024-04-25T01:17:50.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.140+0000 7f8e68200640 1 -- 172.21.15.90:0/821288303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e48003f60 msgr2=0x7f8e48004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:50.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.140+0000 7f8e68200640 1 --2- 172.21.15.90:0/821288303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e48003f60 0x7f8e48004800 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c009a90 tx=0x7f8e48005830 comp rx=0 tx=0).stop 2024-04-25T01:17:50.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.140+0000 7f8e66200640 1 -- 172.21.15.90:0/821288303 reap_dead start 2024-04-25T01:17:50.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.141+0000 7f8e68200640 1 -- 172.21.15.90:0/821288303 shutdown_connections 2024-04-25T01:17:50.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.141+0000 7f8e68200640 1 -- 172.21.15.90:0/821288303 >> 172.21.15.90:0/821288303 conn(0x7f8e6006b720 msgr2=0x7f8e600b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:50.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.141+0000 7f8e68200640 1 -- 172.21.15.90:0/821288303 shutdown_connections 2024-04-25T01:17:50.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.141+0000 7f8e68200640 1 -- 172.21.15.90:0/821288303 wait complete. 2024-04-25T01:17:50.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.142+0000 7f8e68200640 1 Processor -- start 2024-04-25T01:17:50.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.143+0000 7f8e68200640 1 -- start start 2024-04-25T01:17:50.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.143+0000 7f8e68200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e48003f60 0x7f8e601417a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:50.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.143+0000 7f8e68200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e601e9910 0x7f8e601e85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:50.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.143+0000 7f8e68200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e60152a40 0x7f8e60152ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:50.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.143+0000 7f8e68200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8e600b3de0 con 0x7f8e60152a40 2024-04-25T01:17:50.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.143+0000 7f8e68200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8e600b3c60 con 0x7f8e48003f60 2024-04-25T01:17:50.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.143+0000 7f8e68200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8e6006df00 con 0x7f8e601e9910 2024-04-25T01:17:50.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.144+0000 7f8e66200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e60152a40 0x7f8e60152ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:50.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.144+0000 7f8e66200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e60152a40 0x7f8e60152ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46268/0 (socket says 172.21.15.90:46268) 2024-04-25T01:17:50.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.144+0000 7f8e66200640 1 -- 172.21.15.90:0/482867212 learned_addr learned my addr 172.21.15.90:0/482867212 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:50.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.144+0000 7f8e65800640 1 --2- 172.21.15.90:0/482867212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e48003f60 0x7f8e601417a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:50.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.144+0000 7f8e66200640 1 -- 172.21.15.90:0/482867212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e601e9910 msgr2=0x7f8e601e85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:50.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.144+0000 7f8e64e00640 1 --2- 172.21.15.90:0/482867212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e601e9910 0x7f8e601e85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:50.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.144+0000 7f8e66200640 1 --2- 172.21.15.90:0/482867212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e601e9910 0x7f8e601e85f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:50.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.144+0000 7f8e66200640 1 -- 172.21.15.90:0/482867212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e48003f60 msgr2=0x7f8e601417a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:50.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.144+0000 7f8e66200640 1 --2- 172.21.15.90:0/482867212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e48003f60 0x7f8e601417a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:50.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.144+0000 7f8e66200640 1 -- 172.21.15.90:0/482867212 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e601e9d70 con 0x7f8e60152a40 2024-04-25T01:17:50.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.144+0000 7f8e65800640 1 --2- 172.21.15.90:0/482867212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e48003f60 0x7f8e601417a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:50.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.144+0000 7f8e66200640 1 --2- 172.21.15.90:0/482867212 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e60152a40 0x7f8e60152ec0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f8e600cf860 tx=0x7f8e5800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:50.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.145+0000 7f8e5d600640 1 -- 172.21.15.90:0/482867212 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5800bb20 con 0x7f8e60152a40 2024-04-25T01:17:50.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.145+0000 7f8e5d600640 1 -- 172.21.15.90:0/482867212 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e5800bcc0 con 0x7f8e60152a40 2024-04-25T01:17:50.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.145+0000 7f8e68200640 1 -- 172.21.15.90:0/482867212 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e6015fad0 con 0x7f8e60152a40 2024-04-25T01:17:50.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.145+0000 7f8e5d600640 1 -- 172.21.15.90:0/482867212 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e580072b0 con 0x7f8e60152a40 2024-04-25T01:17:50.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.145+0000 7f8e68200640 1 -- 172.21.15.90:0/482867212 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e60160090 con 0x7f8e60152a40 2024-04-25T01:17:50.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.146+0000 7f8e68200640 1 -- 172.21.15.90:0/482867212 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e60068fb0 con 0x7f8e60152a40 2024-04-25T01:17:50.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.147+0000 7f8e5d600640 1 -- 172.21.15.90:0/482867212 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8e5801b020 con 0x7f8e60152a40 2024-04-25T01:17:50.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.148+0000 7f8e5d600640 1 --2- 172.21.15.90:0/482867212 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f8e4006be30 0x7f8e4006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:50.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.148+0000 7f8e5d600640 1 -- 172.21.15.90:0/482867212 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6676+0+0 (secure 0 0 0) 0x7f8e5808b5c0 con 0x7f8e60152a40 2024-04-25T01:17:50.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.148+0000 7f8e65800640 1 --2- 172.21.15.90:0/482867212 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f8e4006be30 0x7f8e4006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:50.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.149+0000 7f8e65800640 1 --2- 172.21.15.90:0/482867212 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f8e4006be30 0x7f8e4006e2f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8e540015f0 tx=0x7f8e54009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:50.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.152+0000 7f8e5d600640 1 -- 172.21.15.90:0/482867212 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8e58059b00 con 0x7f8e60152a40 2024-04-25T01:17:50.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.393+0000 7f8e68200640 1 -- 172.21.15.90:0/482867212 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f8e601e9f40 con 0x7f8e60152a40 2024-04-25T01:17:50.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.924+0000 7f8e5d600640 1 -- 172.21.15.90:0/482867212 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7f8e58059400 con 0x7f8e60152a40 2024-04-25T01:17:50.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.927+0000 7f8e68200640 1 -- 172.21.15.90:0/482867212 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f8e4006be30 msgr2=0x7f8e4006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:50.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.927+0000 7f8e68200640 1 --2- 172.21.15.90:0/482867212 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f8e4006be30 0x7f8e4006e2f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8e540015f0 tx=0x7f8e54009040 comp rx=0 tx=0).stop 2024-04-25T01:17:50.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.927+0000 7f8e68200640 1 -- 172.21.15.90:0/482867212 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e60152a40 msgr2=0x7f8e60152ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:50.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.927+0000 7f8e68200640 1 --2- 172.21.15.90:0/482867212 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e60152a40 0x7f8e60152ec0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f8e600cf860 tx=0x7f8e5800b4a0 comp rx=0 tx=0).stop 2024-04-25T01:17:50.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.928+0000 7f8e68200640 1 -- 172.21.15.90:0/482867212 shutdown_connections 2024-04-25T01:17:50.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.928+0000 7f8e68200640 1 --2- 172.21.15.90:0/482867212 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f8e4006be30 0x7f8e4006e2f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:50.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.928+0000 7f8e68200640 1 --2- 172.21.15.90:0/482867212 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e60152a40 0x7f8e60152ec0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:50.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.928+0000 7f8e68200640 1 --2- 172.21.15.90:0/482867212 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8e601e9910 0x7f8e601e85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:50.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.928+0000 7f8e68200640 1 --2- 172.21.15.90:0/482867212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e48003f60 0x7f8e601417a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:50.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.928+0000 7f8e68200640 1 -- 172.21.15.90:0/482867212 >> 172.21.15.90:0/482867212 conn(0x7f8e6006b720 msgr2=0x7f8e6006e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:50.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.928+0000 7f8e68200640 1 -- 172.21.15.90:0/482867212 shutdown_connections 2024-04-25T01:17:50.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:50.928+0000 7f8e68200640 1 -- 172.21.15.90:0/482867212 wait complete. 2024-04-25T01:17:50.960 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T01:17:51.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.193+0000 7fdc33200640 1 Processor -- start 2024-04-25T01:17:51.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.194+0000 7fdc33200640 1 -- start start 2024-04-25T01:17:51.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.195+0000 7fdc33200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdc3415e0a0 0x7fdc340d4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:51.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.195+0000 7fdc33200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc340d2b10 0x7fdc340cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:51.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.195+0000 7fdc33200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc340cfb90 con 0x7fdc3415e480 2024-04-25T01:17:51.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.195+0000 7fdc33200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fdc340c7970 con 0x7fdc3415e0a0 2024-04-25T01:17:51.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.195+0000 7fdc33200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdc340c7bc0 con 0x7fdc340d2b10 2024-04-25T01:17:51.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.196+0000 7fdc31e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdc3415e0a0 0x7fdc340d4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:51.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.196+0000 7fdc31e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdc3415e0a0 0x7fdc340d4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46278/0 (socket says 172.21.15.90:46278) 2024-04-25T01:17:51.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.196+0000 7fdc31e00640 1 -- 172.21.15.90:0/197384877 learned_addr learned my addr 172.21.15.90:0/197384877 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:51.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.196+0000 7fdc31e00640 1 -- 172.21.15.90:0/197384877 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc340d2b10 msgr2=0x7fdc340cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:51.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.196+0000 7fdc31e00640 1 --2- 172.21.15.90:0/197384877 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc340d2b10 0x7fdc340cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:51.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.196+0000 7fdc31e00640 1 -- 172.21.15.90:0/197384877 >> v1:172.21.15.103:6789/0 conn(0x7fdc3415e480 legacy=0x7fdc340d2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:51.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.196+0000 7fdc31e00640 1 -- 172.21.15.90:0/197384877 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc340c7ea0 con 0x7fdc3415e0a0 2024-04-25T01:17:51.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.197+0000 7fdc31e00640 1 --2- 172.21.15.90:0/197384877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc3415e0a0 0x7fdc340d4940 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fdc3408c130 tx=0x7fdc2402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac9c3a3d56f90c86 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:51.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.197+0000 7fdc30a00640 1 -- 172.21.15.90:0/197384877 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc24037070 con 0x7fdc3415e0a0 2024-04-25T01:17:51.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.198+0000 7fdc30a00640 1 -- 172.21.15.90:0/197384877 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc24032440 con 0x7fdc3415e0a0 2024-04-25T01:17:51.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.198+0000 7fdc30a00640 1 -- 172.21.15.90:0/197384877 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc24032810 con 0x7fdc3415e0a0 2024-04-25T01:17:51.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.198+0000 7fdc33200640 1 -- 172.21.15.90:0/197384877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc3415e0a0 msgr2=0x7fdc340d4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:51.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.198+0000 7fdc33200640 1 --2- 172.21.15.90:0/197384877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc3415e0a0 0x7fdc340d4940 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fdc3408c130 tx=0x7fdc2402a1e0 comp rx=0 tx=0).stop 2024-04-25T01:17:51.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.198+0000 7fdc33200640 1 -- 172.21.15.90:0/197384877 shutdown_connections 2024-04-25T01:17:51.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.198+0000 7fdc33200640 1 --2- 172.21.15.90:0/197384877 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc340d2b10 0x7fdc340cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:51.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.198+0000 7fdc33200640 1 --2- 172.21.15.90:0/197384877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc3415e0a0 0x7fdc340d4940 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:51.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.198+0000 7fdc33200640 1 -- 172.21.15.90:0/197384877 >> 172.21.15.90:0/197384877 conn(0x7fdc340d3e20 msgr2=0x7fdc340d4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:51.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.198+0000 7fdc33200640 1 -- 172.21.15.90:0/197384877 shutdown_connections 2024-04-25T01:17:51.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.198+0000 7fdc33200640 1 -- 172.21.15.90:0/197384877 wait complete. 2024-04-25T01:17:51.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.200+0000 7fdc33200640 1 Processor -- start 2024-04-25T01:17:51.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.200+0000 7fdc33200640 1 -- start start 2024-04-25T01:17:51.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.201+0000 7fdc33200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc340d2b10 0x7fdc34151860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:51.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.201+0000 7fdc33200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc3415e0a0 0x7fdc34151da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:51.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.201+0000 7fdc33200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc34148920 0x7fdc34148dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:51.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.201+0000 7fdc33200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdc340b4bd0 con 0x7fdc3415e0a0 2024-04-25T01:17:51.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.201+0000 7fdc33200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdc340b4a50 con 0x7fdc34148920 2024-04-25T01:17:51.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.201+0000 7fdc33200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdc340b4d50 con 0x7fdc340d2b10 2024-04-25T01:17:51.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.201+0000 7fdc31400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc3415e0a0 0x7fdc34151da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:51.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.201+0000 7fdc31400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc3415e0a0 0x7fdc34151da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46284/0 (socket says 172.21.15.90:46284) 2024-04-25T01:17:51.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.201+0000 7fdc31400640 1 -- 172.21.15.90:0/152044762 learned_addr learned my addr 172.21.15.90:0/152044762 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:51.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.202+0000 7fdc31e00640 1 --2- 172.21.15.90:0/152044762 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc340d2b10 0x7fdc34151860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:51.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.202+0000 7fdc31400640 1 -- 172.21.15.90:0/152044762 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc340d2b10 msgr2=0x7fdc34151860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:51.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.202+0000 7fdc31400640 1 --2- 172.21.15.90:0/152044762 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc340d2b10 0x7fdc34151860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:51.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.202+0000 7fdc31400640 1 -- 172.21.15.90:0/152044762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc34148920 msgr2=0x7fdc34148dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:51.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.202+0000 7fdc31400640 1 --2- 172.21.15.90:0/152044762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc34148920 0x7fdc34148dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:51.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.202+0000 7fdc31400640 1 -- 172.21.15.90:0/152044762 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc34152840 con 0x7fdc3415e0a0 2024-04-25T01:17:51.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.202+0000 7fdc31400640 1 --2- 172.21.15.90:0/152044762 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc3415e0a0 0x7fdc34151da0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c00ba50 tx=0x7fdc1c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:51.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.202+0000 7fdc2a000640 1 -- 172.21.15.90:0/152044762 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c019070 con 0x7fdc3415e0a0 2024-04-25T01:17:51.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.202+0000 7fdc2a000640 1 -- 172.21.15.90:0/152044762 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc1c00cc20 con 0x7fdc3415e0a0 2024-04-25T01:17:51.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.202+0000 7fdc2a000640 1 -- 172.21.15.90:0/152044762 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c014680 con 0x7fdc3415e0a0 2024-04-25T01:17:51.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.202+0000 7fdc33200640 1 -- 172.21.15.90:0/152044762 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc3415e9f0 con 0x7fdc3415e0a0 2024-04-25T01:17:51.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.203+0000 7fdc33200640 1 -- 172.21.15.90:0/152044762 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc3415efb0 con 0x7fdc3415e0a0 2024-04-25T01:17:51.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.204+0000 7fdc33200640 1 -- 172.21.15.90:0/152044762 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc340753c0 con 0x7fdc3415e0a0 2024-04-25T01:17:51.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.205+0000 7fdc2a000640 1 -- 172.21.15.90:0/152044762 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fdc1c01c070 con 0x7fdc3415e0a0 2024-04-25T01:17:51.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.206+0000 7fdc2a000640 1 --2- 172.21.15.90:0/152044762 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdc1006be30 0x7fdc1006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:51.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.206+0000 7fdc2a000640 1 -- 172.21.15.90:0/152044762 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6676+0+0 (secure 0 0 0) 0x7fdc1c08ae50 con 0x7fdc3415e0a0 2024-04-25T01:17:51.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.206+0000 7fdc31e00640 1 --2- 172.21.15.90:0/152044762 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdc1006be30 0x7fdc1006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:51.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.207+0000 7fdc31e00640 1 --2- 172.21.15.90:0/152044762 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdc1006be30 0x7fdc1006e2f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdc24034000 tx=0x7fdc2402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:51.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.210+0000 7fdc2a000640 1 -- 172.21.15.90:0/152044762 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fdc1c058e70 con 0x7fdc3415e0a0 2024-04-25T01:17:51.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:51.409+0000 7fdc33200640 1 -- 172.21.15.90:0/152044762 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fdc341522e0 con 0x7fdc3415e0a0 2024-04-25T01:17:52.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.123+0000 7fdc2a000640 1 -- 172.21.15.90:0/152044762 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v30) ==== 181+0+0 (secure 0 0 0) 0x7fdc1c05d170 con 0x7fdc3415e0a0 2024-04-25T01:17:52.107 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_metadata' removed 2024-04-25T01:17:52.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.127+0000 7fdc33200640 1 -- 172.21.15.90:0/152044762 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdc1006be30 msgr2=0x7fdc1006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:52.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.127+0000 7fdc33200640 1 --2- 172.21.15.90:0/152044762 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdc1006be30 0x7fdc1006e2f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdc24034000 tx=0x7fdc2402e000 comp rx=0 tx=0).stop 2024-04-25T01:17:52.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.127+0000 7fdc33200640 1 -- 172.21.15.90:0/152044762 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc3415e0a0 msgr2=0x7fdc34151da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:52.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.127+0000 7fdc33200640 1 --2- 172.21.15.90:0/152044762 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc3415e0a0 0x7fdc34151da0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c00ba50 tx=0x7fdc1c00bf10 comp rx=0 tx=0).stop 2024-04-25T01:17:52.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.127+0000 7fdc33200640 1 -- 172.21.15.90:0/152044762 shutdown_connections 2024-04-25T01:17:52.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.127+0000 7fdc33200640 1 --2- 172.21.15.90:0/152044762 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdc1006be30 0x7fdc1006e2f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:52.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.127+0000 7fdc33200640 1 --2- 172.21.15.90:0/152044762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc34148920 0x7fdc34148dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:52.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.127+0000 7fdc33200640 1 --2- 172.21.15.90:0/152044762 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc3415e0a0 0x7fdc34151da0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:52.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.127+0000 7fdc33200640 1 --2- 172.21.15.90:0/152044762 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc340d2b10 0x7fdc34151860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:52.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.127+0000 7fdc33200640 1 -- 172.21.15.90:0/152044762 >> 172.21.15.90:0/152044762 conn(0x7fdc340d3e20 msgr2=0x7fdc3406b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:52.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.128+0000 7fdc33200640 1 -- 172.21.15.90:0/152044762 shutdown_connections 2024-04-25T01:17:52.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.129+0000 7fdc33200640 1 -- 172.21.15.90:0/152044762 wait complete. 2024-04-25T01:17:52.139 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T01:17:52.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.348+0000 7fa012200640 1 Processor -- start 2024-04-25T01:17:52.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.349+0000 7fa012200640 1 -- start start 2024-04-25T01:17:52.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.350+0000 7fa012200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa00c06c640 0x7fa00c06cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:52.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.350+0000 7fa012200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa00c0d12e0 con 0x7fa00c15e2f0 2024-04-25T01:17:52.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.350+0000 7fa012200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa00c0cc480 con 0x7fa00c06d160 2024-04-25T01:17:52.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.350+0000 7fa012200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fa00c0b4ae0 con 0x7fa00c06c640 2024-04-25T01:17:52.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.351+0000 7fa00aa00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa00c06c640 0x7fa00c06cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:52.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.351+0000 7fa00b400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa00c15e2f0 0x7fa00c0d09e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47928/0 (socket says 172.21.15.90:47928) 2024-04-25T01:17:52.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.351+0000 7fa00b400640 1 -- 172.21.15.90:0/557526298 learned_addr learned my addr 172.21.15.90:0/557526298 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:52.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.351+0000 7fa00a000640 1 -- 172.21.15.90:0/557526298 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3091057909 0 0) 0x7fa00c0d12e0 con 0x7fa00c15e2f0 2024-04-25T01:17:52.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.351+0000 7fa00a000640 1 -- 172.21.15.90:0/557526298 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ff0003520 con 0x7fa00c15e2f0 2024-04-25T01:17:52.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.351+0000 7fa00aa00640 1 -- 172.21.15.90:0/557526298 >> v1:172.21.15.103:6789/0 conn(0x7fa00c06d160 legacy=0x7fa00c0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:52.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.351+0000 7fa00aa00640 1 -- 172.21.15.90:0/557526298 >> v1:172.21.15.90:6789/0 conn(0x7fa00c15e2f0 legacy=0x7fa00c0d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:52.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.351+0000 7fa00aa00640 1 -- 172.21.15.90:0/557526298 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa00c0cc760 con 0x7fa00c06c640 2024-04-25T01:17:52.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.352+0000 7fa00aa00640 1 --2- 172.21.15.90:0/557526298 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa00c06c640 0x7fa00c06cc20 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc009d90 tx=0x7f9ffc02a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dd964431f2648c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:52.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.353+0000 7fa00a000640 1 -- 172.21.15.90:0/557526298 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc007740 con 0x7fa00c06c640 2024-04-25T01:17:52.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.353+0000 7fa00a000640 1 -- 172.21.15.90:0/557526298 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ffc0078e0 con 0x7fa00c06c640 2024-04-25T01:17:52.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.353+0000 7fa00a000640 1 -- 172.21.15.90:0/557526298 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc007c90 con 0x7fa00c06c640 2024-04-25T01:17:52.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.353+0000 7fa012200640 1 -- 172.21.15.90:0/557526298 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa00c06c640 msgr2=0x7fa00c06cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:52.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.353+0000 7fa012200640 1 --2- 172.21.15.90:0/557526298 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa00c06c640 0x7fa00c06cc20 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc009d90 tx=0x7f9ffc02a730 comp rx=0 tx=0).stop 2024-04-25T01:17:52.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.353+0000 7fa012200640 1 -- 172.21.15.90:0/557526298 shutdown_connections 2024-04-25T01:17:52.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.353+0000 7fa012200640 1 --2- 172.21.15.90:0/557526298 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa00c06c640 0x7fa00c06cc20 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:52.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.353+0000 7fa012200640 1 -- 172.21.15.90:0/557526298 >> 172.21.15.90:0/557526298 conn(0x7fa00c0d1c90 msgr2=0x7fa00c0d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:52.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.353+0000 7fa012200640 1 -- 172.21.15.90:0/557526298 shutdown_connections 2024-04-25T01:17:52.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.353+0000 7fa012200640 1 -- 172.21.15.90:0/557526298 wait complete. 2024-04-25T01:17:52.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.355+0000 7fa012200640 1 Processor -- start 2024-04-25T01:17:52.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.355+0000 7fa012200640 1 -- start start 2024-04-25T01:17:52.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.356+0000 7fa012200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa00c06c640 0x7fa00c1f4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:52.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.356+0000 7fa012200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa00c06d160 0x7fa00c154da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:52.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.356+0000 7fa012200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa00c15e2f0 0x7fa00c0cb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:52.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.356+0000 7fa012200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa00c06d590 con 0x7fa00c06c640 2024-04-25T01:17:52.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.356+0000 7fa012200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa00c0ccad0 con 0x7fa00c15e2f0 2024-04-25T01:17:52.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.356+0000 7fa012200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa00c0cd670 con 0x7fa00c06d160 2024-04-25T01:17:52.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.356+0000 7fa00b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa00c06c640 0x7fa00c1f4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:52.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.357+0000 7fa00b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa00c06c640 0x7fa00c1f4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46292/0 (socket says 172.21.15.90:46292) 2024-04-25T01:17:52.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.357+0000 7fa00b400640 1 -- 172.21.15.90:0/3791764948 learned_addr learned my addr 172.21.15.90:0/3791764948 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:52.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.357+0000 7fa00aa00640 1 --2- 172.21.15.90:0/3791764948 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa00c06d160 0x7fa00c154da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:52.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.357+0000 7fa00be00640 1 --2- 172.21.15.90:0/3791764948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa00c15e2f0 0x7fa00c0cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:52.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.357+0000 7fa00b400640 1 -- 172.21.15.90:0/3791764948 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa00c06d160 msgr2=0x7fa00c154da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:52.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.357+0000 7fa00b400640 1 --2- 172.21.15.90:0/3791764948 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa00c06d160 0x7fa00c154da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:52.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.357+0000 7fa00b400640 1 -- 172.21.15.90:0/3791764948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa00c15e2f0 msgr2=0x7fa00c0cb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:52.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.357+0000 7fa00b400640 1 --2- 172.21.15.90:0/3791764948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa00c15e2f0 0x7fa00c0cb290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:52.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.357+0000 7fa00b400640 1 -- 172.21.15.90:0/3791764948 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa00c0c9e30 con 0x7fa00c06c640 2024-04-25T01:17:52.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.357+0000 7fa00b400640 1 --2- 172.21.15.90:0/3791764948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa00c06c640 0x7fa00c1f4c20 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f9ff400c390 tx=0x7f9ff400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:52.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.357+0000 7fa003400640 1 -- 172.21.15.90:0/3791764948 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff4017070 con 0x7fa00c06c640 2024-04-25T01:17:52.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.358+0000 7fa003400640 1 -- 172.21.15.90:0/3791764948 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ff4005ce0 con 0x7fa00c06c640 2024-04-25T01:17:52.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.358+0000 7fa003400640 1 -- 172.21.15.90:0/3791764948 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff40072f0 con 0x7fa00c06c640 2024-04-25T01:17:52.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.358+0000 7fa012200640 1 -- 172.21.15.90:0/3791764948 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa00c1557f0 con 0x7fa00c06c640 2024-04-25T01:17:52.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.358+0000 7fa012200640 1 -- 172.21.15.90:0/3791764948 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa00c1589e0 con 0x7fa00c06c640 2024-04-25T01:17:52.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.359+0000 7fa012200640 1 -- 172.21.15.90:0/3791764948 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa00c0b3c00 con 0x7fa00c06c640 2024-04-25T01:17:52.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.360+0000 7fa003400640 1 -- 172.21.15.90:0/3791764948 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9ff401a070 con 0x7fa00c06c640 2024-04-25T01:17:52.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.361+0000 7fa003400640 1 --2- 172.21.15.90:0/3791764948 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9fe806be30 0x7f9fe806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:52.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.361+0000 7fa003400640 1 -- 172.21.15.90:0/3791764948 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6238+0+0 (secure 0 0 0) 0x7f9ff408a470 con 0x7fa00c06c640 2024-04-25T01:17:52.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.362+0000 7fa00aa00640 1 --2- 172.21.15.90:0/3791764948 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9fe806be30 0x7f9fe806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:52.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.362+0000 7fa00aa00640 1 --2- 172.21.15.90:0/3791764948 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9fe806be30 0x7f9fe806e2f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc034000 tx=0x7f9ffc02f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:52.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.366+0000 7fa003400640 1 -- 172.21.15.90:0/3791764948 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa00c0b3c00 con 0x7fa00c06c640 2024-04-25T01:17:52.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:52.572+0000 7fa012200640 1 -- 172.21.15.90:0/3791764948 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa00c068fb0 con 0x7fa00c06c640 2024-04-25T01:17:53.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.127+0000 7fa003400640 1 -- 172.21.15.90:0/3791764948 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v31) ==== 169+0+0 (secure 0 0 0) 0x7fa00c1557f0 con 0x7fa00c06c640 2024-04-25T01:17:53.110 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data' removed 2024-04-25T01:17:53.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.131+0000 7fa012200640 1 -- 172.21.15.90:0/3791764948 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9fe806be30 msgr2=0x7f9fe806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:53.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.131+0000 7fa012200640 1 --2- 172.21.15.90:0/3791764948 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9fe806be30 0x7f9fe806e2f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc034000 tx=0x7f9ffc02f000 comp rx=0 tx=0).stop 2024-04-25T01:17:53.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.131+0000 7fa012200640 1 -- 172.21.15.90:0/3791764948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa00c06c640 msgr2=0x7fa00c1f4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:53.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.131+0000 7fa012200640 1 --2- 172.21.15.90:0/3791764948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa00c06c640 0x7fa00c1f4c20 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f9ff400c390 tx=0x7f9ff400c850 comp rx=0 tx=0).stop 2024-04-25T01:17:53.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.131+0000 7fa012200640 1 -- 172.21.15.90:0/3791764948 shutdown_connections 2024-04-25T01:17:53.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.131+0000 7fa012200640 1 --2- 172.21.15.90:0/3791764948 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9fe806be30 0x7f9fe806e2f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:53.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.131+0000 7fa012200640 1 --2- 172.21.15.90:0/3791764948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa00c15e2f0 0x7fa00c0cb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:53.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.131+0000 7fa012200640 1 --2- 172.21.15.90:0/3791764948 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa00c06d160 0x7fa00c154da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:53.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.131+0000 7fa012200640 1 --2- 172.21.15.90:0/3791764948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa00c06c640 0x7fa00c1f4c20 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:53.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.131+0000 7fa012200640 1 -- 172.21.15.90:0/3791764948 >> 172.21.15.90:0/3791764948 conn(0x7fa00c0d1c90 msgr2=0x7fa00c0b34f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:53.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.131+0000 7fa012200640 1 -- 172.21.15.90:0/3791764948 shutdown_connections 2024-04-25T01:17:53.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.131+0000 7fa012200640 1 -- 172.21.15.90:0/3791764948 wait complete. 2024-04-25T01:17:53.142 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-25T01:17:53.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.356+0000 7f5de0a00640 1 Processor -- start 2024-04-25T01:17:53.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.356+0000 7f5de0a00640 1 -- start start 2024-04-25T01:17:53.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.358+0000 7f5de0a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ddc06e460 0x7f5ddc0b2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:53.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.358+0000 7f5de0a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5ddc06c9d0 0x7f5ddc06ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:53.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.358+0000 7f5de0a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ddc0b4470 con 0x7f5ddc162290 2024-04-25T01:17:53.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.358+0000 7f5de0a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f5ddc0b2d50 con 0x7f5ddc06c9d0 2024-04-25T01:17:53.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.358+0000 7f5de0a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5ddc070a50 con 0x7f5ddc06e460 2024-04-25T01:17:53.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.358+0000 7f5ddbe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5ddc06c9d0 0x7f5ddc06ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:53.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.359+0000 7f5ddbe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5ddc06c9d0 0x7f5ddc06ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46308/0 (socket says 172.21.15.90:46308) 2024-04-25T01:17:53.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.359+0000 7f5ddbe00640 1 -- 172.21.15.90:0/2818446543 learned_addr learned my addr 172.21.15.90:0/2818446543 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:53.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.359+0000 7f5ddaa00640 1 --2- 172.21.15.90:0/2818446543 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ddc06e460 0x7f5ddc0b2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:53.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.359+0000 7f5ddbe00640 1 -- 172.21.15.90:0/2818446543 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ddc06e460 msgr2=0x7f5ddc0b2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:53.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.359+0000 7f5ddbe00640 1 --2- 172.21.15.90:0/2818446543 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ddc06e460 0x7f5ddc0b2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:53.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.359+0000 7f5ddbe00640 1 -- 172.21.15.90:0/2818446543 >> v1:172.21.15.103:6789/0 conn(0x7f5ddc162290 legacy=0x7f5ddc0cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:53.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.359+0000 7f5ddbe00640 1 -- 172.21.15.90:0/2818446543 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ddc0b4710 con 0x7f5ddc06c9d0 2024-04-25T01:17:53.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.360+0000 7f5ddbe00640 1 --2- 172.21.15.90:0/2818446543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ddc06c9d0 0x7f5ddc06ce60 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4009e50 tx=0x7f5dd402a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6c18122b907ba4e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:53.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.360+0000 7f5dda000640 1 -- 172.21.15.90:0/2818446543 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd4036070 con 0x7f5ddc06c9d0 2024-04-25T01:17:53.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.360+0000 7f5dda000640 1 -- 172.21.15.90:0/2818446543 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dd4031600 con 0x7f5ddc06c9d0 2024-04-25T01:17:53.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.361+0000 7f5dda000640 1 -- 172.21.15.90:0/2818446543 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd40319d0 con 0x7f5ddc06c9d0 2024-04-25T01:17:53.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.361+0000 7f5de0a00640 1 -- 172.21.15.90:0/2818446543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ddc06c9d0 msgr2=0x7f5ddc06ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:53.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.361+0000 7f5de0a00640 1 --2- 172.21.15.90:0/2818446543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ddc06c9d0 0x7f5ddc06ce60 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4009e50 tx=0x7f5dd402a6c0 comp rx=0 tx=0).stop 2024-04-25T01:17:53.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.361+0000 7f5de0a00640 1 -- 172.21.15.90:0/2818446543 shutdown_connections 2024-04-25T01:17:53.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.361+0000 7f5de0a00640 1 --2- 172.21.15.90:0/2818446543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ddc06c9d0 0x7f5ddc06ce60 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:53.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.361+0000 7f5de0a00640 1 --2- 172.21.15.90:0/2818446543 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ddc06e460 0x7f5ddc0b2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:53.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.361+0000 7f5de0a00640 1 -- 172.21.15.90:0/2818446543 >> 172.21.15.90:0/2818446543 conn(0x7f5ddc06b120 msgr2=0x7f5ddc06b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:53.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.361+0000 7f5de0a00640 1 -- 172.21.15.90:0/2818446543 shutdown_connections 2024-04-25T01:17:53.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.361+0000 7f5de0a00640 1 -- 172.21.15.90:0/2818446543 wait complete. 2024-04-25T01:17:53.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.363+0000 7f5de0a00640 1 Processor -- start 2024-04-25T01:17:53.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.363+0000 7f5de0a00640 1 -- start start 2024-04-25T01:17:53.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.364+0000 7f5de0a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ddc06c9d0 0x7f5ddc0ada50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:53.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.364+0000 7f5de0a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ddc06e460 0x7f5ddc0b22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:53.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.364+0000 7f5de0a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ddc162290 0x7f5ddc160150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:53.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.364+0000 7f5de0a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5ddc0b4cd0 con 0x7f5ddc162290 2024-04-25T01:17:53.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.364+0000 7f5de0a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5ddc162670 con 0x7f5ddc06e460 2024-04-25T01:17:53.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.364+0000 7f5de0a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5ddc070f30 con 0x7f5ddc06c9d0 2024-04-25T01:17:53.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.365+0000 7f5ddbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ddc162290 0x7f5ddc160150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:53.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.365+0000 7f5ddbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ddc162290 0x7f5ddc160150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46320/0 (socket says 172.21.15.90:46320) 2024-04-25T01:17:53.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.365+0000 7f5ddbe00640 1 -- 172.21.15.90:0/1953537731 learned_addr learned my addr 172.21.15.90:0/1953537731 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:53.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.365+0000 7f5ddaa00640 1 --2- 172.21.15.90:0/1953537731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ddc06e460 0x7f5ddc0b22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:53.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.365+0000 7f5ddb400640 1 --2- 172.21.15.90:0/1953537731 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ddc06c9d0 0x7f5ddc0ada50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:53.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.365+0000 7f5ddbe00640 1 -- 172.21.15.90:0/1953537731 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ddc06c9d0 msgr2=0x7f5ddc0ada50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:53.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.365+0000 7f5ddbe00640 1 --2- 172.21.15.90:0/1953537731 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ddc06c9d0 0x7f5ddc0ada50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:53.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.365+0000 7f5ddbe00640 1 -- 172.21.15.90:0/1953537731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ddc06e460 msgr2=0x7f5ddc0b22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:53.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.365+0000 7f5ddbe00640 1 --2- 172.21.15.90:0/1953537731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ddc06e460 0x7f5ddc0b22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:53.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.365+0000 7f5ddbe00640 1 -- 172.21.15.90:0/1953537731 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ddc0adf90 con 0x7f5ddc162290 2024-04-25T01:17:53.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.365+0000 7f5ddbe00640 1 --2- 172.21.15.90:0/1953537731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ddc162290 0x7f5ddc160150 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4034040 tx=0x7f5dd4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:53.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.366+0000 7f5dd3400640 1 -- 172.21.15.90:0/1953537731 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd4036050 con 0x7f5ddc162290 2024-04-25T01:17:53.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.366+0000 7f5dd3400640 1 -- 172.21.15.90:0/1953537731 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dd4042070 con 0x7f5ddc162290 2024-04-25T01:17:53.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.366+0000 7f5de0a00640 1 -- 172.21.15.90:0/1953537731 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ddc0b1250 con 0x7f5ddc162290 2024-04-25T01:17:53.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.366+0000 7f5dd3400640 1 -- 172.21.15.90:0/1953537731 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd4009510 con 0x7f5ddc162290 2024-04-25T01:17:53.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.366+0000 7f5de0a00640 1 -- 172.21.15.90:0/1953537731 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ddc0b1af0 con 0x7f5ddc162290 2024-04-25T01:17:53.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.367+0000 7f5de0a00640 1 -- 172.21.15.90:0/1953537731 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ddc0b1640 con 0x7f5ddc162290 2024-04-25T01:17:53.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.368+0000 7f5dd3400640 1 -- 172.21.15.90:0/1953537731 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5dd4007bd0 con 0x7f5ddc162290 2024-04-25T01:17:53.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.369+0000 7f5dd3400640 1 --2- 172.21.15.90:0/1953537731 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f5dbc06be30 0x7f5dbc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:53.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.370+0000 7f5dd3400640 1 -- 172.21.15.90:0/1953537731 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5795+0+0 (secure 0 0 0) 0x7f5dd40abea0 con 0x7f5ddc162290 2024-04-25T01:17:53.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.370+0000 7f5ddb400640 1 --2- 172.21.15.90:0/1953537731 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f5dbc06be30 0x7f5dbc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:53.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.370+0000 7f5ddb400640 1 --2- 172.21.15.90:0/1953537731 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f5dbc06be30 0x7f5dbc06e2f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc00a750 tx=0x7f5dcc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:53.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.374+0000 7f5dd3400640 1 -- 172.21.15.90:0/1953537731 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f5dd407a7e0 con 0x7f5ddc162290 2024-04-25T01:17:53.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:53.571+0000 7f5de0a00640 1 -- 172.21.15.90:0/1953537731 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5ddc0753c0 con 0x7f5ddc162290 2024-04-25T01:17:54.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.150+0000 7f5dd3400640 1 -- 172.21.15.90:0/1953537731 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v32) ==== 178+0+0 (secure 0 0 0) 0x7f5dd407a0e0 con 0x7f5ddc162290 2024-04-25T01:17:54.134 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data_ec' removed 2024-04-25T01:17:54.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.154+0000 7f5de0a00640 1 -- 172.21.15.90:0/1953537731 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f5dbc06be30 msgr2=0x7f5dbc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:54.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.154+0000 7f5de0a00640 1 --2- 172.21.15.90:0/1953537731 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f5dbc06be30 0x7f5dbc06e2f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc00a750 tx=0x7f5dcc008040 comp rx=0 tx=0).stop 2024-04-25T01:17:54.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.154+0000 7f5de0a00640 1 -- 172.21.15.90:0/1953537731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ddc162290 msgr2=0x7f5ddc160150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:54.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.154+0000 7f5de0a00640 1 --2- 172.21.15.90:0/1953537731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ddc162290 0x7f5ddc160150 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4034040 tx=0x7f5dd4004a40 comp rx=0 tx=0).stop 2024-04-25T01:17:54.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.155+0000 7f5de0a00640 1 -- 172.21.15.90:0/1953537731 shutdown_connections 2024-04-25T01:17:54.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.155+0000 7f5de0a00640 1 --2- 172.21.15.90:0/1953537731 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f5dbc06be30 0x7f5dbc06e2f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:54.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.155+0000 7f5de0a00640 1 --2- 172.21.15.90:0/1953537731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ddc162290 0x7f5ddc160150 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:54.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.155+0000 7f5de0a00640 1 --2- 172.21.15.90:0/1953537731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ddc06e460 0x7f5ddc0b22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:54.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.155+0000 7f5de0a00640 1 --2- 172.21.15.90:0/1953537731 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5ddc06c9d0 0x7f5ddc0ada50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:54.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.155+0000 7f5de0a00640 1 -- 172.21.15.90:0/1953537731 >> 172.21.15.90:0/1953537731 conn(0x7f5ddc06b120 msgr2=0x7f5ddc072460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:54.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.155+0000 7f5de0a00640 1 -- 172.21.15.90:0/1953537731 shutdown_connections 2024-04-25T01:17:54.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.155+0000 7f5de0a00640 1 -- 172.21.15.90:0/1953537731 wait complete. 2024-04-25T01:17:54.169 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T01:17:54.170 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-25T01:17:54.170 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T01:17:54.170 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:17:54.170 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T01:17:54.170 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-25T01:17:54.170 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T01:17:54.170 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:17:54.171 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T01:17:54.171 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-25T01:17:54.171 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T01:17:54.171 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:17:54.171 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T01:17:54.171 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-25T01:17:54.171 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T01:17:54.172 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:17:54.189 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T01:17:54.189 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T01:17:54.193 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T01:17:54.193 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T01:17:54.196 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T01:17:54.196 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T01:17:54.213 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T01:17:54.214 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T01:17:54.241 INFO:tasks.ceph.mds.a:Started 2024-04-25T01:17:54.242 DEBUG:teuthology.parallel:result is None 2024-04-25T01:17:54.248 INFO:tasks.ceph.mds.c:Started 2024-04-25T01:17:54.248 DEBUG:teuthology.parallel:result is None 2024-04-25T01:17:54.261 INFO:tasks.ceph.mds.b:Started 2024-04-25T01:17:54.261 DEBUG:teuthology.parallel:result is None 2024-04-25T01:17:54.267 INFO:tasks.ceph.mds.d:Started 2024-04-25T01:17:54.267 DEBUG:teuthology.parallel:result is None 2024-04-25T01:17:54.267 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-25T01:17:54.388 INFO:tasks.ceph.mds.a.smithi090.stdout:starting mds.a at 2024-04-25T01:17:54.393 INFO:tasks.ceph.mds.c.smithi090.stdout:starting mds.c at 2024-04-25T01:17:54.414 INFO:tasks.ceph.mds.b.smithi103.stdout:starting mds.b at 2024-04-25T01:17:54.415 INFO:tasks.ceph.mds.d.smithi103.stdout:starting mds.d at 2024-04-25T01:17:54.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.463+0000 7f549ee00640 1 Processor -- start 2024-04-25T01:17:54.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.464+0000 7f549ee00640 1 -- start start 2024-04-25T01:17:54.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.465+0000 7f549ee00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f5498143940 0x7f549806e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:54.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.465+0000 7f549ee00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5498143d20 0x7f549806eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:54.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.465+0000 7f549ee00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f549815ecf0 0x7f549815f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:54.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.465+0000 7f549ee00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f5498070e60 con 0x7f5498143d20 2024-04-25T01:17:54.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.465+0000 7f549ee00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f5498071140 con 0x7f5498143940 2024-04-25T01:17:54.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.465+0000 7f549ee00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5498071420 con 0x7f549815ecf0 2024-04-25T01:17:54.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.465+0000 7f5497400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5498143d20 0x7f549806eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:54.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.465+0000 7f5497400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5498143d20 0x7f549806eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46340/0 (socket says 172.21.15.90:46340) 2024-04-25T01:17:54.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.465+0000 7f5497400640 1 -- 172.21.15.90:0/3774388242 learned_addr learned my addr 172.21.15.90:0/3774388242 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:54.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.465+0000 7f5497e00640 1 --2- 172.21.15.90:0/3774388242 >> v2:172.21.15.103:3300/0 conn(0x7f5498143940 0x7f549806e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:54.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.466+0000 7f549ce00640 1 --2- 172.21.15.90:0/3774388242 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f549815ecf0 0x7f549815f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:54.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.466+0000 7f5497400640 1 -- 172.21.15.90:0/3774388242 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f549815ecf0 msgr2=0x7f549815f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:54.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.466+0000 7f5497400640 1 --2- 172.21.15.90:0/3774388242 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f549815ecf0 0x7f549815f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:54.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.466+0000 7f5497400640 1 -- 172.21.15.90:0/3774388242 >> v2:172.21.15.103:3300/0 conn(0x7f5498143940 msgr2=0x7f549806e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:54.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.466+0000 7f5497400640 1 --2- 172.21.15.90:0/3774388242 >> v2:172.21.15.103:3300/0 conn(0x7f5498143940 0x7f549806e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:54.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.466+0000 7f5497400640 1 -- 172.21.15.90:0/3774388242 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5498071700 con 0x7f5498143d20 2024-04-25T01:17:54.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.467+0000 7f5497400640 1 --2- 172.21.15.90:0/3774388242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5498143d20 0x7f549806eb90 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f5484009870 tx=0x7f548402a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb043bbd39ce0a4d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:54.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.467+0000 7f5496a00640 1 -- 172.21.15.90:0/3774388242 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5484037070 con 0x7f5498143d20 2024-04-25T01:17:54.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.467+0000 7f5496a00640 1 -- 172.21.15.90:0/3774388242 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5484032440 con 0x7f5498143d20 2024-04-25T01:17:54.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.467+0000 7f5496a00640 1 -- 172.21.15.90:0/3774388242 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5484032810 con 0x7f5498143d20 2024-04-25T01:17:54.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.468+0000 7f549ee00640 1 -- 172.21.15.90:0/3774388242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5498143d20 msgr2=0x7f549806eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:54.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.468+0000 7f549ee00640 1 --2- 172.21.15.90:0/3774388242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5498143d20 0x7f549806eb90 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f5484009870 tx=0x7f548402a200 comp rx=0 tx=0).stop 2024-04-25T01:17:54.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.468+0000 7f549ee00640 1 -- 172.21.15.90:0/3774388242 shutdown_connections 2024-04-25T01:17:54.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.468+0000 7f549ee00640 1 --2- 172.21.15.90:0/3774388242 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f549815ecf0 0x7f549815f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:54.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.468+0000 7f549ee00640 1 --2- 172.21.15.90:0/3774388242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5498143d20 0x7f549806eb90 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:54.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.468+0000 7f549ee00640 1 --2- 172.21.15.90:0/3774388242 >> v2:172.21.15.103:3300/0 conn(0x7f5498143940 0x7f549806e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:54.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.468+0000 7f549ee00640 1 -- 172.21.15.90:0/3774388242 >> 172.21.15.90:0/3774388242 conn(0x7f54980b1820 msgr2=0x7f54980b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:54.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.468+0000 7f549ee00640 1 -- 172.21.15.90:0/3774388242 shutdown_connections 2024-04-25T01:17:54.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.468+0000 7f549ee00640 1 -- 172.21.15.90:0/3774388242 wait complete. 2024-04-25T01:17:54.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.470+0000 7f549ee00640 1 Processor -- start 2024-04-25T01:17:54.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.470+0000 7f549ee00640 1 -- start start 2024-04-25T01:17:54.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.471+0000 7f549ee00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5498143940 0x7f549815af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:54.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.471+0000 7f549ee00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5498143d20 0x7f54980a9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:54.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.471+0000 7f549ee00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f549815ecf0 0x7f5498158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:54.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.471+0000 7f549ee00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f54980b3a00 con 0x7f5498143d20 2024-04-25T01:17:54.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.471+0000 7f549ee00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f54980b3880 con 0x7f549815ecf0 2024-04-25T01:17:54.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.471+0000 7f549ee00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f54980b3b80 con 0x7f5498143940 2024-04-25T01:17:54.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.471+0000 7f5497400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5498143d20 0x7f54980a9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:54.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.471+0000 7f5497400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5498143d20 0x7f54980a9ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46344/0 (socket says 172.21.15.90:46344) 2024-04-25T01:17:54.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.471+0000 7f5497400640 1 -- 172.21.15.90:0/3046718983 learned_addr learned my addr 172.21.15.90:0/3046718983 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:54.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.472+0000 7f549ce00640 1 --2- 172.21.15.90:0/3046718983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f549815ecf0 0x7f5498158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:54.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.472+0000 7f5497e00640 1 --2- 172.21.15.90:0/3046718983 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5498143940 0x7f549815af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:54.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.472+0000 7f5497400640 1 -- 172.21.15.90:0/3046718983 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5498143940 msgr2=0x7f549815af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:54.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.472+0000 7f5497400640 1 --2- 172.21.15.90:0/3046718983 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5498143940 0x7f549815af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:54.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.472+0000 7f5497400640 1 -- 172.21.15.90:0/3046718983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f549815ecf0 msgr2=0x7f5498158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:54.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.472+0000 7f5497400640 1 --2- 172.21.15.90:0/3046718983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f549815ecf0 0x7f5498158eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:54.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.472+0000 7f5497400640 1 -- 172.21.15.90:0/3046718983 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54980aa850 con 0x7f5498143d20 2024-04-25T01:17:54.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.472+0000 7f5497e00640 1 --2- 172.21.15.90:0/3046718983 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5498143940 0x7f549815af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:17:54.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.472+0000 7f549ce00640 1 --2- 172.21.15.90:0/3046718983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f549815ecf0 0x7f5498158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:17:54.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.472+0000 7f5497400640 1 --2- 172.21.15.90:0/3046718983 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5498143d20 0x7f54980a9ee0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f5484000c00 tx=0x7f54840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:54.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.472+0000 7f548fe00640 1 -- 172.21.15.90:0/3046718983 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5484041020 con 0x7f5498143d20 2024-04-25T01:17:54.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.472+0000 7f548fe00640 1 -- 172.21.15.90:0/3046718983 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f548403d620 con 0x7f5498143d20 2024-04-25T01:17:54.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.472+0000 7f549ee00640 1 -- 172.21.15.90:0/3046718983 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54980aab30 con 0x7f5498143d20 2024-04-25T01:17:54.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.473+0000 7f548fe00640 1 -- 172.21.15.90:0/3046718983 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5484037040 con 0x7f5498143d20 2024-04-25T01:17:54.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.473+0000 7f549ee00640 1 -- 172.21.15.90:0/3046718983 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54980ab0f0 con 0x7f5498143d20 2024-04-25T01:17:54.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.474+0000 7f549ee00640 1 -- 172.21.15.90:0/3046718983 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5498068fb0 con 0x7f5498143d20 2024-04-25T01:17:54.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.475+0000 7f548fe00640 1 -- 172.21.15.90:0/3046718983 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5484044020 con 0x7f5498143d20 2024-04-25T01:17:54.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.475+0000 7f548fe00640 1 --2- 172.21.15.90:0/3046718983 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f547806be30 0x7f547806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:54.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.476+0000 7f548fe00640 1 -- 172.21.15.90:0/3046718983 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5335+0+0 (secure 0 0 0) 0x7f54840ab440 con 0x7f5498143d20 2024-04-25T01:17:54.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.476+0000 7f5497e00640 1 --2- 172.21.15.90:0/3046718983 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f547806be30 0x7f547806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:54.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.477+0000 7f5497e00640 1 --2- 172.21.15.90:0/3046718983 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f547806be30 0x7f547806e2f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5488009b10 tx=0x7f5488008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:54.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.480+0000 7f548fe00640 1 -- 172.21.15.90:0/3046718983 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f548404b030 con 0x7f5498143d20 2024-04-25T01:17:54.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:54.710+0000 7f549ee00640 1 -- 172.21.15.90:0/3046718983 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f549806d150 con 0x7f5498143d20 2024-04-25T01:17:55.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.174+0000 7f548fe00640 1 -- 172.21.15.90:0/3046718983 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v34) ==== 97+0+0 (secure 0 0 0) 0x7f5484079ec0 con 0x7f5498143d20 2024-04-25T01:17:55.158 INFO:teuthology.orchestra.run.smithi090.stderr: removed all blocklist entries 2024-04-25T01:17:55.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.178+0000 7f549ee00640 1 -- 172.21.15.90:0/3046718983 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f547806be30 msgr2=0x7f547806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:55.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.178+0000 7f549ee00640 1 --2- 172.21.15.90:0/3046718983 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f547806be30 0x7f547806e2f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5488009b10 tx=0x7f5488008040 comp rx=0 tx=0).stop 2024-04-25T01:17:55.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.178+0000 7f549ee00640 1 -- 172.21.15.90:0/3046718983 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5498143d20 msgr2=0x7f54980a9ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:55.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.178+0000 7f549ee00640 1 --2- 172.21.15.90:0/3046718983 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5498143d20 0x7f54980a9ee0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f5484000c00 tx=0x7f54840046c0 comp rx=0 tx=0).stop 2024-04-25T01:17:55.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.178+0000 7f549ee00640 1 -- 172.21.15.90:0/3046718983 shutdown_connections 2024-04-25T01:17:55.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.178+0000 7f549ee00640 1 --2- 172.21.15.90:0/3046718983 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f547806be30 0x7f547806e2f0 secure :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5488009b10 tx=0x7f5488008040 comp rx=0 tx=0).stop 2024-04-25T01:17:55.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.178+0000 7f549ee00640 1 --2- 172.21.15.90:0/3046718983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f549815ecf0 0x7f5498158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:55.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.178+0000 7f549ee00640 1 --2- 172.21.15.90:0/3046718983 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5498143d20 0x7f54980a9ee0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:55.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.178+0000 7f549ee00640 1 --2- 172.21.15.90:0/3046718983 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5498143940 0x7f549815af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:55.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.178+0000 7f549ee00640 1 -- 172.21.15.90:0/3046718983 >> 172.21.15.90:0/3046718983 conn(0x7f54980b1820 msgr2=0x7f54980cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:55.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.178+0000 7f549ee00640 1 -- 172.21.15.90:0/3046718983 shutdown_connections 2024-04-25T01:17:55.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.179+0000 7f549ee00640 1 -- 172.21.15.90:0/3046718983 wait complete. 2024-04-25T01:17:55.185 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-25T01:17:55.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.387+0000 7fca7a600640 1 Processor -- start 2024-04-25T01:17:55.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.387+0000 7fca7a600640 1 -- start start 2024-04-25T01:17:55.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.388+0000 7fca7a600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fca740aea00 0x7fca7415e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:55.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.388+0000 7fca7a600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca74072930 con 0x7fca7415a380 2024-04-25T01:17:55.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.388+0000 7fca7a600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca7406e030 con 0x7fca740b4590 2024-04-25T01:17:55.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.388+0000 7fca7a600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fca7406e3a0 con 0x7fca740aea00 2024-04-25T01:17:55.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.388+0000 7fca79200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fca7415a380 0x7fca740b3e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:54668/0 (socket says 172.21.15.90:54668) 2024-04-25T01:17:55.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.388+0000 7fca79200640 1 -- 172.21.15.90:0/3089842052 learned_addr learned my addr 172.21.15.90:0/3089842052 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:55.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.389+0000 7fca73e00640 1 --2- 172.21.15.90:0/3089842052 >> v2:172.21.15.103:3300/0 conn(0x7fca740aea00 0x7fca7415e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:55.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.389+0000 7fca73400640 1 -- 172.21.15.90:0/3089842052 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 193789618 0 0) 0x7fca74072930 con 0x7fca7415a380 2024-04-25T01:17:55.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.389+0000 7fca73400640 1 -- 172.21.15.90:0/3089842052 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca58003520 con 0x7fca7415a380 2024-04-25T01:17:55.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.389+0000 7fca73400640 1 -- 172.21.15.90:0/3089842052 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1962824064 0 0) 0x7fca58003520 con 0x7fca7415a380 2024-04-25T01:17:55.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.389+0000 7fca73400640 1 -- 172.21.15.90:0/3089842052 >> v2:172.21.15.103:3300/0 conn(0x7fca740aea00 msgr2=0x7fca7415e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:55.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.389+0000 7fca73400640 1 --2- 172.21.15.90:0/3089842052 >> v2:172.21.15.103:3300/0 conn(0x7fca740aea00 0x7fca7415e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:55.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.389+0000 7fca73400640 1 -- 172.21.15.90:0/3089842052 >> v1:172.21.15.103:6789/0 conn(0x7fca740b4590 legacy=0x7fca74070a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:55.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.389+0000 7fca73400640 1 -- 172.21.15.90:0/3089842052 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca7415a760 con 0x7fca7415a380 2024-04-25T01:17:55.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.389+0000 7fca73400640 1 -- 172.21.15.90:0/3089842052 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7fca5c001670 con 0x7fca7415a380 2024-04-25T01:17:55.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.389+0000 7fca73400640 1 -- 172.21.15.90:0/3089842052 >> v1:172.21.15.90:6789/0 conn(0x7fca7415a380 legacy=0x7fca740b3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:55.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.389+0000 7fca73400640 1 --2- 172.21.15.90:0/3089842052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca580039b0 0x7fca58003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:55.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.389+0000 7fca73400640 1 --2- 172.21.15.90:0/3089842052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca58004800 0x7fca58004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:55.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.389+0000 7fca73400640 1 --2- 172.21.15.90:0/3089842052 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca580051c0 0x7fca5800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:55.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.389+0000 7fca73400640 1 -- 172.21.15.90:0/3089842052 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fca7406e3a0 con 0x7fca580039b0 2024-04-25T01:17:55.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.389+0000 7fca73400640 1 -- 172.21.15.90:0/3089842052 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fca5800c040 con 0x7fca58004800 2024-04-25T01:17:55.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.389+0000 7fca73400640 1 -- 172.21.15.90:0/3089842052 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fca5800c240 con 0x7fca580051c0 2024-04-25T01:17:55.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.389+0000 7fca79200640 1 --2- 172.21.15.90:0/3089842052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca580039b0 0x7fca58003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:55.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.390+0000 7fca73e00640 1 --2- 172.21.15.90:0/3089842052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca58004800 0x7fca58004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:55.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.390+0000 7fca79200640 1 -- 172.21.15.90:0/3089842052 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca580051c0 msgr2=0x7fca5800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:55.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.390+0000 7fca79200640 1 --2- 172.21.15.90:0/3089842052 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca580051c0 0x7fca5800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:55.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.390+0000 7fca79200640 1 -- 172.21.15.90:0/3089842052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca58004800 msgr2=0x7fca58004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:55.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.390+0000 7fca79200640 1 --2- 172.21.15.90:0/3089842052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca58004800 0x7fca58004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:55.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.390+0000 7fca79200640 1 -- 172.21.15.90:0/3089842052 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca58003520 con 0x7fca580039b0 2024-04-25T01:17:55.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.390+0000 7fca79c00640 1 -- 172.21.15.90:0/3089842052 reap_dead start 2024-04-25T01:17:55.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.391+0000 7fca79200640 1 --2- 172.21.15.90:0/3089842052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca580039b0 0x7fca58003db0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fca5c00c240 tx=0x7fca58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18dc5fe874c80ddc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:55.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.391+0000 7fca73400640 1 -- 172.21.15.90:0/3089842052 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca5c02e070 con 0x7fca580039b0 2024-04-25T01:17:55.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.391+0000 7fca73400640 1 -- 172.21.15.90:0/3089842052 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca5c005af0 con 0x7fca580039b0 2024-04-25T01:17:55.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.391+0000 7fca73400640 1 -- 172.21.15.90:0/3089842052 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca5c007d30 con 0x7fca580039b0 2024-04-25T01:17:55.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.392+0000 7fca7a600640 1 -- 172.21.15.90:0/3089842052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca580039b0 msgr2=0x7fca58003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:55.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.392+0000 7fca7a600640 1 --2- 172.21.15.90:0/3089842052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca580039b0 0x7fca58003db0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fca5c00c240 tx=0x7fca58005830 comp rx=0 tx=0).stop 2024-04-25T01:17:55.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.392+0000 7fca7a600640 1 -- 172.21.15.90:0/3089842052 shutdown_connections 2024-04-25T01:17:55.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.392+0000 7fca7a600640 1 --2- 172.21.15.90:0/3089842052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca580039b0 0x7fca58003db0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:55.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.392+0000 7fca7a600640 1 -- 172.21.15.90:0/3089842052 >> 172.21.15.90:0/3089842052 conn(0x7fca740b1660 msgr2=0x7fca740b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:55.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.392+0000 7fca7a600640 1 -- 172.21.15.90:0/3089842052 shutdown_connections 2024-04-25T01:17:55.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.392+0000 7fca7a600640 1 -- 172.21.15.90:0/3089842052 wait complete. 2024-04-25T01:17:55.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.394+0000 7fca7a600640 1 Processor -- start 2024-04-25T01:17:55.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.394+0000 7fca7a600640 1 -- start start 2024-04-25T01:17:55.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.395+0000 7fca7a600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca580039b0 0x7fca741552c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:55.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.395+0000 7fca7a600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca741593a0 0x7fca741566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:55.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.395+0000 7fca7a600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca74156c10 0x7fca741e4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:55.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.395+0000 7fca7a600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fca7415e970 con 0x7fca580039b0 2024-04-25T01:17:55.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.395+0000 7fca7a600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fca740af150 con 0x7fca741593a0 2024-04-25T01:17:55.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.395+0000 7fca7a600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fca7415eaf0 con 0x7fca74156c10 2024-04-25T01:17:55.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.395+0000 7fca79200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca580039b0 0x7fca741552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:55.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.395+0000 7fca79200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca580039b0 0x7fca741552c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56952/0 (socket says 172.21.15.90:56952) 2024-04-25T01:17:55.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.395+0000 7fca79200640 1 -- 172.21.15.90:0/2075342467 learned_addr learned my addr 172.21.15.90:0/2075342467 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:55.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.395+0000 7fca79c00640 1 --2- 172.21.15.90:0/2075342467 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca74156c10 0x7fca741e4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:55.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.395+0000 7fca73e00640 1 --2- 172.21.15.90:0/2075342467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca741593a0 0x7fca741566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:55.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.396+0000 7fca79200640 1 -- 172.21.15.90:0/2075342467 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca74156c10 msgr2=0x7fca741e4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:55.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.396+0000 7fca79200640 1 --2- 172.21.15.90:0/2075342467 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca74156c10 0x7fca741e4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:55.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.396+0000 7fca79200640 1 -- 172.21.15.90:0/2075342467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca741593a0 msgr2=0x7fca741566d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:55.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.396+0000 7fca79200640 1 --2- 172.21.15.90:0/2075342467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca741593a0 0x7fca741566d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:55.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.396+0000 7fca79200640 1 -- 172.21.15.90:0/2075342467 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca741e3c60 con 0x7fca580039b0 2024-04-25T01:17:55.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.396+0000 7fca79200640 1 --2- 172.21.15.90:0/2075342467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca580039b0 0x7fca741552c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fca5c00c360 tx=0x7fca7406e310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:55.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.396+0000 7fca70c00640 1 -- 172.21.15.90:0/2075342467 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca5c042020 con 0x7fca580039b0 2024-04-25T01:17:55.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.396+0000 7fca70c00640 1 -- 172.21.15.90:0/2075342467 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca5c004bf0 con 0x7fca580039b0 2024-04-25T01:17:55.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.396+0000 7fca70c00640 1 -- 172.21.15.90:0/2075342467 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca5c007070 con 0x7fca580039b0 2024-04-25T01:17:55.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.396+0000 7fca7a600640 1 -- 172.21.15.90:0/2075342467 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca741e3f40 con 0x7fca580039b0 2024-04-25T01:17:55.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.396+0000 7fca7a600640 1 -- 172.21.15.90:0/2075342467 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca741e4500 con 0x7fca580039b0 2024-04-25T01:17:55.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.398+0000 7fca7a600640 1 -- 172.21.15.90:0/2075342467 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca74075350 con 0x7fca580039b0 2024-04-25T01:17:55.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.398+0000 7fca70c00640 1 -- 172.21.15.90:0/2075342467 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fca5c045020 con 0x7fca580039b0 2024-04-25T01:17:55.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.399+0000 7fca70c00640 1 --2- 172.21.15.90:0/2075342467 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fca5006be30 0x7fca5006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:55.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.399+0000 7fca70c00640 1 -- 172.21.15.90:0/2075342467 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5249+0+0 (secure 0 0 0) 0x7fca5c0ab310 con 0x7fca580039b0 2024-04-25T01:17:55.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.400+0000 7fca73e00640 1 --2- 172.21.15.90:0/2075342467 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fca5006be30 0x7fca5006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:55.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.400+0000 7fca73e00640 1 --2- 172.21.15.90:0/2075342467 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fca5006be30 0x7fca5006e2f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fca74071b10 tx=0x7fca64008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:55.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.404+0000 7fca70c00640 1 -- 172.21.15.90:0/2075342467 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fca5c079e70 con 0x7fca580039b0 2024-04-25T01:17:55.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.679+0000 7fca7a600640 1 -- 172.21.15.90:0/2075342467 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fca7406bf90 con 0x7fca580039b0 2024-04-25T01:17:55.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.680+0000 7fca70c00640 1 -- 172.21.15.90:0/2075342467 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) ==== 80+0+6645 (secure 0 0 0) 0x7fca5c0ab0d0 con 0x7fca580039b0 2024-04-25T01:17:55.665 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:17:55.665 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T01:17:55.665 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_dump": [ 2024-04-25T01:17:55.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T01:17:55.665 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mds.a", 2024-04-25T01:17:55.665 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCrrilmjb9qFBAAlnPjHsgFagy1je4+qyjyOQ==", 2024-04-25T01:17:55.665 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-04-25T01:17:55.665 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-04-25T01:17:55.665 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow *", 2024-04-25T01:17:55.665 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow *", 2024-04-25T01:17:55.665 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-04-25T01:17:55.665 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:17:55.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T01:17:55.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T01:17:55.666 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mds.b", 2024-04-25T01:17:55.666 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCrrilmUZHfMBAAbh51emKl6HSLVHIaUOllXA==", 2024-04-25T01:17:55.666 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-04-25T01:17:55.666 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-04-25T01:17:55.666 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow *", 2024-04-25T01:17:55.666 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow *", 2024-04-25T01:17:55.666 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-04-25T01:17:55.666 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:17:55.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T01:17:55.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T01:17:55.666 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mds.c", 2024-04-25T01:17:55.666 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCrrilmmi9BIhAA2s5UuqgNa1mTnzELh6kg1Q==", 2024-04-25T01:17:55.666 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-04-25T01:17:55.666 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-04-25T01:17:55.667 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow *", 2024-04-25T01:17:55.667 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow *", 2024-04-25T01:17:55.667 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-04-25T01:17:55.667 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:17:55.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T01:17:55.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T01:17:55.667 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mds.d", 2024-04-25T01:17:55.667 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCsrilm/zZiARAAP8D+OSj0K9A3NGjnI+NOLA==", 2024-04-25T01:17:55.667 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-04-25T01:17:55.667 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-04-25T01:17:55.667 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow *", 2024-04-25T01:17:55.667 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow *", 2024-04-25T01:17:55.667 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-04-25T01:17:55.667 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:17:55.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T01:17:55.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T01:17:55.668 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.0", 2024-04-25T01:17:55.668 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCvrilmONAAOhAAMfwg6+ChCofN/OS6Jp7DLg==", 2024-04-25T01:17:55.668 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-04-25T01:17:55.668 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-04-25T01:17:55.668 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-04-25T01:17:55.668 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-04-25T01:17:55.668 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:17:55.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T01:17:55.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T01:17:55.668 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.1", 2024-04-25T01:17:55.668 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCzrilmaKqpGhAAMFnbJ9tBiOgP7aUcxHxdAQ==", 2024-04-25T01:17:55.668 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-04-25T01:17:55.668 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-04-25T01:17:55.669 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-04-25T01:17:55.669 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-04-25T01:17:55.669 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:17:55.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T01:17:55.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T01:17:55.669 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.2", 2024-04-25T01:17:55.669 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQC2rilm9bRPNBAAh9uf5qeKxKVGZjeURZN2ng==", 2024-04-25T01:17:55.669 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-04-25T01:17:55.669 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-04-25T01:17:55.669 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-04-25T01:17:55.669 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-04-25T01:17:55.669 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:17:55.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T01:17:55.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T01:17:55.669 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.3", 2024-04-25T01:17:55.670 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQC6rilm335uAxAAzL5VZalis0ydEKCAc9p20w==", 2024-04-25T01:17:55.670 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-04-25T01:17:55.670 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-04-25T01:17:55.670 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-04-25T01:17:55.670 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-04-25T01:17:55.670 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:17:55.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T01:17:55.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T01:17:55.670 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.4", 2024-04-25T01:17:55.671 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDArilmkRNZDRAAPLVSDr1+ERL09nIchRCNVw==", 2024-04-25T01:17:55.671 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-04-25T01:17:55.671 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-04-25T01:17:55.671 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-04-25T01:17:55.671 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-04-25T01:17:55.671 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:17:55.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T01:17:55.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T01:17:55.671 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.5", 2024-04-25T01:17:55.671 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDDrilmM3k5GRAA6toW36myIctvYTYY5kzayA==", 2024-04-25T01:17:55.671 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-04-25T01:17:55.671 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-04-25T01:17:55.671 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-04-25T01:17:55.671 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-04-25T01:17:55.672 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:17:55.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T01:17:55.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T01:17:55.672 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.6", 2024-04-25T01:17:55.672 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDGrilmZJEXNRAARn43EOUZQzSdNDmifLDQpg==", 2024-04-25T01:17:55.672 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-04-25T01:17:55.672 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-04-25T01:17:55.672 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-04-25T01:17:55.672 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-04-25T01:17:55.672 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:17:55.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T01:17:55.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T01:17:55.672 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.7", 2024-04-25T01:17:55.672 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDKrilm60XzBRAAz37R48OIHSB4petorwdLDQ==", 2024-04-25T01:17:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-04-25T01:17:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-04-25T01:17:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-04-25T01:17:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-04-25T01:17:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:17:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T01:17:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T01:17:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.0", 2024-04-25T01:17:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCsrilmTsovDhAAVqcIHshdGrSHncSLyuueoQ==", 2024-04-25T01:17:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-04-25T01:17:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-04-25T01:17:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow r", 2024-04-25T01:17:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow rw", 2024-04-25T01:17:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow rwx" 2024-04-25T01:17:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:17:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T01:17:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T01:17:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.1", 2024-04-25T01:17:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCsrilmUIgRIRAAmTYrl9YlFG/l6hAHrMhYAQ==", 2024-04-25T01:17:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-04-25T01:17:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-04-25T01:17:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow r", 2024-04-25T01:17:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow rw", 2024-04-25T01:17:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow rwx" 2024-04-25T01:17:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:17:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T01:17:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T01:17:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.2", 2024-04-25T01:17:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCsrilmLMVnFxAAQv8hYmJ1f8WsFc4+C6QFWg==", 2024-04-25T01:17:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-04-25T01:17:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-04-25T01:17:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow r", 2024-04-25T01:17:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow rw", 2024-04-25T01:17:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow rwx" 2024-04-25T01:17:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:17:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T01:17:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T01:17:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.3", 2024-04-25T01:17:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCsrilmYhNVKhAAsFW+T2gQAEsOl8EFhBicyg==", 2024-04-25T01:17:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-04-25T01:17:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-04-25T01:17:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow r", 2024-04-25T01:17:55.676 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow rw", 2024-04-25T01:17:55.676 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow rwx" 2024-04-25T01:17:55.676 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:17:55.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T01:17:55.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T01:17:55.676 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.admin", 2024-04-25T01:17:55.676 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCqrilmh/iHEBAAgsZHeCTqYg+KFJfuoKK7Mw==", 2024-04-25T01:17:55.676 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-04-25T01:17:55.676 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow *", 2024-04-25T01:17:55.676 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow *", 2024-04-25T01:17:55.676 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow *", 2024-04-25T01:17:55.676 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-04-25T01:17:55.676 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:17:55.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T01:17:55.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T01:17:55.677 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.bootstrap-mds", 2024-04-25T01:17:55.677 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDSrilmcF21LxAABkObFAmGG4uT3Dtoj++uhA==", 2024-04-25T01:17:55.677 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-04-25T01:17:55.677 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile bootstrap-mds" 2024-04-25T01:17:55.677 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:17:55.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T01:17:55.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T01:17:55.677 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.bootstrap-mgr", 2024-04-25T01:17:55.677 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDSrilmFmy1LxAAOuDEu6mBJkcawKNOAFKnhg==", 2024-04-25T01:17:55.677 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-04-25T01:17:55.677 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile bootstrap-mgr" 2024-04-25T01:17:55.677 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:17:55.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T01:17:55.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T01:17:55.678 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.bootstrap-osd", 2024-04-25T01:17:55.678 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDSrilmRHq1LxAAf/8W65rjdoV/eU1qncnUIA==", 2024-04-25T01:17:55.678 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-04-25T01:17:55.678 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile bootstrap-osd" 2024-04-25T01:17:55.678 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:17:55.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T01:17:55.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T01:17:55.678 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.bootstrap-rbd", 2024-04-25T01:17:55.678 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDSrilmU4a1LxAApdLev+Mthi6kOOeny3ywVA==", 2024-04-25T01:17:55.678 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-04-25T01:17:55.678 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile bootstrap-rbd" 2024-04-25T01:17:55.678 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:17:55.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T01:17:55.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T01:17:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-04-25T01:17:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDSrilm15K1LxAA8h55ARcl7HQZtl7n3drzag==", 2024-04-25T01:17:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-04-25T01:17:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-04-25T01:17:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:17:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T01:17:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T01:17:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.bootstrap-rgw", 2024-04-25T01:17:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDSrilmaJ+1LxAAi8TLkXQBK4I7NnTkR/nCRA==", 2024-04-25T01:17:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-04-25T01:17:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile bootstrap-rgw" 2024-04-25T01:17:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:17:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T01:17:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T01:17:55.680 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mgr.x", 2024-04-25T01:17:55.680 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCrrilm5IGgChAALBQG+sk4Pr+UuJWnHXh/ZA==", 2024-04-25T01:17:55.680 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-04-25T01:17:55.680 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow *", 2024-04-25T01:17:55.680 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile mgr", 2024-04-25T01:17:55.680 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-04-25T01:17:55.680 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:17:55.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T01:17:55.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T01:17:55.680 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mgr.y", 2024-04-25T01:17:55.680 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCqrilm53U1OxAAV/IYzYKURT+G44pUIEkejg==", 2024-04-25T01:17:55.680 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-04-25T01:17:55.680 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow *", 2024-04-25T01:17:55.680 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile mgr", 2024-04-25T01:17:55.681 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-04-25T01:17:55.681 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:17:55.681 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:17:55.681 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-25T01:17:55.681 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T01:17:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.683+0000 7fca7a600640 1 -- 172.21.15.90:0/2075342467 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fca5006be30 msgr2=0x7fca5006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.683+0000 7fca7a600640 1 --2- 172.21.15.90:0/2075342467 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fca5006be30 0x7fca5006e2f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fca74071b10 tx=0x7fca64008040 comp rx=0 tx=0).stop 2024-04-25T01:17:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.683+0000 7fca7a600640 1 -- 172.21.15.90:0/2075342467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca580039b0 msgr2=0x7fca741552c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.683+0000 7fca7a600640 1 --2- 172.21.15.90:0/2075342467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca580039b0 0x7fca741552c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fca5c00c360 tx=0x7fca7406e310 comp rx=0 tx=0).stop 2024-04-25T01:17:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.683+0000 7fca7a600640 1 -- 172.21.15.90:0/2075342467 shutdown_connections 2024-04-25T01:17:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.683+0000 7fca7a600640 1 --2- 172.21.15.90:0/2075342467 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fca5006be30 0x7fca5006e2f0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.683+0000 7fca7a600640 1 --2- 172.21.15.90:0/2075342467 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fca74156c10 0x7fca741e4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.683+0000 7fca7a600640 1 --2- 172.21.15.90:0/2075342467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca741593a0 0x7fca741566d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.683+0000 7fca7a600640 1 --2- 172.21.15.90:0/2075342467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca580039b0 0x7fca741552c0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.683+0000 7fca7a600640 1 -- 172.21.15.90:0/2075342467 >> 172.21.15.90:0/2075342467 conn(0x7fca740b1660 msgr2=0x7fca740ceeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:55.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.683+0000 7fca7a600640 1 -- 172.21.15.90:0/2075342467 shutdown_connections 2024-04-25T01:17:55.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.683+0000 7fca7a600640 1 -- 172.21.15.90:0/2075342467 wait complete. 2024-04-25T01:17:55.685 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-04-25T01:17:55.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.909+0000 7ff85ee00640 1 Processor -- start 2024-04-25T01:17:55.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.909+0000 7ff85ee00640 1 -- start start 2024-04-25T01:17:55.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.911+0000 7ff85ee00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff85815f9b0 0x7ff85815fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:55.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.911+0000 7ff85ee00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8580b4240 con 0x7ff858145590 2024-04-25T01:17:55.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.911+0000 7ff85ee00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8580b39a0 con 0x7ff85806d270 2024-04-25T01:17:55.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.911+0000 7ff85ee00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7ff858160830 con 0x7ff85815f9b0 2024-04-25T01:17:55.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.911+0000 7ff857e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff858145590 0x7ff8580b4a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:54674/0 (socket says 172.21.15.90:54674) 2024-04-25T01:17:55.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.911+0000 7ff857e00640 1 -- 172.21.15.90:0/993118993 learned_addr learned my addr 172.21.15.90:0/993118993 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:55.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.911+0000 7ff857400640 1 --2- 172.21.15.90:0/993118993 >> v2:172.21.15.90:3300/0 conn(0x7ff85815f9b0 0x7ff85815fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:55.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.912+0000 7ff856a00640 1 -- 172.21.15.90:0/993118993 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2330588846 0 0) 0x7ff8580b4240 con 0x7ff858145590 2024-04-25T01:17:55.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.912+0000 7ff856a00640 1 -- 172.21.15.90:0/993118993 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff838003520 con 0x7ff858145590 2024-04-25T01:17:55.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.912+0000 7ff857400640 1 -- 172.21.15.90:0/993118993 >> v1:172.21.15.103:6789/0 conn(0x7ff85806d270 legacy=0x7ff85806d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:55.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.912+0000 7ff857400640 1 -- 172.21.15.90:0/993118993 >> v1:172.21.15.90:6789/0 conn(0x7ff858145590 legacy=0x7ff8580b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:55.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.912+0000 7ff857400640 1 -- 172.21.15.90:0/993118993 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8580b3c80 con 0x7ff85815f9b0 2024-04-25T01:17:55.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.913+0000 7ff857400640 1 --2- 172.21.15.90:0/993118993 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff85815f9b0 0x7ff85815fe10 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7ff840009d90 tx=0x7ff84002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f09add18cbf269b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:55.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.913+0000 7ff856a00640 1 -- 172.21.15.90:0/993118993 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff840007740 con 0x7ff85815f9b0 2024-04-25T01:17:55.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.913+0000 7ff856a00640 1 -- 172.21.15.90:0/993118993 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8400078e0 con 0x7ff85815f9b0 2024-04-25T01:17:55.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.914+0000 7ff856a00640 1 -- 172.21.15.90:0/993118993 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff840007cb0 con 0x7ff85815f9b0 2024-04-25T01:17:55.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.914+0000 7ff85ee00640 1 -- 172.21.15.90:0/993118993 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff85815f9b0 msgr2=0x7ff85815fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:55.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.914+0000 7ff85ee00640 1 --2- 172.21.15.90:0/993118993 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff85815f9b0 0x7ff85815fe10 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7ff840009d90 tx=0x7ff84002a6a0 comp rx=0 tx=0).stop 2024-04-25T01:17:55.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.914+0000 7ff85ee00640 1 -- 172.21.15.90:0/993118993 shutdown_connections 2024-04-25T01:17:55.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.914+0000 7ff85ee00640 1 --2- 172.21.15.90:0/993118993 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff85815f9b0 0x7ff85815fe10 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:55.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.914+0000 7ff85ee00640 1 -- 172.21.15.90:0/993118993 >> 172.21.15.90:0/993118993 conn(0x7ff8580b1820 msgr2=0x7ff8580b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:55.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.914+0000 7ff85ee00640 1 -- 172.21.15.90:0/993118993 shutdown_connections 2024-04-25T01:17:55.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.914+0000 7ff85ee00640 1 -- 172.21.15.90:0/993118993 wait complete. 2024-04-25T01:17:55.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.916+0000 7ff85ee00640 1 Processor -- start 2024-04-25T01:17:55.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.916+0000 7ff85ee00640 1 -- start start 2024-04-25T01:17:55.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.917+0000 7ff85ee00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff85806d270 0x7ff8580aee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:55.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.917+0000 7ff85ee00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff858145590 0x7ff85815b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:55.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.917+0000 7ff85ee00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff85815f9b0 0x7ff858071850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:55.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.917+0000 7ff85ee00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff85806cde0 con 0x7ff85806d270 2024-04-25T01:17:55.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.917+0000 7ff85ee00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff858145970 con 0x7ff858145590 2024-04-25T01:17:55.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.917+0000 7ff85ee00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff8580b3530 con 0x7ff85815f9b0 2024-04-25T01:17:55.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.917+0000 7ff857e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff85806d270 0x7ff8580aee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:55.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.918+0000 7ff857e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff85806d270 0x7ff8580aee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56966/0 (socket says 172.21.15.90:56966) 2024-04-25T01:17:55.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.918+0000 7ff857e00640 1 -- 172.21.15.90:0/370572583 learned_addr learned my addr 172.21.15.90:0/370572583 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:55.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.918+0000 7ff85ce00640 1 --2- 172.21.15.90:0/370572583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff85815f9b0 0x7ff858071850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:55.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.918+0000 7ff857e00640 1 -- 172.21.15.90:0/370572583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff85815f9b0 msgr2=0x7ff858071850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:55.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.918+0000 7ff857400640 1 --2- 172.21.15.90:0/370572583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff858145590 0x7ff85815b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:55.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.918+0000 7ff857e00640 1 --2- 172.21.15.90:0/370572583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff85815f9b0 0x7ff858071850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:55.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.918+0000 7ff857e00640 1 -- 172.21.15.90:0/370572583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff858145590 msgr2=0x7ff85815b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:55.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.918+0000 7ff857e00640 1 --2- 172.21.15.90:0/370572583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff858145590 0x7ff85815b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:55.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.918+0000 7ff857e00640 1 -- 172.21.15.90:0/370572583 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff858071d90 con 0x7ff85806d270 2024-04-25T01:17:55.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.918+0000 7ff85ce00640 1 --2- 172.21.15.90:0/370572583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff85815f9b0 0x7ff858071850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:55.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.918+0000 7ff857e00640 1 --2- 172.21.15.90:0/370572583 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff85806d270 0x7ff8580aee20 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7ff84800c460 tx=0x7ff84800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:55.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.919+0000 7ff84fe00640 1 -- 172.21.15.90:0/370572583 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff848017070 con 0x7ff85806d270 2024-04-25T01:17:55.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.919+0000 7ff84fe00640 1 -- 172.21.15.90:0/370572583 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff848005ce0 con 0x7ff85806d270 2024-04-25T01:17:55.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.919+0000 7ff84fe00640 1 -- 172.21.15.90:0/370572583 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff848007290 con 0x7ff85806d270 2024-04-25T01:17:55.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.919+0000 7ff85ee00640 1 -- 172.21.15.90:0/370572583 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff858072030 con 0x7ff85806d270 2024-04-25T01:17:55.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.919+0000 7ff85ee00640 1 -- 172.21.15.90:0/370572583 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8580725f0 con 0x7ff85806d270 2024-04-25T01:17:55.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.920+0000 7ff85ee00640 1 -- 172.21.15.90:0/370572583 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff858068fb0 con 0x7ff85806d270 2024-04-25T01:17:55.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.921+0000 7ff84fe00640 1 -- 172.21.15.90:0/370572583 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff84801a070 con 0x7ff85806d270 2024-04-25T01:17:55.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.922+0000 7ff84fe00640 1 --2- 172.21.15.90:0/370572583 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff83406be30 0x7ff83406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:55.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.922+0000 7ff84fe00640 1 -- 172.21.15.90:0/370572583 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5249+0+0 (secure 0 0 0) 0x7ff84808a100 con 0x7ff85806d270 2024-04-25T01:17:55.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.922+0000 7ff857400640 1 --2- 172.21.15.90:0/370572583 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff83406be30 0x7ff83406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:55.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.923+0000 7ff857400640 1 --2- 172.21.15.90:0/370572583 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff83406be30 0x7ff83406e2f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff840034000 tx=0x7ff840033040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:55.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:55.927+0000 7ff84fe00640 1 -- 172.21.15.90:0/370572583 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff848058bd0 con 0x7ff85806d270 2024-04-25T01:17:56.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.200+0000 7ff85ee00640 1 -- 172.21.15.90:0/370572583 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7ff85806ea50 con 0x7ff85806d270 2024-04-25T01:17:56.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.204+0000 7ff84fe00640 1 -- 172.21.15.90:0/370572583 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v7) ==== 90+0+0 (secure 0 0 0) 0x7ff8480584d0 con 0x7ff85806d270 2024-04-25T01:17:56.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.207+0000 7ff85ee00640 1 -- 172.21.15.90:0/370572583 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff83406be30 msgr2=0x7ff83406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:56.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.207+0000 7ff85ee00640 1 --2- 172.21.15.90:0/370572583 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff83406be30 0x7ff83406e2f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff840034000 tx=0x7ff840033040 comp rx=0 tx=0).stop 2024-04-25T01:17:56.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.207+0000 7ff85ee00640 1 -- 172.21.15.90:0/370572583 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff85806d270 msgr2=0x7ff8580aee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:56.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.207+0000 7ff85ee00640 1 --2- 172.21.15.90:0/370572583 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff85806d270 0x7ff8580aee20 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7ff84800c460 tx=0x7ff84800c920 comp rx=0 tx=0).stop 2024-04-25T01:17:56.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.207+0000 7ff85ee00640 1 -- 172.21.15.90:0/370572583 shutdown_connections 2024-04-25T01:17:56.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.207+0000 7ff85ee00640 1 --2- 172.21.15.90:0/370572583 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff83406be30 0x7ff83406e2f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:56.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.207+0000 7ff85ee00640 1 --2- 172.21.15.90:0/370572583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff85815f9b0 0x7ff858071850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:56.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.207+0000 7ff85ee00640 1 --2- 172.21.15.90:0/370572583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff858145590 0x7ff85815b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:56.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.207+0000 7ff85ee00640 1 --2- 172.21.15.90:0/370572583 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff85806d270 0x7ff8580aee20 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:56.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.207+0000 7ff85ee00640 1 -- 172.21.15.90:0/370572583 >> 172.21.15.90:0/370572583 conn(0x7ff8580b1820 msgr2=0x7ff858140b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:56.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.207+0000 7ff85ee00640 1 -- 172.21.15.90:0/370572583 shutdown_connections 2024-04-25T01:17:56.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.207+0000 7ff85ee00640 1 -- 172.21.15.90:0/370572583 wait complete. 2024-04-25T01:17:56.211 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-04-25T01:17:56.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.426+0000 7faae5e00640 1 Processor -- start 2024-04-25T01:17:56.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.426+0000 7faae5e00640 1 -- start start 2024-04-25T01:17:56.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.428+0000 7faae5e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faae015ad40 0x7faae006b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:56.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.428+0000 7faae5e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7faae006d910 0x7faae00cd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:56.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.428+0000 7faae5e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faae006ba90 con 0x7faae00cf520 2024-04-25T01:17:56.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.428+0000 7faae5e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7faae006cb50 con 0x7faae006d910 2024-04-25T01:17:56.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.428+0000 7faae5e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7faae006ce30 con 0x7faae015ad40 2024-04-25T01:17:56.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.428+0000 7faad7e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7faae00cf520 0x7faae00ae730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:54680/0 (socket says 172.21.15.90:54680) 2024-04-25T01:17:56.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.428+0000 7faad7e00640 1 -- 172.21.15.90:0/2987481070 learned_addr learned my addr 172.21.15.90:0/2987481070 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:56.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.429+0000 7faae4a00640 1 --2- 172.21.15.90:0/2987481070 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faae015ad40 0x7faae006b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:56.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.429+0000 7faae5400640 1 --2- 172.21.15.90:0/2987481070 >> v2:172.21.15.103:3300/0 conn(0x7faae006d910 0x7faae00cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:56.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.429+0000 7faadfe00640 1 -- 172.21.15.90:0/2987481070 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2280681822 0 0) 0x7faae006ba90 con 0x7faae00cf520 2024-04-25T01:17:56.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.429+0000 7faadfe00640 1 -- 172.21.15.90:0/2987481070 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faac4003520 con 0x7faae00cf520 2024-04-25T01:17:56.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.429+0000 7faadfe00640 1 -- 172.21.15.90:0/2987481070 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4244495974 0 0) 0x7faac4003520 con 0x7faae00cf520 2024-04-25T01:17:56.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.429+0000 7faadfe00640 1 -- 172.21.15.90:0/2987481070 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faae015ad40 msgr2=0x7faae006b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:56.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.429+0000 7faadfe00640 1 --2- 172.21.15.90:0/2987481070 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faae015ad40 0x7faae006b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:56.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.429+0000 7faadfe00640 1 -- 172.21.15.90:0/2987481070 >> v2:172.21.15.103:3300/0 conn(0x7faae006d910 msgr2=0x7faae00cd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:56.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.429+0000 7faadfe00640 1 --2- 172.21.15.90:0/2987481070 >> v2:172.21.15.103:3300/0 conn(0x7faae006d910 0x7faae00cd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:56.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.429+0000 7faadfe00640 1 -- 172.21.15.90:0/2987481070 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faae006d110 con 0x7faae00cf520 2024-04-25T01:17:56.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.429+0000 7faadfe00640 1 -- 172.21.15.90:0/2987481070 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7faac8001670 con 0x7faae00cf520 2024-04-25T01:17:56.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.429+0000 7faae4a00640 1 --2- 172.21.15.90:0/2987481070 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faae015ad40 0x7faae006b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:17:56.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.429+0000 7faadfe00640 1 -- 172.21.15.90:0/2987481070 >> v1:172.21.15.90:6789/0 conn(0x7faae00cf520 legacy=0x7faae00ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:56.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.429+0000 7faadfe00640 1 --2- 172.21.15.90:0/2987481070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faac40039b0 0x7faac4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:56.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.429+0000 7faadfe00640 1 --2- 172.21.15.90:0/2987481070 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faae015ad40 0x7faac4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:56.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.429+0000 7faadfe00640 1 --2- 172.21.15.90:0/2987481070 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faac4004d40 0x7faac40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:56.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.429+0000 7faadfe00640 1 -- 172.21.15.90:0/2987481070 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7faae006cb50 con 0x7faac4004d40 2024-04-25T01:17:56.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.429+0000 7faadfe00640 1 -- 172.21.15.90:0/2987481070 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faae006ce30 con 0x7faac40039b0 2024-04-25T01:17:56.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.429+0000 7faadfe00640 1 -- 172.21.15.90:0/2987481070 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7faac400b9c0 con 0x7faae015ad40 2024-04-25T01:17:56.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.430+0000 7faae5400640 1 --2- 172.21.15.90:0/2987481070 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faac4004d40 0x7faac40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:56.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.430+0000 7faad7e00640 1 --2- 172.21.15.90:0/2987481070 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faae015ad40 0x7faac4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:56.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.430+0000 7faae4a00640 1 --2- 172.21.15.90:0/2987481070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faac40039b0 0x7faac4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:56.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.430+0000 7faae5400640 1 -- 172.21.15.90:0/2987481070 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faae015ad40 msgr2=0x7faac4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:56.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.430+0000 7faae5400640 1 --2- 172.21.15.90:0/2987481070 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faae015ad40 0x7faac4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:56.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.430+0000 7faae5400640 1 -- 172.21.15.90:0/2987481070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faac40039b0 msgr2=0x7faac4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:56.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.430+0000 7faae5400640 1 --2- 172.21.15.90:0/2987481070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faac40039b0 0x7faac4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:56.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.430+0000 7faae5400640 1 -- 172.21.15.90:0/2987481070 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faac4003520 con 0x7faac4004d40 2024-04-25T01:17:56.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.430+0000 7faae4a00640 1 --2- 172.21.15.90:0/2987481070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faac40039b0 0x7faac4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:56.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.431+0000 7faae5400640 1 --2- 172.21.15.90:0/2987481070 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faac4004d40 0x7faac40051c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7faae0070a00 tx=0x7faac4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbce400f546a4768 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:56.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.432+0000 7faadfe00640 1 -- 172.21.15.90:0/2987481070 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad8009c10 con 0x7faac4004d40 2024-04-25T01:17:56.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.432+0000 7faadfe00640 1 -- 172.21.15.90:0/2987481070 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faad8030610 con 0x7faac4004d40 2024-04-25T01:17:56.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.432+0000 7faadfe00640 1 -- 172.21.15.90:0/2987481070 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad8030a00 con 0x7faac4004d40 2024-04-25T01:17:56.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.432+0000 7faae5e00640 1 -- 172.21.15.90:0/2987481070 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faac4004d40 msgr2=0x7faac40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:56.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.432+0000 7faae5e00640 1 --2- 172.21.15.90:0/2987481070 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faac4004d40 0x7faac40051c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7faae0070a00 tx=0x7faac4005830 comp rx=0 tx=0).stop 2024-04-25T01:17:56.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.432+0000 7faae5400640 1 -- 172.21.15.90:0/2987481070 reap_dead start 2024-04-25T01:17:56.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.432+0000 7faae5e00640 1 -- 172.21.15.90:0/2987481070 shutdown_connections 2024-04-25T01:17:56.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.432+0000 7faae5e00640 1 -- 172.21.15.90:0/2987481070 >> 172.21.15.90:0/2987481070 conn(0x7faae00b1660 msgr2=0x7faae00b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:56.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.432+0000 7faae5e00640 1 -- 172.21.15.90:0/2987481070 shutdown_connections 2024-04-25T01:17:56.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.432+0000 7faae5e00640 1 -- 172.21.15.90:0/2987481070 wait complete. 2024-04-25T01:17:56.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.434+0000 7faae5e00640 1 Processor -- start 2024-04-25T01:17:56.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.434+0000 7faae5e00640 1 -- start start 2024-04-25T01:17:56.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.435+0000 7faae5e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faac4004d40 0x7faae015c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:56.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.435+0000 7faae5e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faae00cdae0 0x7faae0151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:56.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.435+0000 7faae5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faae00cdf60 0x7faae015d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:56.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.435+0000 7faae5e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7faae00aefc0 con 0x7faae00cdf60 2024-04-25T01:17:56.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.435+0000 7faae5e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faae00aee40 con 0x7faae00cdae0 2024-04-25T01:17:56.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.435+0000 7faae5e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7faae006b120 con 0x7faac4004d40 2024-04-25T01:17:56.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.435+0000 7faae5400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faae00cdf60 0x7faae015d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:56.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.436+0000 7faae5400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faae00cdf60 0x7faae015d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56992/0 (socket says 172.21.15.90:56992) 2024-04-25T01:17:56.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.436+0000 7faae5400640 1 -- 172.21.15.90:0/2013699649 learned_addr learned my addr 172.21.15.90:0/2013699649 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:56.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.436+0000 7faad7e00640 1 --2- 172.21.15.90:0/2013699649 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faae00cdae0 0x7faae0151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:56.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.436+0000 7faae4a00640 1 --2- 172.21.15.90:0/2013699649 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faac4004d40 0x7faae015c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:56.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.436+0000 7faae5400640 1 -- 172.21.15.90:0/2013699649 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faac4004d40 msgr2=0x7faae015c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:56.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.436+0000 7faae5400640 1 --2- 172.21.15.90:0/2013699649 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faac4004d40 0x7faae015c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:56.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.436+0000 7faae5400640 1 -- 172.21.15.90:0/2013699649 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faae00cdae0 msgr2=0x7faae0151a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:56.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.436+0000 7faae5400640 1 --2- 172.21.15.90:0/2013699649 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faae00cdae0 0x7faae0151a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:56.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.436+0000 7faae5400640 1 -- 172.21.15.90:0/2013699649 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faae01f7e50 con 0x7faae00cdf60 2024-04-25T01:17:56.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.436+0000 7faad7e00640 1 --2- 172.21.15.90:0/2013699649 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faae00cdae0 0x7faae0151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:56.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.436+0000 7faae4a00640 1 --2- 172.21.15.90:0/2013699649 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faac4004d40 0x7faae015c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:56.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.436+0000 7faae5400640 1 --2- 172.21.15.90:0/2013699649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faae00cdf60 0x7faae015d660 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7faad802d090 tx=0x7faad8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:56.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.436+0000 7faadd600640 1 -- 172.21.15.90:0/2013699649 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad8009c10 con 0x7faae00cdf60 2024-04-25T01:17:56.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.436+0000 7faadd600640 1 -- 172.21.15.90:0/2013699649 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faad8030410 con 0x7faae00cdf60 2024-04-25T01:17:56.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.436+0000 7faadd600640 1 -- 172.21.15.90:0/2013699649 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad8003bb0 con 0x7faae00cdf60 2024-04-25T01:17:56.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.437+0000 7faae5e00640 1 -- 172.21.15.90:0/2013699649 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faae01f8130 con 0x7faae00cdf60 2024-04-25T01:17:56.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.437+0000 7faae5e00640 1 -- 172.21.15.90:0/2013699649 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faae01f86f0 con 0x7faae00cdf60 2024-04-25T01:17:56.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.438+0000 7faae5e00640 1 -- 172.21.15.90:0/2013699649 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faae00692b0 con 0x7faae00cdf60 2024-04-25T01:17:56.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.439+0000 7faadd600640 1 -- 172.21.15.90:0/2013699649 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7faad80388c0 con 0x7faae00cdf60 2024-04-25T01:17:56.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.440+0000 7faadd600640 1 --2- 172.21.15.90:0/2013699649 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7faabc06be30 0x7faabc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:56.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.440+0000 7faadd600640 1 -- 172.21.15.90:0/2013699649 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5249+0+0 (secure 0 0 0) 0x7faad80ab550 con 0x7faae00cdf60 2024-04-25T01:17:56.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.440+0000 7faae4a00640 1 --2- 172.21.15.90:0/2013699649 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7faabc06be30 0x7faabc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:56.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.441+0000 7faae4a00640 1 --2- 172.21.15.90:0/2013699649 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7faabc06be30 0x7faabc06e2f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7faad000b9e0 tx=0x7faad0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:56.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.444+0000 7faadd600640 1 -- 172.21.15.90:0/2013699649 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7faae00692b0 con 0x7faae00cdf60 2024-04-25T01:17:56.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.716+0000 7faae5e00640 1 -- 172.21.15.90:0/2013699649 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7faae0075350 con 0x7faae00cdf60 2024-04-25T01:17:56.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.721+0000 7faadd600640 1 -- 172.21.15.90:0/2013699649 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7faad8010070 con 0x7faae00cdf60 2024-04-25T01:17:56.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.723+0000 7faae5e00640 1 -- 172.21.15.90:0/2013699649 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7faabc06be30 msgr2=0x7faabc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:56.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.723+0000 7faae5e00640 1 --2- 172.21.15.90:0/2013699649 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7faabc06be30 0x7faabc06e2f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7faad000b9e0 tx=0x7faad0008040 comp rx=0 tx=0).stop 2024-04-25T01:17:56.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.723+0000 7faae5e00640 1 -- 172.21.15.90:0/2013699649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faae00cdf60 msgr2=0x7faae015d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:56.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.723+0000 7faae5e00640 1 --2- 172.21.15.90:0/2013699649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faae00cdf60 0x7faae015d660 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7faad802d090 tx=0x7faad8009750 comp rx=0 tx=0).stop 2024-04-25T01:17:56.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.723+0000 7faae5e00640 1 -- 172.21.15.90:0/2013699649 shutdown_connections 2024-04-25T01:17:56.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.724+0000 7faae5e00640 1 --2- 172.21.15.90:0/2013699649 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7faabc06be30 0x7faabc06e2f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:56.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.724+0000 7faae5e00640 1 --2- 172.21.15.90:0/2013699649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faae00cdf60 0x7faae015d660 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:56.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.724+0000 7faae5e00640 1 --2- 172.21.15.90:0/2013699649 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faae00cdae0 0x7faae0151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:56.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.724+0000 7faae5e00640 1 --2- 172.21.15.90:0/2013699649 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7faac4004d40 0x7faae015c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:56.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.724+0000 7faae5e00640 1 -- 172.21.15.90:0/2013699649 >> 172.21.15.90:0/2013699649 conn(0x7faae00b1660 msgr2=0x7faae00ae730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:56.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.724+0000 7faae5e00640 1 -- 172.21.15.90:0/2013699649 shutdown_connections 2024-04-25T01:17:56.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.724+0000 7faae5e00640 1 -- 172.21.15.90:0/2013699649 wait complete. 2024-04-25T01:17:56.725 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-04-25T01:17:56.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.940+0000 7ffa07e00640 1 Processor -- start 2024-04-25T01:17:56.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.940+0000 7ffa07e00640 1 -- start start 2024-04-25T01:17:56.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.942+0000 7ffa07e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ffa080ae9d0 0x7ffa0815ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:56.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.942+0000 7ffa07e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa080adc10 con 0x7ffa080b41f0 2024-04-25T01:17:56.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.942+0000 7ffa07e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa080adef0 con 0x7ffa0806b830 2024-04-25T01:17:56.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.942+0000 7ffa07e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ffa080ae1d0 con 0x7ffa080ae9d0 2024-04-25T01:17:56.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.942+0000 7ffa06a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ffa080b41f0 0x7ffa0806b120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:54684/0 (socket says 172.21.15.90:54684) 2024-04-25T01:17:56.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.942+0000 7ffa06a00640 1 -- 172.21.15.90:0/3456879970 learned_addr learned my addr 172.21.15.90:0/3456879970 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:56.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.943+0000 7ffa07400640 1 --2- 172.21.15.90:0/3456879970 >> v2:172.21.15.103:3300/0 conn(0x7ffa080ae9d0 0x7ffa0815ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:56.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.943+0000 7ffa06000640 1 -- 172.21.15.90:0/3456879970 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1428467812 0 0) 0x7ffa080adc10 con 0x7ffa080b41f0 2024-04-25T01:17:56.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.943+0000 7ffa06000640 1 -- 172.21.15.90:0/3456879970 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9ec003520 con 0x7ffa080b41f0 2024-04-25T01:17:56.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.943+0000 7ffa06000640 1 -- 172.21.15.90:0/3456879970 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 618107898 0 0) 0x7ff9ec003520 con 0x7ffa080b41f0 2024-04-25T01:17:56.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.943+0000 7ffa06000640 1 -- 172.21.15.90:0/3456879970 >> v2:172.21.15.103:3300/0 conn(0x7ffa080ae9d0 msgr2=0x7ffa0815ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:56.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.943+0000 7ffa06000640 1 --2- 172.21.15.90:0/3456879970 >> v2:172.21.15.103:3300/0 conn(0x7ffa080ae9d0 0x7ffa0815ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:56.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.943+0000 7ffa06000640 1 -- 172.21.15.90:0/3456879970 >> v1:172.21.15.103:6789/0 conn(0x7ffa0806b830 legacy=0x7ffa0815e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:56.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.943+0000 7ffa06000640 1 -- 172.21.15.90:0/3456879970 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa080ae4b0 con 0x7ffa080b41f0 2024-04-25T01:17:56.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.943+0000 7ffa06000640 1 -- 172.21.15.90:0/3456879970 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7ff9f8001670 con 0x7ffa080b41f0 2024-04-25T01:17:56.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.943+0000 7ffa06000640 1 -- 172.21.15.90:0/3456879970 >> v1:172.21.15.90:6789/0 conn(0x7ffa080b41f0 legacy=0x7ffa0806b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:56.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.943+0000 7ffa06000640 1 --2- 172.21.15.90:0/3456879970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9ec0039b0 0x7ff9ec003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:56.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.943+0000 7ffa06000640 1 --2- 172.21.15.90:0/3456879970 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9ec004800 0x7ff9ec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:56.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.943+0000 7ffa06000640 1 --2- 172.21.15.90:0/3456879970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9ec0051c0 0x7ff9ec00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:56.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.943+0000 7ffa06000640 1 -- 172.21.15.90:0/3456879970 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ffa080ae1d0 con 0x7ff9ec0039b0 2024-04-25T01:17:56.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.943+0000 7ffa06000640 1 -- 172.21.15.90:0/3456879970 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff9ec00c060 con 0x7ff9ec0051c0 2024-04-25T01:17:56.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.943+0000 7ffa06000640 1 -- 172.21.15.90:0/3456879970 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff9ec00c260 con 0x7ff9ec004800 2024-04-25T01:17:56.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.944+0000 7ffa06a00640 1 --2- 172.21.15.90:0/3456879970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9ec0039b0 0x7ff9ec003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:56.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.944+0000 7ffa07400640 1 --2- 172.21.15.90:0/3456879970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9ec0051c0 0x7ff9ec00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:56.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.944+0000 7ff9fe000640 1 --2- 172.21.15.90:0/3456879970 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9ec004800 0x7ff9ec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:56.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.944+0000 7ffa06a00640 1 -- 172.21.15.90:0/3456879970 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9ec004800 msgr2=0x7ff9ec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:56.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.944+0000 7ffa06a00640 1 --2- 172.21.15.90:0/3456879970 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff9ec004800 0x7ff9ec004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:56.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.944+0000 7ffa06a00640 1 -- 172.21.15.90:0/3456879970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9ec0051c0 msgr2=0x7ff9ec00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:56.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.944+0000 7ffa06a00640 1 --2- 172.21.15.90:0/3456879970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9ec0051c0 0x7ff9ec00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:56.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.944+0000 7ffa06a00640 1 -- 172.21.15.90:0/3456879970 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9ec003520 con 0x7ff9ec0039b0 2024-04-25T01:17:56.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.944+0000 7ffa07400640 1 --2- 172.21.15.90:0/3456879970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9ec0051c0 0x7ff9ec00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:56.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.944+0000 7ffa07400640 1 -- 172.21.15.90:0/3456879970 reap_dead start 2024-04-25T01:17:56.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.945+0000 7ffa06a00640 1 --2- 172.21.15.90:0/3456879970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9ec0039b0 0x7ff9ec003dd0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff9f800c240 tx=0x7ff9ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47f2d47c8eee71e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:56.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.945+0000 7ffa06000640 1 -- 172.21.15.90:0/3456879970 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f8007d10 con 0x7ff9ec0039b0 2024-04-25T01:17:56.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.945+0000 7ffa06000640 1 -- 172.21.15.90:0/3456879970 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9f8005ad0 con 0x7ff9ec0039b0 2024-04-25T01:17:56.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.945+0000 7ffa06000640 1 -- 172.21.15.90:0/3456879970 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f8007070 con 0x7ff9ec0039b0 2024-04-25T01:17:56.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.946+0000 7ffa07e00640 1 -- 172.21.15.90:0/3456879970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9ec0039b0 msgr2=0x7ff9ec003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:56.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.946+0000 7ffa07e00640 1 --2- 172.21.15.90:0/3456879970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9ec0039b0 0x7ff9ec003dd0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff9f800c240 tx=0x7ff9ec005830 comp rx=0 tx=0).stop 2024-04-25T01:17:56.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.946+0000 7ffa07e00640 1 -- 172.21.15.90:0/3456879970 shutdown_connections 2024-04-25T01:17:56.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.946+0000 7ffa07e00640 1 --2- 172.21.15.90:0/3456879970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9ec0039b0 0x7ff9ec003dd0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:56.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.946+0000 7ffa07e00640 1 -- 172.21.15.90:0/3456879970 >> 172.21.15.90:0/3456879970 conn(0x7ffa080b1660 msgr2=0x7ffa080b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:56.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.946+0000 7ffa07e00640 1 -- 172.21.15.90:0/3456879970 shutdown_connections 2024-04-25T01:17:56.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.946+0000 7ffa07e00640 1 -- 172.21.15.90:0/3456879970 wait complete. 2024-04-25T01:17:56.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.948+0000 7ffa07e00640 1 Processor -- start 2024-04-25T01:17:56.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.948+0000 7ffa07e00640 1 -- start start 2024-04-25T01:17:56.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.949+0000 7ffa07e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9ec0039b0 0x7ffa08144e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:56.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.949+0000 7ffa07e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa08072550 0x7ffa08145390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:56.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.949+0000 7ffa07e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa08147d80 0x7ffa08148200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:56.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.949+0000 7ffa07e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0806b4b0 con 0x7ffa08072550 2024-04-25T01:17:56.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.949+0000 7ffa07e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0806b330 con 0x7ff9ec0039b0 2024-04-25T01:17:56.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.949+0000 7ffa07e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ffa0806b630 con 0x7ffa08147d80 2024-04-25T01:17:56.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.949+0000 7ff9fe000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa08072550 0x7ffa08145390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:56.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.949+0000 7ff9fe000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa08072550 0x7ffa08145390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57002/0 (socket says 172.21.15.90:57002) 2024-04-25T01:17:56.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.949+0000 7ff9fe000640 1 -- 172.21.15.90:0/3170897525 learned_addr learned my addr 172.21.15.90:0/3170897525 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:56.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.949+0000 7ffa06a00640 1 --2- 172.21.15.90:0/3170897525 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9ec0039b0 0x7ffa08144e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:56.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.949+0000 7ff9fe000640 1 -- 172.21.15.90:0/3170897525 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa08147d80 msgr2=0x7ffa08148200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:56.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.949+0000 7ff9fe000640 1 --2- 172.21.15.90:0/3170897525 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa08147d80 0x7ffa08148200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:56.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.949+0000 7ff9fe000640 1 -- 172.21.15.90:0/3170897525 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9ec0039b0 msgr2=0x7ffa08144e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:56.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.949+0000 7ff9fe000640 1 --2- 172.21.15.90:0/3170897525 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9ec0039b0 0x7ffa08144e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:56.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.949+0000 7ff9fe000640 1 -- 172.21.15.90:0/3170897525 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa081600e0 con 0x7ffa08072550 2024-04-25T01:17:56.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.950+0000 7ff9fe000640 1 --2- 172.21.15.90:0/3170897525 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa08072550 0x7ffa08145390 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff9f000d5a0 tx=0x7ff9f000da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:56.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.950+0000 7ff9ff400640 1 -- 172.21.15.90:0/3170897525 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f0003610 con 0x7ffa08072550 2024-04-25T01:17:56.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.950+0000 7ff9ff400640 1 -- 172.21.15.90:0/3170897525 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9f0017070 con 0x7ffa08072550 2024-04-25T01:17:56.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.950+0000 7ff9ff400640 1 -- 172.21.15.90:0/3170897525 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f00039d0 con 0x7ffa08072550 2024-04-25T01:17:56.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.950+0000 7ffa07e00640 1 -- 172.21.15.90:0/3170897525 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa081603c0 con 0x7ffa08072550 2024-04-25T01:17:56.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.950+0000 7ffa07e00640 1 -- 172.21.15.90:0/3170897525 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa08160980 con 0x7ffa08072550 2024-04-25T01:17:56.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.951+0000 7ffa07e00640 1 -- 172.21.15.90:0/3170897525 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa0815e4b0 con 0x7ffa08072550 2024-04-25T01:17:56.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.952+0000 7ff9ff400640 1 -- 172.21.15.90:0/3170897525 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff9f0003b70 con 0x7ffa08072550 2024-04-25T01:17:56.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.953+0000 7ff9ff400640 1 --2- 172.21.15.90:0/3170897525 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff9e406be60 0x7ff9e406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:56.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.953+0000 7ff9ff400640 1 -- 172.21.15.90:0/3170897525 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5249+0+0 (secure 0 0 0) 0x7ff9f008aa60 con 0x7ffa08072550 2024-04-25T01:17:56.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.953+0000 7ffa06a00640 1 --2- 172.21.15.90:0/3170897525 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff9e406be60 0x7ff9e406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:56.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.954+0000 7ffa06a00640 1 --2- 172.21.15.90:0/3170897525 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff9e406be60 0x7ff9e406e320 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8032000 tx=0x7ff9f802d000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:56.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:56.957+0000 7ff9ff400640 1 -- 172.21.15.90:0/3170897525 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ffa0815e4b0 con 0x7ffa08072550 2024-04-25T01:17:57.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.224+0000 7ffa07e00640 1 -- 172.21.15.90:0/3170897525 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7ffa0815fe20 con 0x7ffa08072550 2024-04-25T01:17:57.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.229+0000 7ff9ff400640 1 -- 172.21.15.90:0/3170897525 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v9) ==== 90+0+0 (secure 0 0 0) 0x7ff9f0059680 con 0x7ffa08072550 2024-04-25T01:17:57.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.232+0000 7ffa07e00640 1 -- 172.21.15.90:0/3170897525 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff9e406be60 msgr2=0x7ff9e406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:57.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.232+0000 7ffa07e00640 1 --2- 172.21.15.90:0/3170897525 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff9e406be60 0x7ff9e406e320 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8032000 tx=0x7ff9f802d000 comp rx=0 tx=0).stop 2024-04-25T01:17:57.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.232+0000 7ffa07e00640 1 -- 172.21.15.90:0/3170897525 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa08072550 msgr2=0x7ffa08145390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:57.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.232+0000 7ffa07e00640 1 --2- 172.21.15.90:0/3170897525 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa08072550 0x7ffa08145390 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff9f000d5a0 tx=0x7ff9f000da60 comp rx=0 tx=0).stop 2024-04-25T01:17:57.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.232+0000 7ffa07e00640 1 -- 172.21.15.90:0/3170897525 shutdown_connections 2024-04-25T01:17:57.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.232+0000 7ffa07e00640 1 --2- 172.21.15.90:0/3170897525 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff9e406be60 0x7ff9e406e320 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:57.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.232+0000 7ffa07e00640 1 --2- 172.21.15.90:0/3170897525 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ffa08147d80 0x7ffa08148200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:57.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.232+0000 7ffa07e00640 1 --2- 172.21.15.90:0/3170897525 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa08072550 0x7ffa08145390 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:57.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.232+0000 7ffa07e00640 1 --2- 172.21.15.90:0/3170897525 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9ec0039b0 0x7ffa08144e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:57.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.232+0000 7ffa07e00640 1 -- 172.21.15.90:0/3170897525 >> 172.21.15.90:0/3170897525 conn(0x7ffa080b1660 msgr2=0x7ffa080af690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:57.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.232+0000 7ffa07e00640 1 -- 172.21.15.90:0/3170897525 shutdown_connections 2024-04-25T01:17:57.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.233+0000 7ffa07e00640 1 -- 172.21.15.90:0/3170897525 wait complete. 2024-04-25T01:17:57.235 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-04-25T01:17:57.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.449+0000 7fcd9ee00640 1 Processor -- start 2024-04-25T01:17:57.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.449+0000 7fcd9ee00640 1 -- start start 2024-04-25T01:17:57.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.450+0000 7fcd9ee00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcd9806b120 0x7fcd9806b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:57.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.451+0000 7fcd9ee00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcd9815ca10 0x7fcd9815ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:57.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.451+0000 7fcd9ee00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd9806ba40 con 0x7fcd980ae730 2024-04-25T01:17:57.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.451+0000 7fcd9ee00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fcd980cef40 con 0x7fcd9815ca10 2024-04-25T01:17:57.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.451+0000 7fcd9ee00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fcd9815c580 con 0x7fcd9806b120 2024-04-25T01:17:57.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.451+0000 7fcd9e400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcd9815ca10 0x7fcd9815ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:57.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.451+0000 7fcd9d000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fcd980ae730 0x7fcd980cd900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:54690/0 (socket says 172.21.15.90:54690) 2024-04-25T01:17:57.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.451+0000 7fcd9d000640 1 -- 172.21.15.90:0/199590367 learned_addr learned my addr 172.21.15.90:0/199590367 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:57.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.451+0000 7fcd9da00640 1 --2- 172.21.15.90:0/199590367 >> v2:172.21.15.103:3300/0 conn(0x7fcd9806b120 0x7fcd9806b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:57.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.451+0000 7fcd9e400640 1 -- 172.21.15.90:0/199590367 >> v2:172.21.15.103:3300/0 conn(0x7fcd9806b120 msgr2=0x7fcd9806b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:57.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.451+0000 7fcd9e400640 1 --2- 172.21.15.90:0/199590367 >> v2:172.21.15.103:3300/0 conn(0x7fcd9806b120 0x7fcd9806b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:57.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.451+0000 7fcd9e400640 1 -- 172.21.15.90:0/199590367 >> v1:172.21.15.90:6789/0 conn(0x7fcd980ae730 legacy=0x7fcd980cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:57.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.451+0000 7fcd97e00640 1 -- 172.21.15.90:0/199590367 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3868421484 0 0) 0x7fcd9806ba40 con 0x7fcd980ae730 2024-04-25T01:17:57.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.452+0000 7fcd9e400640 1 -- 172.21.15.90:0/199590367 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd9815c220 con 0x7fcd9815ca10 2024-04-25T01:17:57.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.453+0000 7fcd9e400640 1 --2- 172.21.15.90:0/199590367 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd9815ca10 0x7fcd9815ce80 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fcd980b2560 tx=0x7fcd9002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b80df2f4576e4a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:57.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.453+0000 7fcd97e00640 1 -- 172.21.15.90:0/199590367 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd90007740 con 0x7fcd9815ca10 2024-04-25T01:17:57.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.453+0000 7fcd97e00640 1 -- 172.21.15.90:0/199590367 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd900078e0 con 0x7fcd9815ca10 2024-04-25T01:17:57.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.453+0000 7fcd97e00640 1 -- 172.21.15.90:0/199590367 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd90007cb0 con 0x7fcd9815ca10 2024-04-25T01:17:57.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.453+0000 7fcd9ee00640 1 -- 172.21.15.90:0/199590367 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd9815ca10 msgr2=0x7fcd9815ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:57.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.453+0000 7fcd9ee00640 1 --2- 172.21.15.90:0/199590367 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd9815ca10 0x7fcd9815ce80 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fcd980b2560 tx=0x7fcd9002a680 comp rx=0 tx=0).stop 2024-04-25T01:17:57.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.454+0000 7fcd9ee00640 1 -- 172.21.15.90:0/199590367 shutdown_connections 2024-04-25T01:17:57.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.454+0000 7fcd9ee00640 1 --2- 172.21.15.90:0/199590367 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd9815ca10 0x7fcd9815ce80 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:57.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.454+0000 7fcd9ee00640 1 --2- 172.21.15.90:0/199590367 >> v2:172.21.15.103:3300/0 conn(0x7fcd9806b120 0x7fcd9806b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:57.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.454+0000 7fcd9ee00640 1 -- 172.21.15.90:0/199590367 >> 172.21.15.90:0/199590367 conn(0x7fcd980b1660 msgr2=0x7fcd980b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:57.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.454+0000 7fcd9ee00640 1 -- 172.21.15.90:0/199590367 shutdown_connections 2024-04-25T01:17:57.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.454+0000 7fcd9ee00640 1 -- 172.21.15.90:0/199590367 wait complete. 2024-04-25T01:17:57.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.456+0000 7fcd9ee00640 1 Processor -- start 2024-04-25T01:17:57.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.456+0000 7fcd9ee00640 1 -- start start 2024-04-25T01:17:57.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.457+0000 7fcd9ee00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd9806b120 0x7fcd9814b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:57.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.457+0000 7fcd9ee00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd980ae730 0x7fcd98158fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:57.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.457+0000 7fcd9ee00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd9815ca10 0x7fcd981594e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:57.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.457+0000 7fcd9ee00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcd980af240 con 0x7fcd980ae730 2024-04-25T01:17:57.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.457+0000 7fcd9ee00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcd980aef60 con 0x7fcd9806b120 2024-04-25T01:17:57.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.457+0000 7fcd9ee00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcd98153080 con 0x7fcd9815ca10 2024-04-25T01:17:57.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.458+0000 7fcd9d000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd980ae730 0x7fcd98158fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:57.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.458+0000 7fcd9d000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd980ae730 0x7fcd98158fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57012/0 (socket says 172.21.15.90:57012) 2024-04-25T01:17:57.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.458+0000 7fcd9d000640 1 -- 172.21.15.90:0/1591695004 learned_addr learned my addr 172.21.15.90:0/1591695004 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:57.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.458+0000 7fcd9da00640 1 --2- 172.21.15.90:0/1591695004 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd9806b120 0x7fcd9814b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:57.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.458+0000 7fcd9d000640 1 -- 172.21.15.90:0/1591695004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd9815ca10 msgr2=0x7fcd981594e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:57.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.458+0000 7fcd9d000640 1 --2- 172.21.15.90:0/1591695004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd9815ca10 0x7fcd981594e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:57.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.458+0000 7fcd9d000640 1 -- 172.21.15.90:0/1591695004 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd9806b120 msgr2=0x7fcd9814b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:57.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.458+0000 7fcd9d000640 1 --2- 172.21.15.90:0/1591695004 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd9806b120 0x7fcd9814b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:57.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.458+0000 7fcd9d000640 1 -- 172.21.15.90:0/1591695004 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd98153eb0 con 0x7fcd980ae730 2024-04-25T01:17:57.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.458+0000 7fcd9da00640 1 --2- 172.21.15.90:0/1591695004 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd9806b120 0x7fcd9814b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:57.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.458+0000 7fcd9d000640 1 --2- 172.21.15.90:0/1591695004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd980ae730 0x7fcd98158fa0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fcd8400c4d0 tx=0x7fcd8400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:57.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.458+0000 7fcd95600640 1 -- 172.21.15.90:0/1591695004 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd84018070 con 0x7fcd980ae730 2024-04-25T01:17:57.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.459+0000 7fcd95600640 1 -- 172.21.15.90:0/1591695004 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd84007500 con 0x7fcd980ae730 2024-04-25T01:17:57.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.459+0000 7fcd95600640 1 -- 172.21.15.90:0/1591695004 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd84004a20 con 0x7fcd980ae730 2024-04-25T01:17:57.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.459+0000 7fcd9ee00640 1 -- 172.21.15.90:0/1591695004 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd98154190 con 0x7fcd980ae730 2024-04-25T01:17:57.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.459+0000 7fcd9ee00640 1 -- 172.21.15.90:0/1591695004 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd98154750 con 0x7fcd980ae730 2024-04-25T01:17:57.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.460+0000 7fcd9ee00640 1 -- 172.21.15.90:0/1591695004 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd98075350 con 0x7fcd980ae730 2024-04-25T01:17:57.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.461+0000 7fcd95600640 1 -- 172.21.15.90:0/1591695004 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fcd84004bc0 con 0x7fcd980ae730 2024-04-25T01:17:57.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.462+0000 7fcd95600640 1 --2- 172.21.15.90:0/1591695004 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fcd7806be30 0x7fcd7806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:57.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.462+0000 7fcd95600640 1 -- 172.21.15.90:0/1591695004 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5249+0+0 (secure 0 0 0) 0x7fcd84089f80 con 0x7fcd980ae730 2024-04-25T01:17:57.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.462+0000 7fcd9da00640 1 --2- 172.21.15.90:0/1591695004 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fcd7806be30 0x7fcd7806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:57.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.463+0000 7fcd9da00640 1 --2- 172.21.15.90:0/1591695004 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fcd7806be30 0x7fcd7806e2f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c006fd0 tx=0x7fcd8c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:57.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.466+0000 7fcd95600640 1 -- 172.21.15.90:0/1591695004 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fcd84058ae0 con 0x7fcd980ae730 2024-04-25T01:17:57.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.719+0000 7fcd9ee00640 1 -- 172.21.15.90:0/1591695004 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7fcd98152a80 con 0x7fcd980ae730 2024-04-25T01:17:57.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.723+0000 7fcd95600640 1 -- 172.21.15.90:0/1591695004 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v10) ==== 90+0+0 (secure 0 0 0) 0x7fcd840583e0 con 0x7fcd980ae730 2024-04-25T01:17:57.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.726+0000 7fcd9ee00640 1 -- 172.21.15.90:0/1591695004 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fcd7806be30 msgr2=0x7fcd7806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:57.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.726+0000 7fcd9ee00640 1 --2- 172.21.15.90:0/1591695004 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fcd7806be30 0x7fcd7806e2f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c006fd0 tx=0x7fcd8c008040 comp rx=0 tx=0).stop 2024-04-25T01:17:57.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.726+0000 7fcd9ee00640 1 -- 172.21.15.90:0/1591695004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd980ae730 msgr2=0x7fcd98158fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:57.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.726+0000 7fcd9ee00640 1 --2- 172.21.15.90:0/1591695004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd980ae730 0x7fcd98158fa0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fcd8400c4d0 tx=0x7fcd8400c990 comp rx=0 tx=0).stop 2024-04-25T01:17:57.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.726+0000 7fcd9ee00640 1 -- 172.21.15.90:0/1591695004 shutdown_connections 2024-04-25T01:17:57.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.726+0000 7fcd9ee00640 1 --2- 172.21.15.90:0/1591695004 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fcd7806be30 0x7fcd7806e2f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:57.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.726+0000 7fcd9ee00640 1 --2- 172.21.15.90:0/1591695004 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcd9815ca10 0x7fcd981594e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:57.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.726+0000 7fcd9ee00640 1 --2- 172.21.15.90:0/1591695004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd980ae730 0x7fcd98158fa0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:57.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.726+0000 7fcd9ee00640 1 --2- 172.21.15.90:0/1591695004 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd9806b120 0x7fcd9814b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:57.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.726+0000 7fcd9ee00640 1 -- 172.21.15.90:0/1591695004 >> 172.21.15.90:0/1591695004 conn(0x7fcd980b1660 msgr2=0x7fcd980b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:57.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.726+0000 7fcd9ee00640 1 -- 172.21.15.90:0/1591695004 shutdown_connections 2024-04-25T01:17:57.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.726+0000 7fcd9ee00640 1 -- 172.21.15.90:0/1591695004 wait complete. 2024-04-25T01:17:57.727 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-04-25T01:17:57.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.942+0000 7fb668400640 1 Processor -- start 2024-04-25T01:17:57.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.942+0000 7fb668400640 1 -- start start 2024-04-25T01:17:57.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.943+0000 7fb668400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb66015f9b0 0x7fb66015fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:57.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.943+0000 7fb668400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb66006d270 0x7fb66006d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:57.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.943+0000 7fb668400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6600b4240 con 0x7fb660145590 2024-04-25T01:17:57.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.943+0000 7fb668400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fb660160830 con 0x7fb66015f9b0 2024-04-25T01:17:57.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.944+0000 7fb668400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb660160350 con 0x7fb66006d270 2024-04-25T01:17:57.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.944+0000 7fb665a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fb660145590 0x7fb6600b4a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:54702/0 (socket says 172.21.15.90:54702) 2024-04-25T01:17:57.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.944+0000 7fb665a00640 1 -- 172.21.15.90:0/2362784743 learned_addr learned my addr 172.21.15.90:0/2362784743 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:57.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.944+0000 7fb665000640 1 --2- 172.21.15.90:0/2362784743 >> v2:172.21.15.90:3300/0 conn(0x7fb66015f9b0 0x7fb66015fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:57.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.944+0000 7fb65fe00640 1 -- 172.21.15.90:0/2362784743 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1393469324 0 0) 0x7fb6600b4240 con 0x7fb660145590 2024-04-25T01:17:57.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.944+0000 7fb65fe00640 1 -- 172.21.15.90:0/2362784743 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb648003520 con 0x7fb660145590 2024-04-25T01:17:57.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.944+0000 7fb666400640 1 --2- 172.21.15.90:0/2362784743 >> v2:172.21.15.103:3300/0 conn(0x7fb66006d270 0x7fb66006d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:57.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.944+0000 7fb665000640 1 -- 172.21.15.90:0/2362784743 >> v2:172.21.15.103:3300/0 conn(0x7fb66006d270 msgr2=0x7fb66006d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:57.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.944+0000 7fb665000640 1 --2- 172.21.15.90:0/2362784743 >> v2:172.21.15.103:3300/0 conn(0x7fb66006d270 0x7fb66006d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:57.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.944+0000 7fb665000640 1 -- 172.21.15.90:0/2362784743 >> v1:172.21.15.90:6789/0 conn(0x7fb660145590 legacy=0x7fb6600b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:57.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.944+0000 7fb665000640 1 -- 172.21.15.90:0/2362784743 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6600b39a0 con 0x7fb66015f9b0 2024-04-25T01:17:57.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.946+0000 7fb665000640 1 --2- 172.21.15.90:0/2362784743 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb66015f9b0 0x7fb66015fe10 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fb660068b60 tx=0x7fb64c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7598821388a75501 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:57.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.946+0000 7fb65fe00640 1 -- 172.21.15.90:0/2362784743 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb64c037070 con 0x7fb66015f9b0 2024-04-25T01:17:57.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.946+0000 7fb65fe00640 1 -- 172.21.15.90:0/2362784743 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb64c032440 con 0x7fb66015f9b0 2024-04-25T01:17:57.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.946+0000 7fb65fe00640 1 -- 172.21.15.90:0/2362784743 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb64c032810 con 0x7fb66015f9b0 2024-04-25T01:17:57.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.946+0000 7fb668400640 1 -- 172.21.15.90:0/2362784743 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb66015f9b0 msgr2=0x7fb66015fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:57.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.946+0000 7fb668400640 1 --2- 172.21.15.90:0/2362784743 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb66015f9b0 0x7fb66015fe10 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fb660068b60 tx=0x7fb64c02a1a0 comp rx=0 tx=0).stop 2024-04-25T01:17:57.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.946+0000 7fb668400640 1 -- 172.21.15.90:0/2362784743 shutdown_connections 2024-04-25T01:17:57.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.946+0000 7fb668400640 1 --2- 172.21.15.90:0/2362784743 >> v2:172.21.15.103:3300/0 conn(0x7fb66006d270 0x7fb66006d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:57.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.946+0000 7fb668400640 1 --2- 172.21.15.90:0/2362784743 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb66015f9b0 0x7fb66015fe10 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:57.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.946+0000 7fb668400640 1 -- 172.21.15.90:0/2362784743 >> 172.21.15.90:0/2362784743 conn(0x7fb6600b1820 msgr2=0x7fb6600b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:57.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.947+0000 7fb668400640 1 -- 172.21.15.90:0/2362784743 shutdown_connections 2024-04-25T01:17:57.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.947+0000 7fb668400640 1 -- 172.21.15.90:0/2362784743 wait complete. 2024-04-25T01:17:57.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.948+0000 7fb668400640 1 Processor -- start 2024-04-25T01:17:57.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.948+0000 7fb668400640 1 -- start start 2024-04-25T01:17:57.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.949+0000 7fb668400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb66006d270 0x7fb6601516e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:57.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.949+0000 7fb668400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb660145590 0x7fb66013a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:57.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.949+0000 7fb668400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb66015f9b0 0x7fb66013ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:57.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.949+0000 7fb668400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb6600b3f60 con 0x7fb660145590 2024-04-25T01:17:57.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.949+0000 7fb668400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb66006cde0 con 0x7fb66015f9b0 2024-04-25T01:17:57.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.949+0000 7fb668400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb660145970 con 0x7fb66006d270 2024-04-25T01:17:57.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.949+0000 7fb665000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb660145590 0x7fb66013a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:57.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.949+0000 7fb665000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb660145590 0x7fb66013a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57028/0 (socket says 172.21.15.90:57028) 2024-04-25T01:17:57.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.949+0000 7fb665000640 1 -- 172.21.15.90:0/3865599122 learned_addr learned my addr 172.21.15.90:0/3865599122 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:57.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.950+0000 7fb665000640 1 -- 172.21.15.90:0/3865599122 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb66006d270 msgr2=0x7fb6601516e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:57.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.950+0000 7fb665000640 1 --2- 172.21.15.90:0/3865599122 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb66006d270 0x7fb6601516e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:57.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.950+0000 7fb665000640 1 -- 172.21.15.90:0/3865599122 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb66015f9b0 msgr2=0x7fb66013ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:57.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.950+0000 7fb665000640 1 --2- 172.21.15.90:0/3865599122 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb66015f9b0 0x7fb66013ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:57.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.950+0000 7fb665000640 1 -- 172.21.15.90:0/3865599122 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb66013ad80 con 0x7fb660145590 2024-04-25T01:17:57.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.950+0000 7fb665000640 1 --2- 172.21.15.90:0/3865599122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb660145590 0x7fb66013a480 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fb64c000c00 tx=0x7fb64c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:57.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.950+0000 7fb65d600640 1 -- 172.21.15.90:0/3865599122 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb64c037040 con 0x7fb660145590 2024-04-25T01:17:57.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.950+0000 7fb65d600640 1 -- 172.21.15.90:0/3865599122 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb64c042070 con 0x7fb660145590 2024-04-25T01:17:57.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.950+0000 7fb668400640 1 -- 172.21.15.90:0/3865599122 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb66013b060 con 0x7fb660145590 2024-04-25T01:17:57.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.950+0000 7fb65d600640 1 -- 172.21.15.90:0/3865599122 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb64c030960 con 0x7fb660145590 2024-04-25T01:17:57.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.951+0000 7fb668400640 1 -- 172.21.15.90:0/3865599122 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb66013b620 con 0x7fb660145590 2024-04-25T01:17:57.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.951+0000 7fb668400640 1 -- 172.21.15.90:0/3865599122 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb660068fb0 con 0x7fb660145590 2024-04-25T01:17:57.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.953+0000 7fb65d600640 1 -- 172.21.15.90:0/3865599122 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb64c030b00 con 0x7fb660145590 2024-04-25T01:17:57.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.953+0000 7fb65d600640 1 --2- 172.21.15.90:0/3865599122 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb64006be30 0x7fb64006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:57.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.953+0000 7fb65d600640 1 -- 172.21.15.90:0/3865599122 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5249+0+0 (secure 0 0 0) 0x7fb64c0ab660 con 0x7fb660145590 2024-04-25T01:17:57.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.954+0000 7fb665a00640 1 --2- 172.21.15.90:0/3865599122 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb64006be30 0x7fb64006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:57.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.954+0000 7fb665a00640 1 --2- 172.21.15.90:0/3865599122 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb64006be30 0x7fb64006e2f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb654000f60 tx=0x7fb6540078f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:57.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:57.958+0000 7fb65d600640 1 -- 172.21.15.90:0/3865599122 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb64c045360 con 0x7fb660145590 2024-04-25T01:17:58.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.222+0000 7fb668400640 1 -- 172.21.15.90:0/3865599122 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7fb66006ea50 con 0x7fb660145590 2024-04-25T01:17:58.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.227+0000 7fb65d600640 1 -- 172.21.15.90:0/3865599122 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v11) ==== 97+0+0 (secure 0 0 0) 0x7fb64c07a1c0 con 0x7fb660145590 2024-04-25T01:17:58.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.229+0000 7fb668400640 1 -- 172.21.15.90:0/3865599122 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb64006be30 msgr2=0x7fb64006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:58.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.229+0000 7fb668400640 1 --2- 172.21.15.90:0/3865599122 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb64006be30 0x7fb64006e2f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb654000f60 tx=0x7fb6540078f0 comp rx=0 tx=0).stop 2024-04-25T01:17:58.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.229+0000 7fb668400640 1 -- 172.21.15.90:0/3865599122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb660145590 msgr2=0x7fb66013a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:58.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.229+0000 7fb668400640 1 --2- 172.21.15.90:0/3865599122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb660145590 0x7fb66013a480 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fb64c000c00 tx=0x7fb64c0046c0 comp rx=0 tx=0).stop 2024-04-25T01:17:58.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.229+0000 7fb668400640 1 -- 172.21.15.90:0/3865599122 shutdown_connections 2024-04-25T01:17:58.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.229+0000 7fb668400640 1 --2- 172.21.15.90:0/3865599122 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb64006be30 0x7fb64006e2f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:58.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.229+0000 7fb668400640 1 --2- 172.21.15.90:0/3865599122 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb66015f9b0 0x7fb66013ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:58.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.229+0000 7fb668400640 1 --2- 172.21.15.90:0/3865599122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb660145590 0x7fb66013a480 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:58.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.229+0000 7fb668400640 1 --2- 172.21.15.90:0/3865599122 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb66006d270 0x7fb6601516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:58.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.229+0000 7fb668400640 1 -- 172.21.15.90:0/3865599122 >> 172.21.15.90:0/3865599122 conn(0x7fb6600b1820 msgr2=0x7fb6601407f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:58.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.230+0000 7fb668400640 1 -- 172.21.15.90:0/3865599122 shutdown_connections 2024-04-25T01:17:58.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.230+0000 7fb668400640 1 -- 172.21.15.90:0/3865599122 wait complete. 2024-04-25T01:17:58.231 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-04-25T01:17:58.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.442+0000 7fe4c3200640 1 Processor -- start 2024-04-25T01:17:58.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.442+0000 7fe4c3200640 1 -- start start 2024-04-25T01:17:58.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.444+0000 7fe4c3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4bc0b50e0 0x7fe4bc161010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:58.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.444+0000 7fe4c3200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe4bc0d0ca0 0x7fe4bc161550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:58.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.444+0000 7fe4c3200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4bc0cf3e0 con 0x7fe4bc0d08c0 2024-04-25T01:17:58.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.444+0000 7fe4c3200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fe4bc0d2e80 con 0x7fe4bc0d0ca0 2024-04-25T01:17:58.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.444+0000 7fe4c3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe4bc0cf6c0 con 0x7fe4bc0b50e0 2024-04-25T01:17:58.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.444+0000 7fe4c2800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe4bc0d0ca0 0x7fe4bc161550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:58.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.444+0000 7fe4c2800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe4bc0d0ca0 0x7fe4bc161550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57030/0 (socket says 172.21.15.90:57030) 2024-04-25T01:17:58.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.444+0000 7fe4c2800640 1 -- 172.21.15.90:0/92472185 learned_addr learned my addr 172.21.15.90:0/92472185 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:58.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.445+0000 7fe4c2800640 1 -- 172.21.15.90:0/92472185 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4bc0b50e0 msgr2=0x7fe4bc161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:58.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.445+0000 7fe4c2800640 1 --2- 172.21.15.90:0/92472185 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4bc0b50e0 0x7fe4bc161010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:58.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.445+0000 7fe4c2800640 1 -- 172.21.15.90:0/92472185 >> v1:172.21.15.103:6789/0 conn(0x7fe4bc0d08c0 legacy=0x7fe4bc0ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:58.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.445+0000 7fe4c2800640 1 -- 172.21.15.90:0/92472185 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4bc0cf9a0 con 0x7fe4bc0d0ca0 2024-04-25T01:17:58.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.446+0000 7fe4c2800640 1 --2- 172.21.15.90:0/92472185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4bc0d0ca0 0x7fe4bc161550 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8009e50 tx=0x7fe4b802a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf7882eba3d7bfe0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:58.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.446+0000 7fe4c0a00640 1 -- 172.21.15.90:0/92472185 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b8036070 con 0x7fe4bc0d0ca0 2024-04-25T01:17:58.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.446+0000 7fe4c0a00640 1 -- 172.21.15.90:0/92472185 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4b8031600 con 0x7fe4bc0d0ca0 2024-04-25T01:17:58.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.446+0000 7fe4c0a00640 1 -- 172.21.15.90:0/92472185 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b80319d0 con 0x7fe4bc0d0ca0 2024-04-25T01:17:58.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.446+0000 7fe4c3200640 1 -- 172.21.15.90:0/92472185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4bc0d0ca0 msgr2=0x7fe4bc161550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:58.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.446+0000 7fe4c3200640 1 --2- 172.21.15.90:0/92472185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4bc0d0ca0 0x7fe4bc161550 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8009e50 tx=0x7fe4b802a6d0 comp rx=0 tx=0).stop 2024-04-25T01:17:58.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.446+0000 7fe4c3200640 1 -- 172.21.15.90:0/92472185 shutdown_connections 2024-04-25T01:17:58.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.446+0000 7fe4c3200640 1 --2- 172.21.15.90:0/92472185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4bc0d0ca0 0x7fe4bc161550 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:58.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.446+0000 7fe4c3200640 1 --2- 172.21.15.90:0/92472185 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4bc0b50e0 0x7fe4bc161010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:58.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.446+0000 7fe4c3200640 1 -- 172.21.15.90:0/92472185 >> 172.21.15.90:0/92472185 conn(0x7fe4bc09d070 msgr2=0x7fe4bc0d3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:58.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.447+0000 7fe4c3200640 1 -- 172.21.15.90:0/92472185 shutdown_connections 2024-04-25T01:17:58.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.447+0000 7fe4c3200640 1 -- 172.21.15.90:0/92472185 wait complete. 2024-04-25T01:17:58.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.448+0000 7fe4c3200640 1 Processor -- start 2024-04-25T01:17:58.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.448+0000 7fe4c3200640 1 -- start start 2024-04-25T01:17:58.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.449+0000 7fe4c3200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4bc0b50e0 0x7fe4bc1510c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:58.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.449+0000 7fe4c3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4bc0d08c0 0x7fe4bc1e8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:58.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.449+0000 7fe4c3200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4bc0d0ca0 0x7fe4bc0c9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:58.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.449+0000 7fe4c3200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe4bc161a90 con 0x7fe4bc0d0ca0 2024-04-25T01:17:58.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.449+0000 7fe4c3200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe4bc0d1950 con 0x7fe4bc0b50e0 2024-04-25T01:17:58.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.449+0000 7fe4c3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe4bc0cf0a0 con 0x7fe4bc0d08c0 2024-04-25T01:17:58.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.450+0000 7fe4c2800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4bc0d0ca0 0x7fe4bc0c9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:58.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.450+0000 7fe4c2800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4bc0d0ca0 0x7fe4bc0c9b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57040/0 (socket says 172.21.15.90:57040) 2024-04-25T01:17:58.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.450+0000 7fe4c2800640 1 -- 172.21.15.90:0/3355505924 learned_addr learned my addr 172.21.15.90:0/3355505924 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:58.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.450+0000 7fe4c1400640 1 --2- 172.21.15.90:0/3355505924 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4bc0d08c0 0x7fe4bc1e8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:58.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.450+0000 7fe4c1e00640 1 --2- 172.21.15.90:0/3355505924 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4bc0b50e0 0x7fe4bc1510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:58.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.450+0000 7fe4c2800640 1 -- 172.21.15.90:0/3355505924 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4bc0d08c0 msgr2=0x7fe4bc1e8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:58.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.450+0000 7fe4c2800640 1 --2- 172.21.15.90:0/3355505924 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4bc0d08c0 0x7fe4bc1e8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:58.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.450+0000 7fe4c2800640 1 -- 172.21.15.90:0/3355505924 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4bc0b50e0 msgr2=0x7fe4bc1510c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:58.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.450+0000 7fe4c2800640 1 --2- 172.21.15.90:0/3355505924 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4bc0b50e0 0x7fe4bc1510c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:58.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.450+0000 7fe4c2800640 1 -- 172.21.15.90:0/3355505924 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4bc1e74f0 con 0x7fe4bc0d0ca0 2024-04-25T01:17:58.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.450+0000 7fe4c1400640 1 --2- 172.21.15.90:0/3355505924 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4bc0d08c0 0x7fe4bc1e8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:58.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.450+0000 7fe4c1e00640 1 --2- 172.21.15.90:0/3355505924 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4bc0b50e0 0x7fe4bc1510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:58.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.451+0000 7fe4c2800640 1 --2- 172.21.15.90:0/3355505924 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4bc0d0ca0 0x7fe4bc0c9b70 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8034040 tx=0x7fe4b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:58.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.451+0000 7fe4b6000640 1 -- 172.21.15.90:0/3355505924 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b8041020 con 0x7fe4bc0d0ca0 2024-04-25T01:17:58.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.451+0000 7fe4b6000640 1 -- 172.21.15.90:0/3355505924 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4b80077a0 con 0x7fe4bc0d0ca0 2024-04-25T01:17:58.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.451+0000 7fe4b6000640 1 -- 172.21.15.90:0/3355505924 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b8036040 con 0x7fe4bc0d0ca0 2024-04-25T01:17:58.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.451+0000 7fe4c3200640 1 -- 172.21.15.90:0/3355505924 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4bc1e77d0 con 0x7fe4bc0d0ca0 2024-04-25T01:17:58.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.451+0000 7fe4c3200640 1 -- 172.21.15.90:0/3355505924 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4bc1e7d90 con 0x7fe4bc0d0ca0 2024-04-25T01:17:58.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.452+0000 7fe4c3200640 1 -- 172.21.15.90:0/3355505924 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4bc075350 con 0x7fe4bc0d0ca0 2024-04-25T01:17:58.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.453+0000 7fe4b6000640 1 -- 172.21.15.90:0/3355505924 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe4b8044020 con 0x7fe4bc0d0ca0 2024-04-25T01:17:58.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.454+0000 7fe4b6000640 1 --2- 172.21.15.90:0/3355505924 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe4a006be30 0x7fe4a006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:58.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.454+0000 7fe4b6000640 1 -- 172.21.15.90:0/3355505924 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5249+0+0 (secure 0 0 0) 0x7fe4b80abb60 con 0x7fe4bc0d0ca0 2024-04-25T01:17:58.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.454+0000 7fe4c1e00640 1 --2- 172.21.15.90:0/3355505924 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe4a006be30 0x7fe4a006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:58.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.455+0000 7fe4c1e00640 1 --2- 172.21.15.90:0/3355505924 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe4a006be30 0x7fe4a006e2f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe4b0007fd0 tx=0x7fe4b0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:58.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.458+0000 7fe4b6000640 1 -- 172.21.15.90:0/3355505924 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe4b804b030 con 0x7fe4bc0d0ca0 2024-04-25T01:17:58.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.727+0000 7fe4c3200640 1 -- 172.21.15.90:0/3355505924 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7fe4bc0b8b60 con 0x7fe4bc0d0ca0 2024-04-25T01:17:58.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.736+0000 7fe4b6000640 1 -- 172.21.15.90:0/3355505924 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v12) ==== 90+0+0 (secure 0 0 0) 0x7fe4b807a6c0 con 0x7fe4bc0d0ca0 2024-04-25T01:17:58.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.739+0000 7fe4c3200640 1 -- 172.21.15.90:0/3355505924 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe4a006be30 msgr2=0x7fe4a006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:58.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.739+0000 7fe4c3200640 1 --2- 172.21.15.90:0/3355505924 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe4a006be30 0x7fe4a006e2f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe4b0007fd0 tx=0x7fe4b0009040 comp rx=0 tx=0).stop 2024-04-25T01:17:58.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.739+0000 7fe4c3200640 1 -- 172.21.15.90:0/3355505924 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4bc0d0ca0 msgr2=0x7fe4bc0c9b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:58.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.739+0000 7fe4c3200640 1 --2- 172.21.15.90:0/3355505924 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4bc0d0ca0 0x7fe4bc0c9b70 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8034040 tx=0x7fe4b8004a40 comp rx=0 tx=0).stop 2024-04-25T01:17:58.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.739+0000 7fe4c3200640 1 -- 172.21.15.90:0/3355505924 shutdown_connections 2024-04-25T01:17:58.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.739+0000 7fe4c3200640 1 --2- 172.21.15.90:0/3355505924 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe4a006be30 0x7fe4a006e2f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:58.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.739+0000 7fe4c3200640 1 --2- 172.21.15.90:0/3355505924 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4bc0d0ca0 0x7fe4bc0c9b70 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:58.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.739+0000 7fe4c3200640 1 --2- 172.21.15.90:0/3355505924 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe4bc0d08c0 0x7fe4bc1e8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:58.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.739+0000 7fe4c3200640 1 --2- 172.21.15.90:0/3355505924 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4bc0b50e0 0x7fe4bc1510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:58.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.739+0000 7fe4c3200640 1 -- 172.21.15.90:0/3355505924 >> 172.21.15.90:0/3355505924 conn(0x7fe4bc09d070 msgr2=0x7fe4bc0cce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:58.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.739+0000 7fe4c3200640 1 -- 172.21.15.90:0/3355505924 shutdown_connections 2024-04-25T01:17:58.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.739+0000 7fe4c3200640 1 -- 172.21.15.90:0/3355505924 wait complete. 2024-04-25T01:17:58.742 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:17:58.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.943+0000 7fd462800640 1 Processor -- start 2024-04-25T01:17:58.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.943+0000 7fd462800640 1 -- start start 2024-04-25T01:17:58.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.945+0000 7fd462800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd45c06b120 0x7fd45c06b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:58.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.945+0000 7fd462800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd45c15ca10 0x7fd45c15cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:58.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.945+0000 7fd462800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd45c06ba40 con 0x7fd45c0ae730 2024-04-25T01:17:58.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.945+0000 7fd462800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd45c0cef40 con 0x7fd45c15ca10 2024-04-25T01:17:58.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.945+0000 7fd462800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd45c15c580 con 0x7fd45c06b120 2024-04-25T01:17:58.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.945+0000 7fd460a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd45c0ae730 0x7fd45c0cd900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:54704/0 (socket says 172.21.15.90:54704) 2024-04-25T01:17:58.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.945+0000 7fd460a00640 1 -- 172.21.15.90:0/458161249 learned_addr learned my addr 172.21.15.90:0/458161249 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:58.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.945+0000 7fd461400640 1 --2- 172.21.15.90:0/458161249 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd45c06b120 0x7fd45c06b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:58.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.945+0000 7fd45be00640 1 -- 172.21.15.90:0/458161249 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 118054140 0 0) 0x7fd45c06ba40 con 0x7fd45c0ae730 2024-04-25T01:17:58.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.945+0000 7fd45be00640 1 -- 172.21.15.90:0/458161249 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd444003520 con 0x7fd45c0ae730 2024-04-25T01:17:58.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.945+0000 7fd461e00640 1 --2- 172.21.15.90:0/458161249 >> v2:172.21.15.103:3300/0 conn(0x7fd45c15ca10 0x7fd45c15cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:58.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.946+0000 7fd45be00640 1 -- 172.21.15.90:0/458161249 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3584863132 0 0) 0x7fd444003520 con 0x7fd45c0ae730 2024-04-25T01:17:58.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.946+0000 7fd45be00640 1 -- 172.21.15.90:0/458161249 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd45c06b120 msgr2=0x7fd45c06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:58.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.946+0000 7fd45be00640 1 --2- 172.21.15.90:0/458161249 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd45c06b120 0x7fd45c06b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:58.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.946+0000 7fd45be00640 1 -- 172.21.15.90:0/458161249 >> v2:172.21.15.103:3300/0 conn(0x7fd45c15ca10 msgr2=0x7fd45c15cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:58.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.946+0000 7fd45be00640 1 --2- 172.21.15.90:0/458161249 >> v2:172.21.15.103:3300/0 conn(0x7fd45c15ca10 0x7fd45c15cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:58.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.946+0000 7fd45be00640 1 -- 172.21.15.90:0/458161249 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd45c15c220 con 0x7fd45c0ae730 2024-04-25T01:17:58.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.946+0000 7fd45be00640 1 -- 172.21.15.90:0/458161249 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7fd448001670 con 0x7fd45c0ae730 2024-04-25T01:17:58.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.946+0000 7fd45be00640 1 -- 172.21.15.90:0/458161249 >> v1:172.21.15.90:6789/0 conn(0x7fd45c0ae730 legacy=0x7fd45c0cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:58.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.946+0000 7fd45be00640 1 --2- 172.21.15.90:0/458161249 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4440039b0 0x7fd444003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:58.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.946+0000 7fd45be00640 1 --2- 172.21.15.90:0/458161249 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd444004800 0x7fd444004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:58.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.946+0000 7fd45be00640 1 --2- 172.21.15.90:0/458161249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4440051c0 0x7fd44400fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:58.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.946+0000 7fd45be00640 1 -- 172.21.15.90:0/458161249 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd45c15c580 con 0x7fd4440051c0 2024-04-25T01:17:58.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.946+0000 7fd45be00640 1 -- 172.21.15.90:0/458161249 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd4440103c0 con 0x7fd4440039b0 2024-04-25T01:17:58.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.946+0000 7fd45be00640 1 -- 172.21.15.90:0/458161249 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd444010590 con 0x7fd444004800 2024-04-25T01:17:58.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.946+0000 7fd461400640 1 --2- 172.21.15.90:0/458161249 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd45c06b120 0x7fd45c06b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:17:58.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.946+0000 7fd461e00640 1 --2- 172.21.15.90:0/458161249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4440051c0 0x7fd44400fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:58.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.947+0000 7fd461e00640 1 -- 172.21.15.90:0/458161249 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd444004800 msgr2=0x7fd444004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:58.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.947+0000 7fd461e00640 1 --2- 172.21.15.90:0/458161249 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd444004800 0x7fd444004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:58.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.947+0000 7fd461e00640 1 -- 172.21.15.90:0/458161249 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4440039b0 msgr2=0x7fd444003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:58.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.947+0000 7fd461e00640 1 --2- 172.21.15.90:0/458161249 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4440039b0 0x7fd444003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:58.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.947+0000 7fd461e00640 1 -- 172.21.15.90:0/458161249 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd444003520 con 0x7fd4440051c0 2024-04-25T01:17:58.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.948+0000 7fd461e00640 1 --2- 172.21.15.90:0/458161249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4440051c0 0x7fd44400fdd0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd454007390 tx=0x7fd444005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=878ea33aacc47500 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:58.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.948+0000 7fd45be00640 1 -- 172.21.15.90:0/458161249 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd454009be0 con 0x7fd4440051c0 2024-04-25T01:17:58.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.948+0000 7fd45be00640 1 -- 172.21.15.90:0/458161249 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd45402e640 con 0x7fd4440051c0 2024-04-25T01:17:58.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.948+0000 7fd45be00640 1 -- 172.21.15.90:0/458161249 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd45402ea10 con 0x7fd4440051c0 2024-04-25T01:17:58.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.948+0000 7fd462800640 1 -- 172.21.15.90:0/458161249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4440051c0 msgr2=0x7fd44400fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:58.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.948+0000 7fd462800640 1 --2- 172.21.15.90:0/458161249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4440051c0 0x7fd44400fdd0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd454007390 tx=0x7fd444005830 comp rx=0 tx=0).stop 2024-04-25T01:17:58.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.948+0000 7fd461e00640 1 -- 172.21.15.90:0/458161249 reap_dead start 2024-04-25T01:17:58.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.948+0000 7fd462800640 1 -- 172.21.15.90:0/458161249 shutdown_connections 2024-04-25T01:17:58.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.948+0000 7fd462800640 1 -- 172.21.15.90:0/458161249 >> 172.21.15.90:0/458161249 conn(0x7fd45c0b1660 msgr2=0x7fd45c0b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:58.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.949+0000 7fd462800640 1 -- 172.21.15.90:0/458161249 shutdown_connections 2024-04-25T01:17:58.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.949+0000 7fd462800640 1 -- 172.21.15.90:0/458161249 wait complete. 2024-04-25T01:17:58.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.950+0000 7fd462800640 1 Processor -- start 2024-04-25T01:17:58.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.950+0000 7fd462800640 1 -- start start 2024-04-25T01:17:58.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.951+0000 7fd462800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4440051c0 0x7fd45c15b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:58.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.951+0000 7fd462800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd45c15abc0 0x7fd45c14e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:58.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.951+0000 7fd462800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd45c0b35e0 0x7fd45c0b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:58.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.951+0000 7fd462800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd45c0af240 con 0x7fd4440051c0 2024-04-25T01:17:58.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.951+0000 7fd462800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd45c0aef60 con 0x7fd45c0b35e0 2024-04-25T01:17:58.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.951+0000 7fd462800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd45c153080 con 0x7fd45c15abc0 2024-04-25T01:17:58.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.951+0000 7fd461400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4440051c0 0x7fd45c15b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:58.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.951+0000 7fd461400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4440051c0 0x7fd45c15b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57056/0 (socket says 172.21.15.90:57056) 2024-04-25T01:17:58.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.951+0000 7fd461400640 1 -- 172.21.15.90:0/1313982158 learned_addr learned my addr 172.21.15.90:0/1313982158 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:58.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.952+0000 7fd461400640 1 -- 172.21.15.90:0/1313982158 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd45c15abc0 msgr2=0x7fd45c14e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:58.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.952+0000 7fd461400640 1 --2- 172.21.15.90:0/1313982158 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd45c15abc0 0x7fd45c14e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:58.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.952+0000 7fd461400640 1 -- 172.21.15.90:0/1313982158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd45c0b35e0 msgr2=0x7fd45c0b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:58.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.952+0000 7fd461400640 1 --2- 172.21.15.90:0/1313982158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd45c0b35e0 0x7fd45c0b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:58.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.952+0000 7fd461400640 1 -- 172.21.15.90:0/1313982158 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd45c1e3fa0 con 0x7fd4440051c0 2024-04-25T01:17:58.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.952+0000 7fd461400640 1 --2- 172.21.15.90:0/1313982158 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4440051c0 0x7fd45c15b930 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fd45000e560 tx=0x7fd45000ea20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:58.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.952+0000 7fd459600640 1 -- 172.21.15.90:0/1313982158 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd45000f040 con 0x7fd4440051c0 2024-04-25T01:17:58.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.952+0000 7fd459600640 1 -- 172.21.15.90:0/1313982158 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd450004250 con 0x7fd4440051c0 2024-04-25T01:17:58.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.952+0000 7fd459600640 1 -- 172.21.15.90:0/1313982158 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd450003610 con 0x7fd4440051c0 2024-04-25T01:17:58.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.952+0000 7fd462800640 1 -- 172.21.15.90:0/1313982158 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd45c1e4240 con 0x7fd4440051c0 2024-04-25T01:17:58.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.953+0000 7fd462800640 1 -- 172.21.15.90:0/1313982158 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd45c1e4800 con 0x7fd4440051c0 2024-04-25T01:17:58.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.954+0000 7fd462800640 1 -- 172.21.15.90:0/1313982158 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd45c075350 con 0x7fd4440051c0 2024-04-25T01:17:58.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.955+0000 7fd459600640 1 -- 172.21.15.90:0/1313982158 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd4500037b0 con 0x7fd4440051c0 2024-04-25T01:17:58.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.955+0000 7fd459600640 1 --2- 172.21.15.90:0/1313982158 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd43c06be30 0x7fd43c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:58.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.956+0000 7fd459600640 1 -- 172.21.15.90:0/1313982158 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5249+0+0 (secure 0 0 0) 0x7fd45008a8e0 con 0x7fd4440051c0 2024-04-25T01:17:58.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.956+0000 7fd460a00640 1 --2- 172.21.15.90:0/1313982158 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd43c06be30 0x7fd43c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:58.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.956+0000 7fd460a00640 1 --2- 172.21.15.90:0/1313982158 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd43c06be30 0x7fd43c06e2f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd4480017e0 tx=0x7fd448001490 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:58.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:58.959+0000 7fd459600640 1 -- 172.21.15.90:0/1313982158 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd450017020 con 0x7fd4440051c0 2024-04-25T01:17:59.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.146+0000 7fd462800640 1 -- 172.21.15.90:0/1313982158 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd45c06b880 con 0x7fd4440051c0 2024-04-25T01:17:59.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.148+0000 7fd459600640 1 -- 172.21.15.90:0/1313982158 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) ==== 74+0+13128 (secure 0 0 0) 0x7fd4500593c0 con 0x7fd4440051c0 2024-04-25T01:17:59.132 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:17:59.133 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":36,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:17:55.185837+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:17:59.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.151+0000 7fd462800640 1 -- 172.21.15.90:0/1313982158 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd43c06be30 msgr2=0x7fd43c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:59.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.151+0000 7fd462800640 1 --2- 172.21.15.90:0/1313982158 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd43c06be30 0x7fd43c06e2f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd4480017e0 tx=0x7fd448001490 comp rx=0 tx=0).stop 2024-04-25T01:17:59.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.151+0000 7fd462800640 1 -- 172.21.15.90:0/1313982158 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4440051c0 msgr2=0x7fd45c15b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:59.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.151+0000 7fd462800640 1 --2- 172.21.15.90:0/1313982158 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4440051c0 0x7fd45c15b930 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fd45000e560 tx=0x7fd45000ea20 comp rx=0 tx=0).stop 2024-04-25T01:17:59.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.151+0000 7fd462800640 1 -- 172.21.15.90:0/1313982158 shutdown_connections 2024-04-25T01:17:59.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.151+0000 7fd462800640 1 --2- 172.21.15.90:0/1313982158 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd43c06be30 0x7fd43c06e2f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:59.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.151+0000 7fd462800640 1 --2- 172.21.15.90:0/1313982158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd45c0b35e0 0x7fd45c0b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:59.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.151+0000 7fd462800640 1 --2- 172.21.15.90:0/1313982158 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd45c15abc0 0x7fd45c14e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:59.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.151+0000 7fd462800640 1 --2- 172.21.15.90:0/1313982158 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4440051c0 0x7fd45c15b930 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:59.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.151+0000 7fd462800640 1 -- 172.21.15.90:0/1313982158 >> 172.21.15.90:0/1313982158 conn(0x7fd45c0b1660 msgr2=0x7fd45c06dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:59.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.151+0000 7fd462800640 1 -- 172.21.15.90:0/1313982158 shutdown_connections 2024-04-25T01:17:59.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.151+0000 7fd462800640 1 -- 172.21.15.90:0/1313982158 wait complete. 2024-04-25T01:17:59.151 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:16:15.894126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:17:59.151 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:17:59.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.356+0000 7f9826600640 1 Processor -- start 2024-04-25T01:17:59.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.356+0000 7f9826600640 1 -- start start 2024-04-25T01:17:59.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.357+0000 7f9826600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9820158a60 0x7f982006df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:59.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.357+0000 7f9826600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f982006d140 con 0x7f982015f4c0 2024-04-25T01:17:59.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.357+0000 7f9826600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f982006d420 con 0x7f9820158680 2024-04-25T01:17:59.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.358+0000 7f9826600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f982006d700 con 0x7f9820158a60 2024-04-25T01:17:59.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.358+0000 7f981fe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f982015f4c0 0x7f98200b3550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:54716/0 (socket says 172.21.15.90:54716) 2024-04-25T01:17:59.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.358+0000 7f981fe00640 1 -- 172.21.15.90:0/3187841151 learned_addr learned my addr 172.21.15.90:0/3187841151 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:59.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.358+0000 7f9816a00640 1 --2- 172.21.15.90:0/3187841151 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9820158a60 0x7f982006df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:59.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.358+0000 7f981ea00640 1 -- 172.21.15.90:0/3187841151 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4136762241 0 0) 0x7f982006d140 con 0x7f982015f4c0 2024-04-25T01:17:59.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.358+0000 7f981ea00640 1 -- 172.21.15.90:0/3187841151 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9804003520 con 0x7f982015f4c0 2024-04-25T01:17:59.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.359+0000 7f981ea00640 1 -- 172.21.15.90:0/3187841151 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 44600475 0 0) 0x7f9804003520 con 0x7f982015f4c0 2024-04-25T01:17:59.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.359+0000 7f981ea00640 1 -- 172.21.15.90:0/3187841151 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9820158a60 msgr2=0x7f982006df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:59.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.359+0000 7f981ea00640 1 --2- 172.21.15.90:0/3187841151 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9820158a60 0x7f982006df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:59.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.359+0000 7f981ea00640 1 -- 172.21.15.90:0/3187841151 >> v1:172.21.15.103:6789/0 conn(0x7f9820158680 legacy=0x7f982006e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:59.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.359+0000 7f981ea00640 1 -- 172.21.15.90:0/3187841151 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f982006d9e0 con 0x7f982015f4c0 2024-04-25T01:17:59.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.359+0000 7f981ea00640 1 -- 172.21.15.90:0/3187841151 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f98180019f0 con 0x7f982015f4c0 2024-04-25T01:17:59.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.359+0000 7f981ea00640 1 -- 172.21.15.90:0/3187841151 >> v1:172.21.15.90:6789/0 conn(0x7f982015f4c0 legacy=0x7f98200b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:59.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.359+0000 7f981ea00640 1 --2- 172.21.15.90:0/3187841151 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9820158a60 0x7f98040039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:59.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.359+0000 7f981ea00640 1 --2- 172.21.15.90:0/3187841151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9804003f60 0x7f9804004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:59.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.359+0000 7f981ea00640 1 --2- 172.21.15.90:0/3187841151 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9804004d40 0x7f98040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:59.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.359+0000 7f981ea00640 1 -- 172.21.15.90:0/3187841151 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f982006d700 con 0x7f9804004d40 2024-04-25T01:17:59.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.359+0000 7f981ea00640 1 -- 172.21.15.90:0/3187841151 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f980400bb50 con 0x7f9804003f60 2024-04-25T01:17:59.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.359+0000 7f981ea00640 1 -- 172.21.15.90:0/3187841151 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f980400bd50 con 0x7f9820158a60 2024-04-25T01:17:59.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.359+0000 7f981fe00640 1 --2- 172.21.15.90:0/3187841151 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9804004d40 0x7f98040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:59.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.359+0000 7f981fe00640 1 -- 172.21.15.90:0/3187841151 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9820158a60 msgr2=0x7f98040039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:59.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.359+0000 7f981fe00640 1 --2- 172.21.15.90:0/3187841151 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9820158a60 0x7f98040039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:59.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.359+0000 7f981fe00640 1 -- 172.21.15.90:0/3187841151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9804003f60 msgr2=0x7f9804004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:59.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.360+0000 7f9816a00640 1 --2- 172.21.15.90:0/3187841151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9804003f60 0x7f9804004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:59.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.360+0000 7f981fe00640 1 --2- 172.21.15.90:0/3187841151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9804003f60 0x7f9804004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:59.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.360+0000 7f981fe00640 1 -- 172.21.15.90:0/3187841151 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9804003520 con 0x7f9804004d40 2024-04-25T01:17:59.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.361+0000 7f981fe00640 1 --2- 172.21.15.90:0/3187841151 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9804004d40 0x7f98040051c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f981800a840 tx=0x7f9804005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8275a6adea7ae04 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:59.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.361+0000 7f981ea00640 1 -- 172.21.15.90:0/3187841151 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f981800a460 con 0x7f9804004d40 2024-04-25T01:17:59.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.361+0000 7f981ea00640 1 -- 172.21.15.90:0/3187841151 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f981802f070 con 0x7f9804004d40 2024-04-25T01:17:59.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.361+0000 7f981ea00640 1 -- 172.21.15.90:0/3187841151 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f981802bcb0 con 0x7f9804004d40 2024-04-25T01:17:59.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.361+0000 7f9826600640 1 -- 172.21.15.90:0/3187841151 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9804004d40 msgr2=0x7f98040051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:59.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.361+0000 7f9826600640 1 --2- 172.21.15.90:0/3187841151 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9804004d40 0x7f98040051c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f981800a840 tx=0x7f9804005830 comp rx=0 tx=0).stop 2024-04-25T01:17:59.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.361+0000 7f981fe00640 1 -- 172.21.15.90:0/3187841151 reap_dead start 2024-04-25T01:17:59.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.361+0000 7f9826600640 1 -- 172.21.15.90:0/3187841151 shutdown_connections 2024-04-25T01:17:59.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.361+0000 7f9826600640 1 -- 172.21.15.90:0/3187841151 >> 172.21.15.90:0/3187841151 conn(0x7f982006b720 msgr2=0x7f98200b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:59.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.362+0000 7f9826600640 1 -- 172.21.15.90:0/3187841151 shutdown_connections 2024-04-25T01:17:59.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.362+0000 7f9826600640 1 -- 172.21.15.90:0/3187841151 wait complete. 2024-04-25T01:17:59.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.363+0000 7f9826600640 1 Processor -- start 2024-04-25T01:17:59.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.363+0000 7f9826600640 1 -- start start 2024-04-25T01:17:59.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.364+0000 7f9826600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9804004d40 0x7f9820144c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:59.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.364+0000 7f9826600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98201451a0 0x7f98200c4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:59.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.364+0000 7f9826600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98200c4ff0 0x7f9820159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:59.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.364+0000 7f9826600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f98200b3de0 con 0x7f98201451a0 2024-04-25T01:17:59.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.364+0000 7f9826600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f98200b3c60 con 0x7f98200c4ff0 2024-04-25T01:17:59.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.364+0000 7f9826600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f982006df00 con 0x7f9804004d40 2024-04-25T01:17:59.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.365+0000 7f9816a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98201451a0 0x7f98200c4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:59.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.365+0000 7f9816a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98201451a0 0x7f98200c4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57070/0 (socket says 172.21.15.90:57070) 2024-04-25T01:17:59.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.365+0000 7f9816a00640 1 -- 172.21.15.90:0/1495254265 learned_addr learned my addr 172.21.15.90:0/1495254265 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:59.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.365+0000 7f9816a00640 1 -- 172.21.15.90:0/1495254265 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9804004d40 msgr2=0x7f9820144c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:59.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.365+0000 7f981f400640 1 --2- 172.21.15.90:0/1495254265 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9804004d40 0x7f9820144c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:59.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.365+0000 7f981fe00640 1 --2- 172.21.15.90:0/1495254265 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98200c4ff0 0x7f9820159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:59.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.365+0000 7f9816a00640 1 --2- 172.21.15.90:0/1495254265 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9804004d40 0x7f9820144c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:59.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.365+0000 7f9816a00640 1 -- 172.21.15.90:0/1495254265 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98200c4ff0 msgr2=0x7f9820159f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:59.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.365+0000 7f9816a00640 1 --2- 172.21.15.90:0/1495254265 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98200c4ff0 0x7f9820159f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:59.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.365+0000 7f9816a00640 1 -- 172.21.15.90:0/1495254265 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9820147bc0 con 0x7f98201451a0 2024-04-25T01:17:59.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.365+0000 7f9816a00640 1 --2- 172.21.15.90:0/1495254265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98201451a0 0x7f98200c4ab0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f98080108c0 tx=0x7f9808010d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:59.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.366+0000 7f9817e00640 1 -- 172.21.15.90:0/1495254265 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f980800f040 con 0x7f98201451a0 2024-04-25T01:17:59.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.366+0000 7f9817e00640 1 -- 172.21.15.90:0/1495254265 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f980800eb60 con 0x7f98201451a0 2024-04-25T01:17:59.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.366+0000 7f9817e00640 1 -- 172.21.15.90:0/1495254265 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f980801c400 con 0x7f98201451a0 2024-04-25T01:17:59.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.366+0000 7f9826600640 1 -- 172.21.15.90:0/1495254265 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f982015ac80 con 0x7f98201451a0 2024-04-25T01:17:59.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.366+0000 7f9826600640 1 -- 172.21.15.90:0/1495254265 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f982015b240 con 0x7f98201451a0 2024-04-25T01:17:59.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.367+0000 7f9826600640 1 -- 172.21.15.90:0/1495254265 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9820068fb0 con 0x7f98201451a0 2024-04-25T01:17:59.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.368+0000 7f9817e00640 1 -- 172.21.15.90:0/1495254265 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f980801c5a0 con 0x7f98201451a0 2024-04-25T01:17:59.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.369+0000 7f9817e00640 1 --2- 172.21.15.90:0/1495254265 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f97fc06be30 0x7f97fc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:59.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.369+0000 7f9817e00640 1 -- 172.21.15.90:0/1495254265 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5249+0+0 (secure 0 0 0) 0x7f980808b370 con 0x7f98201451a0 2024-04-25T01:17:59.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.369+0000 7f981f400640 1 --2- 172.21.15.90:0/1495254265 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f97fc06be30 0x7f97fc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:59.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.370+0000 7f981f400640 1 --2- 172.21.15.90:0/1495254265 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f97fc06be30 0x7f97fc06e2f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f98100015d0 tx=0x7f9810009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:59.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.373+0000 7f9817e00640 1 -- 172.21.15.90:0/1495254265 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9808018030 con 0x7f98201451a0 2024-04-25T01:17:59.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.568+0000 7f9826600640 1 -- 172.21.15.90:0/1495254265 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f98200c53d0 con 0x7f98201451a0 2024-04-25T01:17:59.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.569+0000 7f9817e00640 1 -- 172.21.15.90:0/1495254265 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) ==== 93+0+10 (secure 0 0 0) 0x7f9808059880 con 0x7f98201451a0 2024-04-25T01:17:59.552 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T01:17:59.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.571+0000 7f9826600640 1 -- 172.21.15.90:0/1495254265 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f97fc06be30 msgr2=0x7f97fc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:59.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.571+0000 7f9826600640 1 --2- 172.21.15.90:0/1495254265 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f97fc06be30 0x7f97fc06e2f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f98100015d0 tx=0x7f9810009040 comp rx=0 tx=0).stop 2024-04-25T01:17:59.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.571+0000 7f9826600640 1 -- 172.21.15.90:0/1495254265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98201451a0 msgr2=0x7f98200c4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:59.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.571+0000 7f9826600640 1 --2- 172.21.15.90:0/1495254265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98201451a0 0x7f98200c4ab0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f98080108c0 tx=0x7f9808010d80 comp rx=0 tx=0).stop 2024-04-25T01:17:59.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.571+0000 7f9826600640 1 -- 172.21.15.90:0/1495254265 shutdown_connections 2024-04-25T01:17:59.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.571+0000 7f9826600640 1 --2- 172.21.15.90:0/1495254265 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f97fc06be30 0x7f97fc06e2f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:59.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.571+0000 7f9826600640 1 --2- 172.21.15.90:0/1495254265 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98200c4ff0 0x7f9820159f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:59.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.571+0000 7f9826600640 1 --2- 172.21.15.90:0/1495254265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98201451a0 0x7f98200c4ab0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:59.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.571+0000 7f9826600640 1 --2- 172.21.15.90:0/1495254265 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9804004d40 0x7f9820144c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:59.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.571+0000 7f9826600640 1 -- 172.21.15.90:0/1495254265 >> 172.21.15.90:0/1495254265 conn(0x7f982006b720 msgr2=0x7f98200b3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:59.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.572+0000 7f9826600640 1 -- 172.21.15.90:0/1495254265 shutdown_connections 2024-04-25T01:17:59.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.572+0000 7f9826600640 1 -- 172.21.15.90:0/1495254265 wait complete. 2024-04-25T01:17:59.575 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T01:17:59.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.789+0000 7f86ff600640 1 Processor -- start 2024-04-25T01:17:59.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.790+0000 7f86ff600640 1 -- start start 2024-04-25T01:17:59.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.791+0000 7f86ff600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f86f815e4c0 0x7f86f80d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:59.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.791+0000 7f86ff600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86f80d2d40 0x7f86f80cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:59.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.791+0000 7f86ff600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86f80d1e90 con 0x7f86f80d4a10 2024-04-25T01:17:59.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.791+0000 7f86ff600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f86f80b54a0 con 0x7f86f815e4c0 2024-04-25T01:17:59.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.791+0000 7f86ff600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f86f80d2130 con 0x7f86f80d2d40 2024-04-25T01:17:59.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.791+0000 7f86fcc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f86f815e4c0 0x7f86f80d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:59.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.791+0000 7f86f7e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f86f80d4a10 0x7f86f80b3ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:54718/0 (socket says 172.21.15.90:54718) 2024-04-25T01:17:59.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.791+0000 7f86f7e00640 1 -- 172.21.15.90:0/2601851852 learned_addr learned my addr 172.21.15.90:0/2601851852 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:17:59.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.791+0000 7f86fd600640 1 --2- 172.21.15.90:0/2601851852 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86f80d2d40 0x7f86f80cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:59.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.792+0000 7f86fcc00640 1 -- 172.21.15.90:0/2601851852 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86f80d2d40 msgr2=0x7f86f80cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:17:59.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.792+0000 7f86fcc00640 1 --2- 172.21.15.90:0/2601851852 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86f80d2d40 0x7f86f80cfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:59.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.792+0000 7f86fcc00640 1 -- 172.21.15.90:0/2601851852 >> v1:172.21.15.90:6789/0 conn(0x7f86f80d4a10 legacy=0x7f86f80b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:59.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.792+0000 7f86fcc00640 1 -- 172.21.15.90:0/2601851852 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86f80b5780 con 0x7f86f815e4c0 2024-04-25T01:17:59.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.793+0000 7f86fcc00640 1 --2- 172.21.15.90:0/2601851852 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f815e4c0 0x7f86f80d2800 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f86e8009d90 tx=0x7f86e802a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a63128e1322c63c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:59.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.793+0000 7f86f7400640 1 -- 172.21.15.90:0/2601851852 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e8031600 con 0x7f86f815e4c0 2024-04-25T01:17:59.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.793+0000 7f86f7400640 1 -- 172.21.15.90:0/2601851852 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86e80317a0 con 0x7f86f815e4c0 2024-04-25T01:17:59.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.793+0000 7f86f7400640 1 -- 172.21.15.90:0/2601851852 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e8031b70 con 0x7f86f815e4c0 2024-04-25T01:17:59.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.794+0000 7f86ff600640 1 -- 172.21.15.90:0/2601851852 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f815e4c0 msgr2=0x7f86f80d2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:59.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.794+0000 7f86ff600640 1 --2- 172.21.15.90:0/2601851852 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f815e4c0 0x7f86f80d2800 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f86e8009d90 tx=0x7f86e802a5d0 comp rx=0 tx=0).stop 2024-04-25T01:17:59.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.794+0000 7f86ff600640 1 -- 172.21.15.90:0/2601851852 shutdown_connections 2024-04-25T01:17:59.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.794+0000 7f86ff600640 1 --2- 172.21.15.90:0/2601851852 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86f80d2d40 0x7f86f80cfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:59.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.794+0000 7f86ff600640 1 --2- 172.21.15.90:0/2601851852 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f815e4c0 0x7f86f80d2800 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:59.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.794+0000 7f86ff600640 1 -- 172.21.15.90:0/2601851852 >> 172.21.15.90:0/2601851852 conn(0x7f86f80d3ef0 msgr2=0x7f86f80d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:59.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.794+0000 7f86ff600640 1 -- 172.21.15.90:0/2601851852 shutdown_connections 2024-04-25T01:17:59.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.794+0000 7f86ff600640 1 -- 172.21.15.90:0/2601851852 wait complete. 2024-04-25T01:17:59.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.795+0000 7f86ff600640 1 Processor -- start 2024-04-25T01:17:59.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.796+0000 7f86ff600640 1 -- start start 2024-04-25T01:17:59.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.796+0000 7f86ff600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f80d2d40 0x7f86f815a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:59.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.797+0000 7f86ff600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86f80d4a10 0x7f86f80cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:59.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.797+0000 7f86ff600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86f815e4c0 0x7f86f80cb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:59.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.797+0000 7f86ff600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f86f815ea20 con 0x7f86f80d2d40 2024-04-25T01:17:59.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.797+0000 7f86ff600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f86f815e8a0 con 0x7f86f815e4c0 2024-04-25T01:17:59.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.797+0000 7f86ff600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f86f80b45f0 con 0x7f86f80d4a10 2024-04-25T01:17:59.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.797+0000 7f86fcc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f80d2d40 0x7f86f815a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:59.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.797+0000 7f86fcc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f80d2d40 0x7f86f815a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57080/0 (socket says 172.21.15.90:57080) 2024-04-25T01:17:59.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.797+0000 7f86fcc00640 1 -- 172.21.15.90:0/1958038270 learned_addr learned my addr 172.21.15.90:0/1958038270 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:17:59.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.797+0000 7f86f7e00640 1 --2- 172.21.15.90:0/1958038270 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86f80d4a10 0x7f86f80cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:59.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.797+0000 7f86fd600640 1 --2- 172.21.15.90:0/1958038270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86f815e4c0 0x7f86f80cb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:59.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.797+0000 7f86fcc00640 1 -- 172.21.15.90:0/1958038270 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86f80d4a10 msgr2=0x7f86f80cdaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:59.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.797+0000 7f86fcc00640 1 --2- 172.21.15.90:0/1958038270 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86f80d4a10 0x7f86f80cdaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:59.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.797+0000 7f86fcc00640 1 -- 172.21.15.90:0/1958038270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86f815e4c0 msgr2=0x7f86f80cb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:59.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.797+0000 7f86fcc00640 1 --2- 172.21.15.90:0/1958038270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86f815e4c0 0x7f86f80cb620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:59.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.797+0000 7f86fcc00640 1 -- 172.21.15.90:0/1958038270 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86f80c9fc0 con 0x7f86f80d2d40 2024-04-25T01:17:59.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.797+0000 7f86f7e00640 1 --2- 172.21.15.90:0/1958038270 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86f80d4a10 0x7f86f80cdaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:17:59.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.798+0000 7f86fcc00640 1 --2- 172.21.15.90:0/1958038270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f80d2d40 0x7f86f815a680 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f86e8009eb0 tx=0x7f86e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:59.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.798+0000 7f86f4c00640 1 -- 172.21.15.90:0/1958038270 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e803f070 con 0x7f86f80d2d40 2024-04-25T01:17:59.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.798+0000 7f86f4c00640 1 -- 172.21.15.90:0/1958038270 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86e8030be0 con 0x7f86f80d2d40 2024-04-25T01:17:59.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.798+0000 7f86f4c00640 1 -- 172.21.15.90:0/1958038270 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e8044460 con 0x7f86f80d2d40 2024-04-25T01:17:59.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.798+0000 7f86ff600640 1 -- 172.21.15.90:0/1958038270 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86f80ca2a0 con 0x7f86f80d2d40 2024-04-25T01:17:59.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.798+0000 7f86ff600640 1 -- 172.21.15.90:0/1958038270 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86f80ca860 con 0x7f86f80d2d40 2024-04-25T01:17:59.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.799+0000 7f86ff600640 1 -- 172.21.15.90:0/1958038270 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86f8068fb0 con 0x7f86f80d2d40 2024-04-25T01:17:59.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.800+0000 7f86f4c00640 1 -- 172.21.15.90:0/1958038270 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f86e8044600 con 0x7f86f80d2d40 2024-04-25T01:17:59.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.801+0000 7f86f4c00640 1 --2- 172.21.15.90:0/1958038270 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f86d406be60 0x7f86d406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:17:59.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.801+0000 7f86f4c00640 1 -- 172.21.15.90:0/1958038270 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5249+0+0 (secure 0 0 0) 0x7f86e80ac3e0 con 0x7f86f80d2d40 2024-04-25T01:17:59.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.801+0000 7f86f7e00640 1 --2- 172.21.15.90:0/1958038270 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f86d406be60 0x7f86d406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:17:59.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.802+0000 7f86f7e00640 1 --2- 172.21.15.90:0/1958038270 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f86d406be60 0x7f86d406e320 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f86f80c87c0 tx=0x7f86e0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:17:59.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:17:59.805+0000 7f86f4c00640 1 -- 172.21.15.90:0/1958038270 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f86e807b000 con 0x7f86f80d2d40 2024-04-25T01:17:59.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.009+0000 7f86ff600640 1 -- 172.21.15.90:0/1958038270 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f86f80d4650 con 0x7f86f80d2d40 2024-04-25T01:17:59.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.010+0000 7f86f4c00640 1 -- 172.21.15.90:0/1958038270 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v36) ==== 84+0+65 (secure 0 0 0) 0x7f86e807a900 con 0x7f86f80d2d40 2024-04-25T01:17:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:17:59.994 INFO:teuthology.orchestra.run.smithi090.stdout:[ 2024-04-25T01:17:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T01:17:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: "poolnum": 1, 2024-04-25T01:17:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: "poolname": ".mgr" 2024-04-25T01:17:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:17:59.994 INFO:teuthology.orchestra.run.smithi090.stdout:] 2024-04-25T01:17:59.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.013+0000 7f86ff600640 1 -- 172.21.15.90:0/1958038270 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f86d406be60 msgr2=0x7f86d406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:59.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.013+0000 7f86ff600640 1 --2- 172.21.15.90:0/1958038270 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f86d406be60 0x7f86d406e320 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f86f80c87c0 tx=0x7f86e0009040 comp rx=0 tx=0).stop 2024-04-25T01:17:59.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.013+0000 7f86ff600640 1 -- 172.21.15.90:0/1958038270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f80d2d40 msgr2=0x7f86f815a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:17:59.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.013+0000 7f86ff600640 1 --2- 172.21.15.90:0/1958038270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f80d2d40 0x7f86f815a680 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f86e8009eb0 tx=0x7f86e8004a40 comp rx=0 tx=0).stop 2024-04-25T01:17:59.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.013+0000 7f86ff600640 1 -- 172.21.15.90:0/1958038270 shutdown_connections 2024-04-25T01:17:59.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.013+0000 7f86ff600640 1 --2- 172.21.15.90:0/1958038270 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f86d406be60 0x7f86d406e320 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:59.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.013+0000 7f86ff600640 1 --2- 172.21.15.90:0/1958038270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86f815e4c0 0x7f86f80cb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:59.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.013+0000 7f86ff600640 1 --2- 172.21.15.90:0/1958038270 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f86f80d4a10 0x7f86f80cdaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:59.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.013+0000 7f86ff600640 1 --2- 172.21.15.90:0/1958038270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f80d2d40 0x7f86f815a680 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:17:59.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.013+0000 7f86ff600640 1 -- 172.21.15.90:0/1958038270 >> 172.21.15.90:0/1958038270 conn(0x7f86f80d3ef0 msgr2=0x7f86f815d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:17:59.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.013+0000 7f86ff600640 1 -- 172.21.15.90:0/1958038270 shutdown_connections 2024-04-25T01:17:59.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.013+0000 7f86ff600640 1 -- 172.21.15.90:0/1958038270 wait complete. 2024-04-25T01:18:00.016 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T01:18:00.016 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T01:18:00.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.236+0000 7f4c4c800640 1 Processor -- start 2024-04-25T01:18:00.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.236+0000 7f4c4c800640 1 -- start start 2024-04-25T01:18:00.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.238+0000 7f4c4c800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c4406cb40 0x7f4c44070bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:00.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.238+0000 7f4c4c800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4c44071110 0x7f4c44071570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:00.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.238+0000 7f4c4c800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c440722c0 con 0x7f4c4415f870 2024-04-25T01:18:00.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.238+0000 7f4c4c800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4c440725a0 con 0x7f4c44071110 2024-04-25T01:18:00.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.238+0000 7f4c4c800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4c440ca1d0 con 0x7f4c4406cb40 2024-04-25T01:18:00.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.239+0000 7f4c4a800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4c44071110 0x7f4c44071570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:00.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.239+0000 7f4c49400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c4406cb40 0x7f4c44070bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:00.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.239+0000 7f4c4a800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4c44071110 0x7f4c44071570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.90:36264/0 (socket says 172.21.15.90:36264) 2024-04-25T01:18:00.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.239+0000 7f4c4a800640 1 -- 172.21.15.90:0/1280317513 learned_addr learned my addr 172.21.15.90:0/1280317513 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:00.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.239+0000 7f4c49400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c4406cb40 0x7f4c44070bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.90:47312/0 (socket says 172.21.15.90:47312) 2024-04-25T01:18:00.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.240+0000 7f4c4a800640 1 -- 172.21.15.90:0/1280317513 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c4406cb40 msgr2=0x7f4c44070bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:00.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.240+0000 7f4c4a800640 1 --2- 172.21.15.90:0/1280317513 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c4406cb40 0x7f4c44070bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:00.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.240+0000 7f4c4a800640 1 -- 172.21.15.90:0/1280317513 >> v1:172.21.15.103:6789/0 conn(0x7f4c4415f870 legacy=0x7f4c4415fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:00.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.240+0000 7f4c4a800640 1 -- 172.21.15.90:0/1280317513 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c440ca4b0 con 0x7f4c44071110 2024-04-25T01:18:00.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.241+0000 7f4c4a800640 1 --2- 172.21.15.90:0/1280317513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c44071110 0x7f4c44071570 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4c40009e50 tx=0x7f4c4002a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36a506eabe3efc48 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:00.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.242+0000 7f4c48a00640 1 -- 172.21.15.90:0/1280317513 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c40036070 con 0x7f4c44071110 2024-04-25T01:18:00.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.242+0000 7f4c48a00640 1 -- 172.21.15.90:0/1280317513 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c40031600 con 0x7f4c44071110 2024-04-25T01:18:00.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.242+0000 7f4c48a00640 1 -- 172.21.15.90:0/1280317513 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c400319d0 con 0x7f4c44071110 2024-04-25T01:18:00.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.242+0000 7f4c4c800640 1 -- 172.21.15.90:0/1280317513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c44071110 msgr2=0x7f4c44071570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:00.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.242+0000 7f4c4c800640 1 --2- 172.21.15.90:0/1280317513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c44071110 0x7f4c44071570 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4c40009e50 tx=0x7f4c4002a710 comp rx=0 tx=0).stop 2024-04-25T01:18:00.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.242+0000 7f4c4c800640 1 -- 172.21.15.90:0/1280317513 shutdown_connections 2024-04-25T01:18:00.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.242+0000 7f4c4c800640 1 --2- 172.21.15.90:0/1280317513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c44071110 0x7f4c44071570 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:00.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.242+0000 7f4c4c800640 1 --2- 172.21.15.90:0/1280317513 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c4406cb40 0x7f4c44070bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:00.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.242+0000 7f4c4c800640 1 -- 172.21.15.90:0/1280317513 >> 172.21.15.90:0/1280317513 conn(0x7f4c440cfae0 msgr2=0x7f4c440cfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:00.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.242+0000 7f4c4c800640 1 -- 172.21.15.90:0/1280317513 shutdown_connections 2024-04-25T01:18:00.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.242+0000 7f4c4c800640 1 -- 172.21.15.90:0/1280317513 wait complete. 2024-04-25T01:18:00.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.244+0000 7f4c4c800640 1 Processor -- start 2024-04-25T01:18:00.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.245+0000 7f4c4c800640 1 -- start start 2024-04-25T01:18:00.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.246+0000 7f4c4c800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c4406cb40 0x7f4c4415e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:00.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.246+0000 7f4c4c800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c44071110 0x7f4c4413bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:00.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.246+0000 7f4c4c800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c4415f870 0x7f4c44143f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:00.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.246+0000 7f4c4c800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4c44145a90 con 0x7f4c4415f870 2024-04-25T01:18:00.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.246+0000 7f4c4c800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4c44145910 con 0x7f4c44071110 2024-04-25T01:18:00.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.246+0000 7f4c4c800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4c4406cf20 con 0x7f4c4406cb40 2024-04-25T01:18:00.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.246+0000 7f4c4a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c4415f870 0x7f4c44143f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:00.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.246+0000 7f4c4a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c4415f870 0x7f4c44143f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57092/0 (socket says 172.21.15.90:57092) 2024-04-25T01:18:00.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.246+0000 7f4c4a800640 1 -- 172.21.15.90:0/2457326960 learned_addr learned my addr 172.21.15.90:0/2457326960 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:00.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.246+0000 7f4c49e00640 1 --2- 172.21.15.90:0/2457326960 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c4406cb40 0x7f4c4415e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:00.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.246+0000 7f4c49400640 1 --2- 172.21.15.90:0/2457326960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c44071110 0x7f4c4413bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:00.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.247+0000 7f4c4a800640 1 -- 172.21.15.90:0/2457326960 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c4406cb40 msgr2=0x7f4c4415e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:00.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.247+0000 7f4c4a800640 1 --2- 172.21.15.90:0/2457326960 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c4406cb40 0x7f4c4415e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:00.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.247+0000 7f4c4a800640 1 -- 172.21.15.90:0/2457326960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c44071110 msgr2=0x7f4c4413bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:00.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.247+0000 7f4c4a800640 1 --2- 172.21.15.90:0/2457326960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c44071110 0x7f4c4413bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:00.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.247+0000 7f4c4a800640 1 -- 172.21.15.90:0/2457326960 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c4413aa40 con 0x7f4c4415f870 2024-04-25T01:18:00.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.247+0000 7f4c4a800640 1 --2- 172.21.15.90:0/2457326960 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c4415f870 0x7f4c44143f50 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f4c40009f70 tx=0x7f4c40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:00.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.247+0000 7f4c3e000640 1 -- 172.21.15.90:0/2457326960 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c40036050 con 0x7f4c4415f870 2024-04-25T01:18:00.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.247+0000 7f4c3e000640 1 -- 172.21.15.90:0/2457326960 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c40042070 con 0x7f4c4415f870 2024-04-25T01:18:00.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.247+0000 7f4c4c800640 1 -- 172.21.15.90:0/2457326960 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c4413ad20 con 0x7f4c4415f870 2024-04-25T01:18:00.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.247+0000 7f4c3e000640 1 -- 172.21.15.90:0/2457326960 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c40009510 con 0x7f4c4415f870 2024-04-25T01:18:00.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.248+0000 7f4c4c800640 1 -- 172.21.15.90:0/2457326960 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c4413b2e0 con 0x7f4c4415f870 2024-04-25T01:18:00.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.249+0000 7f4c4c800640 1 -- 172.21.15.90:0/2457326960 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c44068fb0 con 0x7f4c4415f870 2024-04-25T01:18:00.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.250+0000 7f4c3e000640 1 -- 172.21.15.90:0/2457326960 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4c40007c70 con 0x7f4c4415f870 2024-04-25T01:18:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.251+0000 7f4c3e000640 1 --2- 172.21.15.90:0/2457326960 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4c2806be60 0x7f4c2806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.251+0000 7f4c3e000640 1 -- 172.21.15.90:0/2457326960 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5249+0+0 (secure 0 0 0) 0x7f4c400abab0 con 0x7f4c4415f870 2024-04-25T01:18:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.251+0000 7f4c49e00640 1 --2- 172.21.15.90:0/2457326960 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4c2806be60 0x7f4c2806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:00.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.252+0000 7f4c49e00640 1 --2- 172.21.15.90:0/2457326960 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4c2806be60 0x7f4c2806e320 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4c38007fd0 tx=0x7f4c38009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:00.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.255+0000 7f4c3e000640 1 -- 172.21.15.90:0/2457326960 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4c4007a6a0 con 0x7f4c4415f870 2024-04-25T01:18:00.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:00.452+0000 7f4c4c800640 1 -- 172.21.15.90:0/2457326960 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f4c440c9ea0 con 0x7f4c4415f870 2024-04-25T01:18:01.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.211+0000 7f4c3e000640 1 -- 172.21.15.90:0/2457326960 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v37) ==== 138+0+0 (secure 0 0 0) 0x7f4c40079fa0 con 0x7f4c4415f870 2024-04-25T01:18:01.195 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_metadata' created 2024-04-25T01:18:01.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.215+0000 7f4c4c800640 1 -- 172.21.15.90:0/2457326960 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4c2806be60 msgr2=0x7f4c2806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:01.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.215+0000 7f4c4c800640 1 --2- 172.21.15.90:0/2457326960 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4c2806be60 0x7f4c2806e320 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4c38007fd0 tx=0x7f4c38009040 comp rx=0 tx=0).stop 2024-04-25T01:18:01.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.215+0000 7f4c4c800640 1 -- 172.21.15.90:0/2457326960 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c4415f870 msgr2=0x7f4c44143f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:01.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.215+0000 7f4c4c800640 1 --2- 172.21.15.90:0/2457326960 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c4415f870 0x7f4c44143f50 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f4c40009f70 tx=0x7f4c40004a40 comp rx=0 tx=0).stop 2024-04-25T01:18:01.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.215+0000 7f4c4c800640 1 -- 172.21.15.90:0/2457326960 shutdown_connections 2024-04-25T01:18:01.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.215+0000 7f4c4c800640 1 --2- 172.21.15.90:0/2457326960 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4c2806be60 0x7f4c2806e320 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:01.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.215+0000 7f4c4c800640 1 --2- 172.21.15.90:0/2457326960 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c4415f870 0x7f4c44143f50 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:01.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.215+0000 7f4c4c800640 1 --2- 172.21.15.90:0/2457326960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c44071110 0x7f4c4413bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:01.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.215+0000 7f4c4c800640 1 --2- 172.21.15.90:0/2457326960 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4c4406cb40 0x7f4c4415e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:01.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.215+0000 7f4c4c800640 1 -- 172.21.15.90:0/2457326960 >> 172.21.15.90:0/2457326960 conn(0x7f4c440cfae0 msgr2=0x7f4c4406e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:01.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.215+0000 7f4c4c800640 1 -- 172.21.15.90:0/2457326960 shutdown_connections 2024-04-25T01:18:01.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.215+0000 7f4c4c800640 1 -- 172.21.15.90:0/2457326960 wait complete. 2024-04-25T01:18:01.226 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T01:18:01.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.445+0000 7f8819a00640 1 Processor -- start 2024-04-25T01:18:01.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.445+0000 7f8819a00640 1 -- start start 2024-04-25T01:18:01.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.446+0000 7f8819a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f88140b4a20 0x7f881415f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:01.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.446+0000 7f8819a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f881406d700 0x7f881415fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:01.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.446+0000 7f8819a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8814160580 con 0x7f88140aee00 2024-04-25T01:18:01.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.446+0000 7f8819a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f881406c9d0 con 0x7f881406d700 2024-04-25T01:18:01.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.446+0000 7f8819a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f881406cc20 con 0x7f88140b4a20 2024-04-25T01:18:01.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.447+0000 7f8819000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f881406d700 0x7f881415fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:01.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.447+0000 7f8813e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f88140aee00 0x7f88140b4310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:54734/0 (socket says 172.21.15.90:54734) 2024-04-25T01:18:01.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.447+0000 7f8813e00640 1 -- 172.21.15.90:0/4050940884 learned_addr learned my addr 172.21.15.90:0/4050940884 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:01.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.447+0000 7f8813400640 1 --2- 172.21.15.90:0/4050940884 >> v2:172.21.15.103:3300/0 conn(0x7f88140b4a20 0x7f881415f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:01.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.447+0000 7f8819000640 1 -- 172.21.15.90:0/4050940884 >> v2:172.21.15.103:3300/0 conn(0x7f88140b4a20 msgr2=0x7f881415f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:01.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.447+0000 7f8819000640 1 --2- 172.21.15.90:0/4050940884 >> v2:172.21.15.103:3300/0 conn(0x7f88140b4a20 0x7f881415f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:01.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.447+0000 7f8819000640 1 -- 172.21.15.90:0/4050940884 >> v1:172.21.15.90:6789/0 conn(0x7f88140aee00 legacy=0x7f88140b4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:01.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.447+0000 7f8819000640 1 -- 172.21.15.90:0/4050940884 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f881406cf00 con 0x7f881406d700 2024-04-25T01:18:01.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.447+0000 7f8812a00640 1 -- 172.21.15.90:0/4050940884 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 887103909 0 0) 0x7f8814160580 con 0x7f88140aee00 2024-04-25T01:18:01.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.447+0000 7f8813400640 1 --2- 172.21.15.90:0/4050940884 >> v2:172.21.15.103:3300/0 conn(0x7f88140b4a20 0x7f881415f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:01.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.448+0000 7f8819000640 1 --2- 172.21.15.90:0/4050940884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f881406d700 0x7f881415fd50 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f8814072c80 tx=0x7f880c02a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a13368b0e771470f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:01.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.448+0000 7f8812a00640 1 -- 172.21.15.90:0/4050940884 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c037070 con 0x7f881406d700 2024-04-25T01:18:01.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.448+0000 7f8812a00640 1 -- 172.21.15.90:0/4050940884 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f880c02add0 con 0x7f881406d700 2024-04-25T01:18:01.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.448+0000 7f8812a00640 1 -- 172.21.15.90:0/4050940884 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c032660 con 0x7f881406d700 2024-04-25T01:18:01.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.449+0000 7f8819a00640 1 -- 172.21.15.90:0/4050940884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f881406d700 msgr2=0x7f881415fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:01.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.449+0000 7f8819a00640 1 --2- 172.21.15.90:0/4050940884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f881406d700 0x7f881415fd50 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f8814072c80 tx=0x7f880c02a080 comp rx=0 tx=0).stop 2024-04-25T01:18:01.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.449+0000 7f8819a00640 1 -- 172.21.15.90:0/4050940884 shutdown_connections 2024-04-25T01:18:01.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.449+0000 7f8819a00640 1 --2- 172.21.15.90:0/4050940884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f881406d700 0x7f881415fd50 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:01.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.449+0000 7f8819a00640 1 --2- 172.21.15.90:0/4050940884 >> v2:172.21.15.103:3300/0 conn(0x7f88140b4a20 0x7f881415f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:01.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.449+0000 7f8819a00640 1 -- 172.21.15.90:0/4050940884 >> 172.21.15.90:0/4050940884 conn(0x7f881406b120 msgr2=0x7f881406b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:01.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.449+0000 7f8819a00640 1 -- 172.21.15.90:0/4050940884 shutdown_connections 2024-04-25T01:18:01.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.449+0000 7f8819a00640 1 -- 172.21.15.90:0/4050940884 wait complete. 2024-04-25T01:18:01.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.451+0000 7f8819a00640 1 Processor -- start 2024-04-25T01:18:01.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.451+0000 7f8819a00640 1 -- start start 2024-04-25T01:18:01.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.452+0000 7f8819a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f881406d700 0x7f8814150f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:01.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.452+0000 7f8819a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f88140aee00 0x7f88141518f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:01.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.452+0000 7f8819a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88140b4a20 0x7f881414d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:01.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.452+0000 7f8819a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f88140af1e0 con 0x7f88140b4a20 2024-04-25T01:18:01.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.452+0000 7f8819a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8814160290 con 0x7f881406d700 2024-04-25T01:18:01.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.452+0000 7f8819a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f881415d370 con 0x7f88140aee00 2024-04-25T01:18:01.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.452+0000 7f8819000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88140b4a20 0x7f881414d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:01.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.452+0000 7f8819000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88140b4a20 0x7f881414d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57120/0 (socket says 172.21.15.90:57120) 2024-04-25T01:18:01.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.452+0000 7f8819000640 1 -- 172.21.15.90:0/3347101228 learned_addr learned my addr 172.21.15.90:0/3347101228 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:01.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.453+0000 7f8813400640 1 --2- 172.21.15.90:0/3347101228 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f88140aee00 0x7f88141518f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:01.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.453+0000 7f8819000640 1 -- 172.21.15.90:0/3347101228 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f88140aee00 msgr2=0x7f88141518f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:01.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.453+0000 7f8819000640 1 --2- 172.21.15.90:0/3347101228 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f88140aee00 0x7f88141518f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:01.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.453+0000 7f8819000640 1 -- 172.21.15.90:0/3347101228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f881406d700 msgr2=0x7f8814150f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:01.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.453+0000 7f8813e00640 1 --2- 172.21.15.90:0/3347101228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f881406d700 0x7f8814150f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:01.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.453+0000 7f8819000640 1 --2- 172.21.15.90:0/3347101228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f881406d700 0x7f8814150f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:01.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.453+0000 7f8819000640 1 -- 172.21.15.90:0/3347101228 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8814070d80 con 0x7f88140b4a20 2024-04-25T01:18:01.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.453+0000 7f8819000640 1 --2- 172.21.15.90:0/3347101228 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88140b4a20 0x7f881414d710 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f880c035040 tx=0x7f880c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:01.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.453+0000 7f880be00640 1 -- 172.21.15.90:0/3347101228 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c037050 con 0x7f88140b4a20 2024-04-25T01:18:01.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.453+0000 7f880be00640 1 -- 172.21.15.90:0/3347101228 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f880c038040 con 0x7f88140b4a20 2024-04-25T01:18:01.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.453+0000 7f880be00640 1 -- 172.21.15.90:0/3347101228 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c004450 con 0x7f88140b4a20 2024-04-25T01:18:01.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.454+0000 7f8819a00640 1 -- 172.21.15.90:0/3347101228 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8814152020 con 0x7f88140b4a20 2024-04-25T01:18:01.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.454+0000 7f8819a00640 1 -- 172.21.15.90:0/3347101228 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f881414dc50 con 0x7f88140b4a20 2024-04-25T01:18:01.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.455+0000 7f8819a00640 1 -- 172.21.15.90:0/3347101228 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88140cfa10 con 0x7f88140b4a20 2024-04-25T01:18:01.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.456+0000 7f880be00640 1 -- 172.21.15.90:0/3347101228 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f880c02ac90 con 0x7f88140b4a20 2024-04-25T01:18:01.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.457+0000 7f880be00640 1 --2- 172.21.15.90:0/3347101228 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f87f006be30 0x7f87f006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:01.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.457+0000 7f880be00640 1 -- 172.21.15.90:0/3347101228 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5619+0+0 (secure 0 0 0) 0x7f880c0ac170 con 0x7f88140b4a20 2024-04-25T01:18:01.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.458+0000 7f8813e00640 1 --2- 172.21.15.90:0/3347101228 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f87f006be30 0x7f87f006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:01.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.458+0000 7f8813e00640 1 --2- 172.21.15.90:0/3347101228 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f87f006be30 0x7f87f006e2f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f88040012e0 tx=0x7f8804009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:01.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.461+0000 7f880be00640 1 -- 172.21.15.90:0/3347101228 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f88140cfa10 con 0x7f88140b4a20 2024-04-25T01:18:01.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:01.660+0000 7f8819a00640 1 -- 172.21.15.90:0/3347101228 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f88140753c0 con 0x7f88140b4a20 2024-04-25T01:18:02.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.227+0000 7f880be00640 1 -- 172.21.15.90:0/3347101228 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v38) ==== 170+0+0 (secure 0 0 0) 0x7f880c07ac70 con 0x7f88140b4a20 2024-04-25T01:18:02.211 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data' created 2024-04-25T01:18:02.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.232+0000 7f8819a00640 1 -- 172.21.15.90:0/3347101228 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f87f006be30 msgr2=0x7f87f006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:02.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.232+0000 7f8819a00640 1 --2- 172.21.15.90:0/3347101228 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f87f006be30 0x7f87f006e2f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f88040012e0 tx=0x7f8804009040 comp rx=0 tx=0).stop 2024-04-25T01:18:02.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.232+0000 7f8819a00640 1 -- 172.21.15.90:0/3347101228 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88140b4a20 msgr2=0x7f881414d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:02.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.232+0000 7f8819a00640 1 --2- 172.21.15.90:0/3347101228 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88140b4a20 0x7f881414d710 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f880c035040 tx=0x7f880c0046c0 comp rx=0 tx=0).stop 2024-04-25T01:18:02.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.233+0000 7f8819a00640 1 -- 172.21.15.90:0/3347101228 shutdown_connections 2024-04-25T01:18:02.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.233+0000 7f8819a00640 1 --2- 172.21.15.90:0/3347101228 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f87f006be30 0x7f87f006e2f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:02.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.233+0000 7f8819a00640 1 --2- 172.21.15.90:0/3347101228 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88140b4a20 0x7f881414d710 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:02.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.233+0000 7f8819a00640 1 --2- 172.21.15.90:0/3347101228 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f88140aee00 0x7f88141518f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:02.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.233+0000 7f8819a00640 1 --2- 172.21.15.90:0/3347101228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f881406d700 0x7f8814150f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:02.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.233+0000 7f8819a00640 1 -- 172.21.15.90:0/3347101228 >> 172.21.15.90:0/3347101228 conn(0x7f881406b120 msgr2=0x7f881415c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:02.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.233+0000 7f8819a00640 1 -- 172.21.15.90:0/3347101228 shutdown_connections 2024-04-25T01:18:02.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.234+0000 7f8819a00640 1 -- 172.21.15.90:0/3347101228 wait complete. 2024-04-25T01:18:02.249 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T01:18:02.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.443+0000 7fdef7e00640 1 Processor -- start 2024-04-25T01:18:02.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.443+0000 7fdef7e00640 1 -- start start 2024-04-25T01:18:02.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.444+0000 7fdef7e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fdef015fc40 0x7fdef01600a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:02.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.444+0000 7fdef7e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdef006d110 con 0x7fdef00722e0 2024-04-25T01:18:02.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.444+0000 7fdef7e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdef006d3f0 con 0x7fdef00cedc0 2024-04-25T01:18:02.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.444+0000 7fdef7e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fdef0160aa0 con 0x7fdef015fc40 2024-04-25T01:18:02.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.445+0000 7fdef5400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fdef00722e0 0x7fdef006ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:54742/0 (socket says 172.21.15.90:54742) 2024-04-25T01:18:02.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.445+0000 7fdef5400640 1 -- 172.21.15.90:0/1122006398 learned_addr learned my addr 172.21.15.90:0/1122006398 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:02.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.445+0000 7fdef4a00640 1 --2- 172.21.15.90:0/1122006398 >> v2:172.21.15.103:3300/0 conn(0x7fdef015fc40 0x7fdef01600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:02.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.445+0000 7fdeefe00640 1 -- 172.21.15.90:0/1122006398 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 575759948 0 0) 0x7fdef006d110 con 0x7fdef00722e0 2024-04-25T01:18:02.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.445+0000 7fdeefe00640 1 -- 172.21.15.90:0/1122006398 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fded8003520 con 0x7fdef00722e0 2024-04-25T01:18:02.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.446+0000 7fdeefe00640 1 -- 172.21.15.90:0/1122006398 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4077837681 0 0) 0x7fded8003520 con 0x7fdef00722e0 2024-04-25T01:18:02.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.446+0000 7fdeefe00640 1 -- 172.21.15.90:0/1122006398 >> v2:172.21.15.103:3300/0 conn(0x7fdef015fc40 msgr2=0x7fdef01600a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:02.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.446+0000 7fdeefe00640 1 --2- 172.21.15.90:0/1122006398 >> v2:172.21.15.103:3300/0 conn(0x7fdef015fc40 0x7fdef01600a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:02.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.446+0000 7fdeefe00640 1 -- 172.21.15.90:0/1122006398 >> v1:172.21.15.103:6789/0 conn(0x7fdef00cedc0 legacy=0x7fdef00cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:02.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.446+0000 7fdeefe00640 1 -- 172.21.15.90:0/1122006398 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdef006d6d0 con 0x7fdef00722e0 2024-04-25T01:18:02.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.446+0000 7fdeefe00640 1 -- 172.21.15.90:0/1122006398 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7fdee4001670 con 0x7fdef00722e0 2024-04-25T01:18:02.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.446+0000 7fdeefe00640 1 -- 172.21.15.90:0/1122006398 >> v1:172.21.15.90:6789/0 conn(0x7fdef00722e0 legacy=0x7fdef006ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:02.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.446+0000 7fdeefe00640 1 --2- 172.21.15.90:0/1122006398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fded80039b0 0x7fded8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:02.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.446+0000 7fdeefe00640 1 --2- 172.21.15.90:0/1122006398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fded8004800 0x7fded8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:02.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.446+0000 7fdeefe00640 1 --2- 172.21.15.90:0/1122006398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fded80051c0 0x7fded800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:02.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.446+0000 7fdeefe00640 1 -- 172.21.15.90:0/1122006398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdef0160aa0 con 0x7fded8004800 2024-04-25T01:18:02.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.446+0000 7fdeefe00640 1 -- 172.21.15.90:0/1122006398 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fded800c160 con 0x7fded80039b0 2024-04-25T01:18:02.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.446+0000 7fdeefe00640 1 -- 172.21.15.90:0/1122006398 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fded800c360 con 0x7fded80051c0 2024-04-25T01:18:02.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.446+0000 7fdef4a00640 1 --2- 172.21.15.90:0/1122006398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fded8004800 0x7fded8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:02.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.446+0000 7fdef5400640 1 --2- 172.21.15.90:0/1122006398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fded80039b0 0x7fded8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:02.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.446+0000 7fdef5e00640 1 --2- 172.21.15.90:0/1122006398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fded80051c0 0x7fded800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:02.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.447+0000 7fdef4a00640 1 -- 172.21.15.90:0/1122006398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fded80051c0 msgr2=0x7fded800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:02.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.447+0000 7fdef4a00640 1 --2- 172.21.15.90:0/1122006398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fded80051c0 0x7fded800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:02.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.447+0000 7fdef4a00640 1 -- 172.21.15.90:0/1122006398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fded80039b0 msgr2=0x7fded8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:02.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.447+0000 7fdef4a00640 1 --2- 172.21.15.90:0/1122006398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fded80039b0 0x7fded8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:02.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.447+0000 7fdef4a00640 1 -- 172.21.15.90:0/1122006398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fded8003520 con 0x7fded8004800 2024-04-25T01:18:02.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.447+0000 7fdef5400640 1 --2- 172.21.15.90:0/1122006398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fded80039b0 0x7fded8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:02.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.447+0000 7fdef5e00640 1 -- 172.21.15.90:0/1122006398 reap_dead start 2024-04-25T01:18:02.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.448+0000 7fdef4a00640 1 --2- 172.21.15.90:0/1122006398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fded8004800 0x7fded8004c80 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fdedc009ac0 tx=0x7fded8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dce5eac3819b5a16 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:02.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.448+0000 7fdeefe00640 1 -- 172.21.15.90:0/1122006398 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdedc036070 con 0x7fded8004800 2024-04-25T01:18:02.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.448+0000 7fdeefe00640 1 -- 172.21.15.90:0/1122006398 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdedc031400 con 0x7fded8004800 2024-04-25T01:18:02.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.448+0000 7fdeefe00640 1 -- 172.21.15.90:0/1122006398 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdedc009690 con 0x7fded8004800 2024-04-25T01:18:02.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.448+0000 7fdef7e00640 1 -- 172.21.15.90:0/1122006398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fded8004800 msgr2=0x7fded8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:02.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.448+0000 7fdef7e00640 1 --2- 172.21.15.90:0/1122006398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fded8004800 0x7fded8004c80 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fdedc009ac0 tx=0x7fded8005830 comp rx=0 tx=0).stop 2024-04-25T01:18:02.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.449+0000 7fdef7e00640 1 -- 172.21.15.90:0/1122006398 shutdown_connections 2024-04-25T01:18:02.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.449+0000 7fdef7e00640 1 --2- 172.21.15.90:0/1122006398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fded8004800 0x7fded8004c80 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:02.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.449+0000 7fdef7e00640 1 -- 172.21.15.90:0/1122006398 >> 172.21.15.90:0/1122006398 conn(0x7fdef006b720 msgr2=0x7fdef00b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:02.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.449+0000 7fdef7e00640 1 -- 172.21.15.90:0/1122006398 shutdown_connections 2024-04-25T01:18:02.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.449+0000 7fdef7e00640 1 -- 172.21.15.90:0/1122006398 wait complete. 2024-04-25T01:18:02.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.450+0000 7fdef7e00640 1 Processor -- start 2024-04-25T01:18:02.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.451+0000 7fdef7e00640 1 -- start start 2024-04-25T01:18:02.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.451+0000 7fdef7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fded8004800 0x7fdef01582a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:02.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.451+0000 7fdef7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdef0159920 0x7fdef013f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:02.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.451+0000 7fdef7e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdef01505c0 0x7fdef01549e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:02.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.452+0000 7fdef7e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdef0160430 con 0x7fdef0159920 2024-04-25T01:18:02.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.452+0000 7fdef7e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdef01602b0 con 0x7fded8004800 2024-04-25T01:18:02.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.452+0000 7fdef7e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdef01605b0 con 0x7fdef01505c0 2024-04-25T01:18:02.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.452+0000 7fdef4a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdef0159920 0x7fdef013f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:02.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.452+0000 7fdef4a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdef0159920 0x7fdef013f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57142/0 (socket says 172.21.15.90:57142) 2024-04-25T01:18:02.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.452+0000 7fdef4a00640 1 -- 172.21.15.90:0/1048984756 learned_addr learned my addr 172.21.15.90:0/1048984756 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:02.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.452+0000 7fdef5e00640 1 --2- 172.21.15.90:0/1048984756 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdef01505c0 0x7fdef01549e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:02.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.452+0000 7fdef5400640 1 --2- 172.21.15.90:0/1048984756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fded8004800 0x7fdef01582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:02.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.452+0000 7fdef4a00640 1 -- 172.21.15.90:0/1048984756 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdef01505c0 msgr2=0x7fdef01549e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:02.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.452+0000 7fdef4a00640 1 --2- 172.21.15.90:0/1048984756 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdef01505c0 0x7fdef01549e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:02.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.452+0000 7fdef4a00640 1 -- 172.21.15.90:0/1048984756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fded8004800 msgr2=0x7fdef01582a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:02.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.452+0000 7fdef4a00640 1 --2- 172.21.15.90:0/1048984756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fded8004800 0x7fdef01582a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:02.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.452+0000 7fdef4a00640 1 -- 172.21.15.90:0/1048984756 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdef0158880 con 0x7fdef0159920 2024-04-25T01:18:02.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.453+0000 7fdef4a00640 1 --2- 172.21.15.90:0/1048984756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdef0159920 0x7fdef013f460 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fdedc034040 tx=0x7fdedc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:02.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.453+0000 7fdeed600640 1 -- 172.21.15.90:0/1048984756 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdedc03f020 con 0x7fdef0159920 2024-04-25T01:18:02.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.453+0000 7fdeed600640 1 -- 172.21.15.90:0/1048984756 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdedc037040 con 0x7fdef0159920 2024-04-25T01:18:02.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.453+0000 7fdef7e00640 1 -- 172.21.15.90:0/1048984756 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdef0158b60 con 0x7fdef0159920 2024-04-25T01:18:02.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.453+0000 7fdeed600640 1 -- 172.21.15.90:0/1048984756 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdedc036070 con 0x7fdef0159920 2024-04-25T01:18:02.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.453+0000 7fdef7e00640 1 -- 172.21.15.90:0/1048984756 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdef0159120 con 0x7fdef0159920 2024-04-25T01:18:02.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.454+0000 7fdef7e00640 1 -- 172.21.15.90:0/1048984756 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdef0068fb0 con 0x7fdef0159920 2024-04-25T01:18:02.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.455+0000 7fdeed600640 1 -- 172.21.15.90:0/1048984756 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fdedc042020 con 0x7fdef0159920 2024-04-25T01:18:02.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.456+0000 7fdeed600640 1 --2- 172.21.15.90:0/1048984756 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fded006be30 0x7fded006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:02.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.456+0000 7fdeed600640 1 -- 172.21.15.90:0/1048984756 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6001+0+0 (secure 0 0 0) 0x7fdedc0ab120 con 0x7fdef0159920 2024-04-25T01:18:02.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.456+0000 7fdef5400640 1 --2- 172.21.15.90:0/1048984756 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fded006be30 0x7fded006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:02.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.457+0000 7fdef5400640 1 --2- 172.21.15.90:0/1048984756 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fded006be30 0x7fded006e2f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fdef0068830 tx=0x7fdee4000ef0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:02.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.461+0000 7fdeed600640 1 -- 172.21.15.90:0/1048984756 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fdedc049030 con 0x7fdef0159920 2024-04-25T01:18:02.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:02.699+0000 7fdef7e00640 1 -- 172.21.15.90:0/1048984756 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fdef015f620 con 0x7fdef0159920 2024-04-25T01:18:03.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.246+0000 7fdeed600640 1 -- 172.21.15.90:0/1048984756 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v14) ==== 172+0+0 (secure 0 0 0) 0x7fdedc079900 con 0x7fdef0159920 2024-04-25T01:18:03.230 INFO:teuthology.orchestra.run.smithi090.stderr:new fs with metadata pool 5 and data pool 6 2024-04-25T01:18:03.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.252+0000 7fdef7e00640 1 -- 172.21.15.90:0/1048984756 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fded006be30 msgr2=0x7fded006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:03.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.252+0000 7fdef7e00640 1 --2- 172.21.15.90:0/1048984756 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fded006be30 0x7fded006e2f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fdef0068830 tx=0x7fdee4000ef0 comp rx=0 tx=0).stop 2024-04-25T01:18:03.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.252+0000 7fdef7e00640 1 -- 172.21.15.90:0/1048984756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdef0159920 msgr2=0x7fdef013f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:03.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.252+0000 7fdef7e00640 1 --2- 172.21.15.90:0/1048984756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdef0159920 0x7fdef013f460 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fdedc034040 tx=0x7fdedc004730 comp rx=0 tx=0).stop 2024-04-25T01:18:03.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.255+0000 7fdef7e00640 1 -- 172.21.15.90:0/1048984756 shutdown_connections 2024-04-25T01:18:03.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.255+0000 7fdef7e00640 1 --2- 172.21.15.90:0/1048984756 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fded006be30 0x7fded006e2f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:03.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.255+0000 7fdef7e00640 1 --2- 172.21.15.90:0/1048984756 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdef01505c0 0x7fdef01549e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:03.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.255+0000 7fdef7e00640 1 --2- 172.21.15.90:0/1048984756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdef0159920 0x7fdef013f460 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:03.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.255+0000 7fdef7e00640 1 --2- 172.21.15.90:0/1048984756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fded8004800 0x7fdef01582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:03.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.255+0000 7fdef7e00640 1 -- 172.21.15.90:0/1048984756 >> 172.21.15.90:0/1048984756 conn(0x7fdef006b720 msgr2=0x7fdef015d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:03.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.258+0000 7fdef7e00640 1 -- 172.21.15.90:0/1048984756 shutdown_connections 2024-04-25T01:18:03.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.259+0000 7fdef7e00640 1 -- 172.21.15.90:0/1048984756 wait complete. 2024-04-25T01:18:03.273 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:18:03.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.458+0000 7f7f91200640 1 Processor -- start 2024-04-25T01:18:03.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.458+0000 7f7f91200640 1 -- start start 2024-04-25T01:18:03.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.460+0000 7f7f91200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7f8c0b50e0 0x7f7f8c161010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:03.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.460+0000 7f7f91200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f8c0d0ca0 0x7f7f8c161550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:03.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.460+0000 7f7f91200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f8c0cf3e0 con 0x7f7f8c0d08c0 2024-04-25T01:18:03.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.460+0000 7f7f91200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f7f8c0d2e80 con 0x7f7f8c0b50e0 2024-04-25T01:18:03.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.460+0000 7f7f91200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7f8c0cf6c0 con 0x7f7f8c0d0ca0 2024-04-25T01:18:03.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.460+0000 7f7f8aa00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7f8c0b50e0 0x7f7f8c161010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:03.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.461+0000 7f7f8aa00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7f8c0b50e0 0x7f7f8c161010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.90:36308/0 (socket says 172.21.15.90:36308) 2024-04-25T01:18:03.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.461+0000 7f7f8aa00640 1 -- 172.21.15.90:0/131196198 learned_addr learned my addr 172.21.15.90:0/131196198 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:03.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.461+0000 7f7f8b400640 1 --1- 172.21.15.90:0/131196198 >> v1:172.21.15.103:6789/0 conn(0x7f7f8c0d08c0 0x7f7f8c0ce990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.90:38334/0 (socket says 172.21.15.90:38334) 2024-04-25T01:18:03.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.461+0000 7f7f8be00640 1 --2- 172.21.15.90:0/131196198 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f8c0d0ca0 0x7f7f8c161550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:03.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.462+0000 7f7f8aa00640 1 -- 172.21.15.90:0/131196198 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f8c0d0ca0 msgr2=0x7f7f8c161550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:03.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.462+0000 7f7f8aa00640 1 --2- 172.21.15.90:0/131196198 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f8c0d0ca0 0x7f7f8c161550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:03.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.462+0000 7f7f8aa00640 1 -- 172.21.15.90:0/131196198 >> v1:172.21.15.103:6789/0 conn(0x7f7f8c0d08c0 legacy=0x7f7f8c0ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:03.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.462+0000 7f7f8aa00640 1 -- 172.21.15.90:0/131196198 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f8c0cf9a0 con 0x7f7f8c0b50e0 2024-04-25T01:18:03.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.462+0000 7f7f8be00640 1 --2- 172.21.15.90:0/131196198 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f8c0d0ca0 0x7f7f8c161550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:18:03.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.462+0000 7f7f8a000640 1 -- 172.21.15.90:0/131196198 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1593215014 0 0) 0x7f7f8c0cf3e0 con 0x7f7f8c0d08c0 2024-04-25T01:18:03.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.463+0000 7f7f8aa00640 1 --2- 172.21.15.90:0/131196198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f8c0b50e0 0x7f7f8c161010 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7f78009870 tx=0x7f7f7802a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f396db449bd33b4c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:03.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.464+0000 7f7f8a000640 1 -- 172.21.15.90:0/131196198 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f78037070 con 0x7f7f8c0b50e0 2024-04-25T01:18:03.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.464+0000 7f7f8a000640 1 -- 172.21.15.90:0/131196198 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f78032440 con 0x7f7f8c0b50e0 2024-04-25T01:18:03.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.464+0000 7f7f8a000640 1 -- 172.21.15.90:0/131196198 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f78032810 con 0x7f7f8c0b50e0 2024-04-25T01:18:03.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.464+0000 7f7f91200640 1 -- 172.21.15.90:0/131196198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f8c0b50e0 msgr2=0x7f7f8c161010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:03.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.464+0000 7f7f91200640 1 --2- 172.21.15.90:0/131196198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f8c0b50e0 0x7f7f8c161010 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7f78009870 tx=0x7f7f7802a250 comp rx=0 tx=0).stop 2024-04-25T01:18:03.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.464+0000 7f7f91200640 1 -- 172.21.15.90:0/131196198 shutdown_connections 2024-04-25T01:18:03.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.464+0000 7f7f91200640 1 --2- 172.21.15.90:0/131196198 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f8c0d0ca0 0x7f7f8c161550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:03.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.464+0000 7f7f91200640 1 --2- 172.21.15.90:0/131196198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f8c0b50e0 0x7f7f8c161010 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:03.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.464+0000 7f7f91200640 1 -- 172.21.15.90:0/131196198 >> 172.21.15.90:0/131196198 conn(0x7f7f8c09d070 msgr2=0x7f7f8c0d3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:03.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.464+0000 7f7f91200640 1 -- 172.21.15.90:0/131196198 shutdown_connections 2024-04-25T01:18:03.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.464+0000 7f7f91200640 1 -- 172.21.15.90:0/131196198 wait complete. 2024-04-25T01:18:03.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.466+0000 7f7f91200640 1 Processor -- start 2024-04-25T01:18:03.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.466+0000 7f7f91200640 1 -- start start 2024-04-25T01:18:03.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.467+0000 7f7f91200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f8c0b50e0 0x7f7f8c149dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:03.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.467+0000 7f7f91200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f8c0d08c0 0x7f7f8c15a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:03.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.467+0000 7f7f91200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f8c0d0ca0 0x7f7f8c15a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:03.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.467+0000 7f7f91200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7f8c161a90 con 0x7f7f8c0d08c0 2024-04-25T01:18:03.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.467+0000 7f7f91200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7f8c0d1950 con 0x7f7f8c0b50e0 2024-04-25T01:18:03.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.467+0000 7f7f91200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7f8c0cf0a0 con 0x7f7f8c0d0ca0 2024-04-25T01:18:03.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.467+0000 7f7f8aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f8c0d08c0 0x7f7f8c15a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:03.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.467+0000 7f7f8aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f8c0d08c0 0x7f7f8c15a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57148/0 (socket says 172.21.15.90:57148) 2024-04-25T01:18:03.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.467+0000 7f7f8aa00640 1 -- 172.21.15.90:0/3661348064 learned_addr learned my addr 172.21.15.90:0/3661348064 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:03.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.468+0000 7f7f8b400640 1 --2- 172.21.15.90:0/3661348064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f8c0b50e0 0x7f7f8c149dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:03.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.468+0000 7f7f8be00640 1 --2- 172.21.15.90:0/3661348064 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f8c0d0ca0 0x7f7f8c15a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:03.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.468+0000 7f7f8aa00640 1 -- 172.21.15.90:0/3661348064 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f8c0d0ca0 msgr2=0x7f7f8c15a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:03.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.468+0000 7f7f8aa00640 1 --2- 172.21.15.90:0/3661348064 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f8c0d0ca0 0x7f7f8c15a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:03.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.468+0000 7f7f8aa00640 1 -- 172.21.15.90:0/3661348064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f8c0b50e0 msgr2=0x7f7f8c149dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:03.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.468+0000 7f7f8aa00640 1 --2- 172.21.15.90:0/3661348064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f8c0b50e0 0x7f7f8c149dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:03.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.468+0000 7f7f8aa00640 1 -- 172.21.15.90:0/3661348064 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f8c0c95c0 con 0x7f7f8c0d08c0 2024-04-25T01:18:03.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.468+0000 7f7f8b400640 1 --2- 172.21.15.90:0/3661348064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f8c0b50e0 0x7f7f8c149dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:03.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.468+0000 7f7f8aa00640 1 --2- 172.21.15.90:0/3661348064 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f8c0d08c0 0x7f7f8c15a460 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f7f78035040 tx=0x7f7f780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:03.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.468+0000 7f7f83400640 1 -- 172.21.15.90:0/3661348064 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f78037040 con 0x7f7f8c0d08c0 2024-04-25T01:18:03.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.469+0000 7f7f83400640 1 -- 172.21.15.90:0/3661348064 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f78004230 con 0x7f7f8c0d08c0 2024-04-25T01:18:03.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.469+0000 7f7f91200640 1 -- 172.21.15.90:0/3661348064 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f8c0c98a0 con 0x7f7f8c0d08c0 2024-04-25T01:18:03.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.469+0000 7f7f83400640 1 -- 172.21.15.90:0/3661348064 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f78003610 con 0x7f7f8c0d08c0 2024-04-25T01:18:03.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.469+0000 7f7f91200640 1 -- 172.21.15.90:0/3661348064 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f8c0c9e60 con 0x7f7f8c0d08c0 2024-04-25T01:18:03.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.470+0000 7f7f91200640 1 -- 172.21.15.90:0/3661348064 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f8c068890 con 0x7f7f8c0d08c0 2024-04-25T01:18:03.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.471+0000 7f7f83400640 1 -- 172.21.15.90:0/3661348064 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7f780037b0 con 0x7f7f8c0d08c0 2024-04-25T01:18:03.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.472+0000 7f7f83400640 1 --2- 172.21.15.90:0/3661348064 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7f6c06be30 0x7f7f6c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:03.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.472+0000 7f7f83400640 1 -- 172.21.15.90:0/3661348064 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f7f780abde0 con 0x7f7f8c0d08c0 2024-04-25T01:18:03.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.472+0000 7f7f8b400640 1 --2- 172.21.15.90:0/3661348064 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7f6c06be30 0x7f7f6c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:03.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.473+0000 7f7f8b400640 1 --2- 172.21.15.90:0/3661348064 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7f6c06be30 0x7f7f6c06e2f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c000f60 tx=0x7f7f7c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:03.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.477+0000 7f7f83400640 1 -- 172.21.15.90:0/3661348064 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7f7807a560 con 0x7f7f8c0d08c0 2024-04-25T01:18:03.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.666+0000 7f7f91200640 1 -- 172.21.15.90:0/3661348064 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7f8c075350 con 0x7f7f8c0d08c0 2024-04-25T01:18:03.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.669+0000 7f7f83400640 1 -- 172.21.15.90:0/3661348064 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+16919 (secure 0 0 0) 0x7f7f78079e60 con 0x7f7f8c0d08c0 2024-04-25T01:18:03.653 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:03.653 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":39,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:18:02.700760+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:18:00.504992+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T01:18:01.714313+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:18:03.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.671+0000 7f7f91200640 1 -- 172.21.15.90:0/3661348064 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7f6c06be30 msgr2=0x7f7f6c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:03.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.671+0000 7f7f91200640 1 --2- 172.21.15.90:0/3661348064 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7f6c06be30 0x7f7f6c06e2f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c000f60 tx=0x7f7f7c009040 comp rx=0 tx=0).stop 2024-04-25T01:18:03.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.671+0000 7f7f91200640 1 -- 172.21.15.90:0/3661348064 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f8c0d08c0 msgr2=0x7f7f8c15a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:03.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.671+0000 7f7f91200640 1 --2- 172.21.15.90:0/3661348064 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f8c0d08c0 0x7f7f8c15a460 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f7f78035040 tx=0x7f7f780046c0 comp rx=0 tx=0).stop 2024-04-25T01:18:03.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.672+0000 7f7f91200640 1 -- 172.21.15.90:0/3661348064 shutdown_connections 2024-04-25T01:18:03.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.672+0000 7f7f91200640 1 --2- 172.21.15.90:0/3661348064 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7f6c06be30 0x7f7f6c06e2f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:03.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.672+0000 7f7f91200640 1 --2- 172.21.15.90:0/3661348064 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7f8c0d0ca0 0x7f7f8c15a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:03.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.672+0000 7f7f91200640 1 --2- 172.21.15.90:0/3661348064 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f8c0d08c0 0x7f7f8c15a460 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:03.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.672+0000 7f7f91200640 1 --2- 172.21.15.90:0/3661348064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f8c0b50e0 0x7f7f8c149dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:03.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.672+0000 7f7f91200640 1 -- 172.21.15.90:0/3661348064 >> 172.21.15.90:0/3661348064 conn(0x7f7f8c09d070 msgr2=0x7f7f8c0d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:03.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.672+0000 7f7f91200640 1 -- 172.21.15.90:0/3661348064 shutdown_connections 2024-04-25T01:18:03.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.672+0000 7f7f91200640 1 -- 172.21.15.90:0/3661348064 wait complete. 2024-04-25T01:18:03.673 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:18:03.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.879+0000 7fb116400640 1 Processor -- start 2024-04-25T01:18:03.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.879+0000 7fb116400640 1 -- start start 2024-04-25T01:18:03.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.880+0000 7fb116400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb11006b120 0x7fb11006b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:03.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.880+0000 7fb116400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb11015ca10 0x7fb11015ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:03.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.880+0000 7fb116400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb11006ba40 con 0x7fb1100ae730 2024-04-25T01:18:03.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.880+0000 7fb116400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fb1100cef40 con 0x7fb11015ca10 2024-04-25T01:18:03.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.880+0000 7fb116400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb11015c580 con 0x7fb11006b120 2024-04-25T01:18:03.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.881+0000 7fb115a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb11015ca10 0x7fb11015ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:03.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.881+0000 7fb115a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb11015ca10 0x7fb11015ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57164/0 (socket says 172.21.15.90:57164) 2024-04-25T01:18:03.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.881+0000 7fb115a00640 1 -- 172.21.15.90:0/3999385279 learned_addr learned my addr 172.21.15.90:0/3999385279 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:03.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.881+0000 7fb115000640 1 --2- 172.21.15.90:0/3999385279 >> v2:172.21.15.103:3300/0 conn(0x7fb11006b120 0x7fb11006b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:03.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.881+0000 7fb115a00640 1 -- 172.21.15.90:0/3999385279 >> v2:172.21.15.103:3300/0 conn(0x7fb11006b120 msgr2=0x7fb11006b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:03.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.881+0000 7fb115a00640 1 --2- 172.21.15.90:0/3999385279 >> v2:172.21.15.103:3300/0 conn(0x7fb11006b120 0x7fb11006b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:03.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.881+0000 7fb115a00640 1 -- 172.21.15.90:0/3999385279 >> v1:172.21.15.103:6789/0 conn(0x7fb1100ae730 legacy=0x7fb1100cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:03.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.881+0000 7fb115a00640 1 -- 172.21.15.90:0/3999385279 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb11015c220 con 0x7fb11015ca10 2024-04-25T01:18:03.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.882+0000 7fb115a00640 1 --2- 172.21.15.90:0/3999385279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb11015ca10 0x7fb11015ce80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb1100b2560 tx=0x7fb10802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb30b4078837b476 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:03.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.883+0000 7fb10f400640 1 -- 172.21.15.90:0/3999385279 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb108036070 con 0x7fb11015ca10 2024-04-25T01:18:03.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.883+0000 7fb10f400640 1 -- 172.21.15.90:0/3999385279 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb108031600 con 0x7fb11015ca10 2024-04-25T01:18:03.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.883+0000 7fb10f400640 1 -- 172.21.15.90:0/3999385279 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1080319d0 con 0x7fb11015ca10 2024-04-25T01:18:03.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.883+0000 7fb116400640 1 -- 172.21.15.90:0/3999385279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb11015ca10 msgr2=0x7fb11015ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:03.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.883+0000 7fb116400640 1 --2- 172.21.15.90:0/3999385279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb11015ca10 0x7fb11015ce80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb1100b2560 tx=0x7fb10802a6f0 comp rx=0 tx=0).stop 2024-04-25T01:18:03.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.883+0000 7fb116400640 1 -- 172.21.15.90:0/3999385279 shutdown_connections 2024-04-25T01:18:03.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.883+0000 7fb116400640 1 --2- 172.21.15.90:0/3999385279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb11015ca10 0x7fb11015ce80 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:03.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.883+0000 7fb116400640 1 --2- 172.21.15.90:0/3999385279 >> v2:172.21.15.103:3300/0 conn(0x7fb11006b120 0x7fb11006b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:03.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.883+0000 7fb116400640 1 -- 172.21.15.90:0/3999385279 >> 172.21.15.90:0/3999385279 conn(0x7fb1100b1660 msgr2=0x7fb1100b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:03.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.883+0000 7fb116400640 1 -- 172.21.15.90:0/3999385279 shutdown_connections 2024-04-25T01:18:03.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.883+0000 7fb116400640 1 -- 172.21.15.90:0/3999385279 wait complete. 2024-04-25T01:18:03.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.885+0000 7fb116400640 1 Processor -- start 2024-04-25T01:18:03.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.885+0000 7fb116400640 1 -- start start 2024-04-25T01:18:03.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.886+0000 7fb116400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb11006b120 0x7fb110146470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:03.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.886+0000 7fb116400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1100ae730 0x7fb1100b4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:03.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.886+0000 7fb116400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb11015ca10 0x7fb1100ad910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:03.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.886+0000 7fb116400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb1100af240 con 0x7fb11015ca10 2024-04-25T01:18:03.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.886+0000 7fb116400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb1100aef60 con 0x7fb1100ae730 2024-04-25T01:18:03.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.886+0000 7fb116400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb110153080 con 0x7fb11006b120 2024-04-25T01:18:03.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.887+0000 7fb115a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb11015ca10 0x7fb1100ad910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:03.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.887+0000 7fb115a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb11015ca10 0x7fb1100ad910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57172/0 (socket says 172.21.15.90:57172) 2024-04-25T01:18:03.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.887+0000 7fb115a00640 1 -- 172.21.15.90:0/662839488 learned_addr learned my addr 172.21.15.90:0/662839488 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:03.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.887+0000 7fb10fe00640 1 --2- 172.21.15.90:0/662839488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1100ae730 0x7fb1100b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:03.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.887+0000 7fb115000640 1 --2- 172.21.15.90:0/662839488 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb11006b120 0x7fb110146470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:03.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.887+0000 7fb115a00640 1 -- 172.21.15.90:0/662839488 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb11006b120 msgr2=0x7fb110146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:03.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.887+0000 7fb115a00640 1 --2- 172.21.15.90:0/662839488 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb11006b120 0x7fb110146470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:03.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.887+0000 7fb115a00640 1 -- 172.21.15.90:0/662839488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1100ae730 msgr2=0x7fb1100b4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:03.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.887+0000 7fb115a00640 1 --2- 172.21.15.90:0/662839488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1100ae730 0x7fb1100b4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:03.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.887+0000 7fb115a00640 1 -- 172.21.15.90:0/662839488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1100b3360 con 0x7fb11015ca10 2024-04-25T01:18:03.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.887+0000 7fb10fe00640 1 --2- 172.21.15.90:0/662839488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1100ae730 0x7fb1100b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:03.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.887+0000 7fb115a00640 1 --2- 172.21.15.90:0/662839488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb11015ca10 0x7fb1100ad910 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb108009f40 tx=0x7fb108004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:03.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.888+0000 7fb10cc00640 1 -- 172.21.15.90:0/662839488 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb10803f020 con 0x7fb11015ca10 2024-04-25T01:18:03.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.888+0000 7fb10cc00640 1 -- 172.21.15.90:0/662839488 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb108037040 con 0x7fb11015ca10 2024-04-25T01:18:03.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.888+0000 7fb10cc00640 1 -- 172.21.15.90:0/662839488 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb108036070 con 0x7fb11015ca10 2024-04-25T01:18:03.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.888+0000 7fb116400640 1 -- 172.21.15.90:0/662839488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1100b3640 con 0x7fb11015ca10 2024-04-25T01:18:03.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.888+0000 7fb116400640 1 -- 172.21.15.90:0/662839488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1100b3c00 con 0x7fb11015ca10 2024-04-25T01:18:03.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.889+0000 7fb116400640 1 -- 172.21.15.90:0/662839488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb110075350 con 0x7fb11015ca10 2024-04-25T01:18:03.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.890+0000 7fb10cc00640 1 -- 172.21.15.90:0/662839488 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb108042020 con 0x7fb11015ca10 2024-04-25T01:18:03.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.891+0000 7fb10cc00640 1 --2- 172.21.15.90:0/662839488 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb0f006be30 0x7fb0f006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:03.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.891+0000 7fb10cc00640 1 -- 172.21.15.90:0/662839488 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fb1080ab590 con 0x7fb11015ca10 2024-04-25T01:18:03.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.891+0000 7fb115000640 1 --2- 172.21.15.90:0/662839488 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb0f006be30 0x7fb0f006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:03.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.891+0000 7fb115000640 1 --2- 172.21.15.90:0/662839488 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb0f006be30 0x7fb0f006e2f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc006fd0 tx=0x7fb0fc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:03.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:03.895+0000 7fb10cc00640 1 -- 172.21.15.90:0/662839488 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb108049030 con 0x7fb11015ca10 2024-04-25T01:18:04.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.096+0000 7fb116400640 1 -- 172.21.15.90:0/662839488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb11006ccd0 con 0x7fb11015ca10 2024-04-25T01:18:04.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.098+0000 7fb10cc00640 1 -- 172.21.15.90:0/662839488 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+16919 (secure 0 0 0) 0x7fb108079d10 con 0x7fb11015ca10 2024-04-25T01:18:04.082 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:04.082 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":39,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:18:02.700760+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:18:00.504992+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T01:18:01.714313+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:18:04.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.100+0000 7fb116400640 1 -- 172.21.15.90:0/662839488 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb0f006be30 msgr2=0x7fb0f006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:04.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.100+0000 7fb116400640 1 --2- 172.21.15.90:0/662839488 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb0f006be30 0x7fb0f006e2f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc006fd0 tx=0x7fb0fc008040 comp rx=0 tx=0).stop 2024-04-25T01:18:04.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.101+0000 7fb116400640 1 -- 172.21.15.90:0/662839488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb11015ca10 msgr2=0x7fb1100ad910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:04.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.101+0000 7fb116400640 1 --2- 172.21.15.90:0/662839488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb11015ca10 0x7fb1100ad910 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb108009f40 tx=0x7fb108004a40 comp rx=0 tx=0).stop 2024-04-25T01:18:04.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.101+0000 7fb116400640 1 -- 172.21.15.90:0/662839488 shutdown_connections 2024-04-25T01:18:04.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.101+0000 7fb116400640 1 --2- 172.21.15.90:0/662839488 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb0f006be30 0x7fb0f006e2f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:04.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.101+0000 7fb116400640 1 --2- 172.21.15.90:0/662839488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb11015ca10 0x7fb1100ad910 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:04.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.101+0000 7fb116400640 1 --2- 172.21.15.90:0/662839488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1100ae730 0x7fb1100b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:04.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.101+0000 7fb116400640 1 --2- 172.21.15.90:0/662839488 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb11006b120 0x7fb110146470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:04.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.101+0000 7fb116400640 1 -- 172.21.15.90:0/662839488 >> 172.21.15.90:0/662839488 conn(0x7fb1100b1660 msgr2=0x7fb1100af5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:04.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.101+0000 7fb116400640 1 -- 172.21.15.90:0/662839488 shutdown_connections 2024-04-25T01:18:04.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.101+0000 7fb116400640 1 -- 172.21.15.90:0/662839488 wait complete. 2024-04-25T01:18:04.102 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T01:18:04.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.323+0000 7f5e97e00640 1 Processor -- start 2024-04-25T01:18:04.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.323+0000 7f5e97e00640 1 -- start start 2024-04-25T01:18:04.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.325+0000 7f5e97e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e90158a60 0x7f5e9006df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:04.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.325+0000 7f5e97e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e9006d140 con 0x7f5e90158680 2024-04-25T01:18:04.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.325+0000 7f5e97e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e9006d420 con 0x7f5e9015f4c0 2024-04-25T01:18:04.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.325+0000 7f5e97e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5e9006d700 con 0x7f5e90158a60 2024-04-25T01:18:04.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.325+0000 7f5e95400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f5e90158680 0x7f5e9006e6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:54758/0 (socket says 172.21.15.90:54758) 2024-04-25T01:18:04.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.325+0000 7f5e95400640 1 -- 172.21.15.90:0/1486251431 learned_addr learned my addr 172.21.15.90:0/1486251431 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:04.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.325+0000 7f5e8fe00640 1 -- 172.21.15.90:0/1486251431 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3988195654 0 0) 0x7f5e9006d140 con 0x7f5e90158680 2024-04-25T01:18:04.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.325+0000 7f5e8fe00640 1 -- 172.21.15.90:0/1486251431 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e78003520 con 0x7f5e90158680 2024-04-25T01:18:04.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.326+0000 7f5e94a00640 1 --2- 172.21.15.90:0/1486251431 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e90158a60 0x7f5e9006df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:04.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.326+0000 7f5e8fe00640 1 -- 172.21.15.90:0/1486251431 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1811782642 0 0) 0x7f5e78003520 con 0x7f5e90158680 2024-04-25T01:18:04.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.326+0000 7f5e8fe00640 1 -- 172.21.15.90:0/1486251431 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e90158a60 msgr2=0x7f5e9006df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:04.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.326+0000 7f5e8fe00640 1 --2- 172.21.15.90:0/1486251431 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e90158a60 0x7f5e9006df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:04.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.326+0000 7f5e8fe00640 1 -- 172.21.15.90:0/1486251431 >> v1:172.21.15.103:6789/0 conn(0x7f5e9015f4c0 legacy=0x7f5e900b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:04.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.326+0000 7f5e8fe00640 1 -- 172.21.15.90:0/1486251431 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e9006d9e0 con 0x7f5e90158680 2024-04-25T01:18:04.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.326+0000 7f5e8fe00640 1 -- 172.21.15.90:0/1486251431 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f5e84001670 con 0x7f5e90158680 2024-04-25T01:18:04.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.326+0000 7f5e8fe00640 1 -- 172.21.15.90:0/1486251431 >> v1:172.21.15.90:6789/0 conn(0x7f5e90158680 legacy=0x7f5e9006e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:04.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.326+0000 7f5e8fe00640 1 --2- 172.21.15.90:0/1486251431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e780039e0 0x7f5e78003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:04.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.326+0000 7f5e8fe00640 1 --2- 172.21.15.90:0/1486251431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e78004800 0x7f5e78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:04.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.326+0000 7f5e8fe00640 1 --2- 172.21.15.90:0/1486251431 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e90158a60 0x7f5e7800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:04.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.326+0000 7f5e8fe00640 1 -- 172.21.15.90:0/1486251431 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9006d700 con 0x7f5e780039e0 2024-04-25T01:18:04.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.326+0000 7f5e8fe00640 1 -- 172.21.15.90:0/1486251431 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5e780054f0 con 0x7f5e78004800 2024-04-25T01:18:04.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.326+0000 7f5e8fe00640 1 -- 172.21.15.90:0/1486251431 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5e7800bf00 con 0x7f5e90158a60 2024-04-25T01:18:04.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.326+0000 7f5e95400640 1 --2- 172.21.15.90:0/1486251431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e780039e0 0x7f5e78003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:04.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.327+0000 7f5e94a00640 1 --2- 172.21.15.90:0/1486251431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e78004800 0x7f5e78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:04.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.327+0000 7f5e95400640 1 -- 172.21.15.90:0/1486251431 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e90158a60 msgr2=0x7f5e7800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:04.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.327+0000 7f5e95400640 1 --2- 172.21.15.90:0/1486251431 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e90158a60 0x7f5e7800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:04.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.327+0000 7f5e95400640 1 -- 172.21.15.90:0/1486251431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e78004800 msgr2=0x7f5e78004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:04.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.327+0000 7f5e95400640 1 --2- 172.21.15.90:0/1486251431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e78004800 0x7f5e78004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:04.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.327+0000 7f5e95400640 1 -- 172.21.15.90:0/1486251431 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e78003520 con 0x7f5e780039e0 2024-04-25T01:18:04.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.328+0000 7f5e95400640 1 --2- 172.21.15.90:0/1486251431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e780039e0 0x7f5e78003e00 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f5e8400c0e0 tx=0x7f5e78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baa6a579badc0fea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:04.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.328+0000 7f5e8fe00640 1 -- 172.21.15.90:0/1486251431 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e84007c20 con 0x7f5e780039e0 2024-04-25T01:18:04.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.328+0000 7f5e8fe00640 1 -- 172.21.15.90:0/1486251431 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e84005af0 con 0x7f5e780039e0 2024-04-25T01:18:04.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.328+0000 7f5e8fe00640 1 -- 172.21.15.90:0/1486251431 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e84007070 con 0x7f5e780039e0 2024-04-25T01:18:04.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.329+0000 7f5e97e00640 1 -- 172.21.15.90:0/1486251431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e780039e0 msgr2=0x7f5e78003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:04.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.329+0000 7f5e97e00640 1 --2- 172.21.15.90:0/1486251431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e780039e0 0x7f5e78003e00 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f5e8400c0e0 tx=0x7f5e78005830 comp rx=0 tx=0).stop 2024-04-25T01:18:04.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.329+0000 7f5e95e00640 1 -- 172.21.15.90:0/1486251431 reap_dead start 2024-04-25T01:18:04.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.329+0000 7f5e97e00640 1 -- 172.21.15.90:0/1486251431 shutdown_connections 2024-04-25T01:18:04.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.329+0000 7f5e97e00640 1 -- 172.21.15.90:0/1486251431 >> 172.21.15.90:0/1486251431 conn(0x7f5e9006b720 msgr2=0x7f5e900b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:04.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.329+0000 7f5e97e00640 1 -- 172.21.15.90:0/1486251431 shutdown_connections 2024-04-25T01:18:04.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.329+0000 7f5e97e00640 1 -- 172.21.15.90:0/1486251431 wait complete. 2024-04-25T01:18:04.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.330+0000 7f5e97e00640 1 Processor -- start 2024-04-25T01:18:04.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.331+0000 7f5e97e00640 1 -- start start 2024-04-25T01:18:04.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.331+0000 7f5e97e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e780039e0 0x7f5e901f5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:04.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.331+0000 7f5e97e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e900c5b30 0x7f5e901f68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:04.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.331+0000 7f5e97e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e900c5fb0 0x7f5e901f6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:04.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.331+0000 7f5e97e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5e900b3de0 con 0x7f5e780039e0 2024-04-25T01:18:04.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.331+0000 7f5e97e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5e900b3c60 con 0x7f5e900c5b30 2024-04-25T01:18:04.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.331+0000 7f5e97e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f5e9006df00 con 0x7f5e900c5fb0 2024-04-25T01:18:04.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.332+0000 7f5e95400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e780039e0 0x7f5e901f5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:04.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.332+0000 7f5e95400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e780039e0 0x7f5e901f5010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57186/0 (socket says 172.21.15.90:57186) 2024-04-25T01:18:04.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.332+0000 7f5e95400640 1 -- 172.21.15.90:0/1825258598 learned_addr learned my addr 172.21.15.90:0/1825258598 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:04.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.332+0000 7f5e95e00640 1 --2- 172.21.15.90:0/1825258598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e900c5fb0 0x7f5e901f6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:04.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.332+0000 7f5e94a00640 1 --2- 172.21.15.90:0/1825258598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e900c5b30 0x7f5e901f68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:04.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.332+0000 7f5e95400640 1 -- 172.21.15.90:0/1825258598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e900c5fb0 msgr2=0x7f5e901f6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:04.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.332+0000 7f5e95400640 1 --2- 172.21.15.90:0/1825258598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e900c5fb0 0x7f5e901f6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:04.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.332+0000 7f5e95400640 1 -- 172.21.15.90:0/1825258598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e900c5b30 msgr2=0x7f5e901f68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:04.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.332+0000 7f5e95400640 1 --2- 172.21.15.90:0/1825258598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e900c5b30 0x7f5e901f68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:04.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.332+0000 7f5e95400640 1 -- 172.21.15.90:0/1825258598 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e9015b040 con 0x7f5e780039e0 2024-04-25T01:18:04.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.332+0000 7f5e95e00640 1 --2- 172.21.15.90:0/1825258598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e900c5fb0 0x7f5e901f6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:04.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.332+0000 7f5e94a00640 1 --2- 172.21.15.90:0/1825258598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e900c5b30 0x7f5e901f68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:04.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.333+0000 7f5e95400640 1 --2- 172.21.15.90:0/1825258598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e780039e0 0x7f5e901f5010 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5e8400c200 tx=0x7f5e9006d310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:04.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.333+0000 7f5e8d600640 1 -- 172.21.15.90:0/1825258598 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e84001630 con 0x7f5e780039e0 2024-04-25T01:18:04.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.333+0000 7f5e8d600640 1 -- 172.21.15.90:0/1825258598 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e84042070 con 0x7f5e780039e0 2024-04-25T01:18:04.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.333+0000 7f5e8d600640 1 -- 172.21.15.90:0/1825258598 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e84007290 con 0x7f5e780039e0 2024-04-25T01:18:04.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.333+0000 7f5e97e00640 1 -- 172.21.15.90:0/1825258598 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e9015ed40 con 0x7f5e780039e0 2024-04-25T01:18:04.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.333+0000 7f5e97e00640 1 -- 172.21.15.90:0/1825258598 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e9015b5c0 con 0x7f5e780039e0 2024-04-25T01:18:04.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.334+0000 7f5e97e00640 1 -- 172.21.15.90:0/1825258598 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e900cf480 con 0x7f5e780039e0 2024-04-25T01:18:04.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.335+0000 7f5e8d600640 1 -- 172.21.15.90:0/1825258598 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5e84007430 con 0x7f5e780039e0 2024-04-25T01:18:04.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.336+0000 7f5e8d600640 1 --2- 172.21.15.90:0/1825258598 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f5e7006be30 0x7f5e7006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:04.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.336+0000 7f5e8d600640 1 -- 172.21.15.90:0/1825258598 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6101+0+0 (secure 0 0 0) 0x7f5e840aafb0 con 0x7f5e780039e0 2024-04-25T01:18:04.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.336+0000 7f5e94a00640 1 --2- 172.21.15.90:0/1825258598 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f5e7006be30 0x7f5e7006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:04.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.337+0000 7f5e94a00640 1 --2- 172.21.15.90:0/1825258598 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f5e7006be30 0x7f5e7006e2f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5e9015d570 tx=0x7f5e7c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:04.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.340+0000 7f5e8d600640 1 -- 172.21.15.90:0/1825258598 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f5e840797e0 con 0x7f5e780039e0 2024-04-25T01:18:04.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:04.558+0000 7f5e97e00640 1 -- 172.21.15.90:0/1825258598 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f5e90068fb0 con 0x7f5e780039e0 2024-04-25T01:18:05.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.254+0000 7f5e8d600640 1 -- 172.21.15.90:0/1825258598 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7f5e840790e0 con 0x7f5e780039e0 2024-04-25T01:18:05.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.257+0000 7f5e97e00640 1 -- 172.21.15.90:0/1825258598 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f5e7006be30 msgr2=0x7f5e7006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:05.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.257+0000 7f5e97e00640 1 --2- 172.21.15.90:0/1825258598 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f5e7006be30 0x7f5e7006e2f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5e9015d570 tx=0x7f5e7c008040 comp rx=0 tx=0).stop 2024-04-25T01:18:05.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.257+0000 7f5e97e00640 1 -- 172.21.15.90:0/1825258598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e780039e0 msgr2=0x7f5e901f5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:05.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.257+0000 7f5e97e00640 1 --2- 172.21.15.90:0/1825258598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e780039e0 0x7f5e901f5010 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5e8400c200 tx=0x7f5e9006d310 comp rx=0 tx=0).stop 2024-04-25T01:18:05.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.257+0000 7f5e97e00640 1 -- 172.21.15.90:0/1825258598 shutdown_connections 2024-04-25T01:18:05.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.257+0000 7f5e97e00640 1 --2- 172.21.15.90:0/1825258598 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f5e7006be30 0x7f5e7006e2f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:05.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.257+0000 7f5e97e00640 1 --2- 172.21.15.90:0/1825258598 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f5e900c5fb0 0x7f5e901f6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:05.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.257+0000 7f5e97e00640 1 --2- 172.21.15.90:0/1825258598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e900c5b30 0x7f5e901f68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:05.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.257+0000 7f5e97e00640 1 --2- 172.21.15.90:0/1825258598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e780039e0 0x7f5e901f5010 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:05.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.257+0000 7f5e97e00640 1 -- 172.21.15.90:0/1825258598 >> 172.21.15.90:0/1825258598 conn(0x7f5e9006b720 msgr2=0x7f5e9006e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:05.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.258+0000 7f5e97e00640 1 -- 172.21.15.90:0/1825258598 shutdown_connections 2024-04-25T01:18:05.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.258+0000 7f5e97e00640 1 -- 172.21.15.90:0/1825258598 wait complete. 2024-04-25T01:18:05.260 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-25T01:18:05.260 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-25T01:18:05.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.472+0000 7fd81b200640 1 Processor -- start 2024-04-25T01:18:05.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.473+0000 7fd81b200640 1 -- start start 2024-04-25T01:18:05.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.474+0000 7fd81b200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd81c15e0e0 0x7fd81c0d0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:05.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.474+0000 7fd81b200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd81c15e4c0 con 0x7fd81c0d29c0 2024-04-25T01:18:05.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.474+0000 7fd81b200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd81c0cebc0 con 0x7fd81c0d49b0 2024-04-25T01:18:05.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.474+0000 7fd81b200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd81c0cf4f0 con 0x7fd81c15e0e0 2024-04-25T01:18:05.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.475+0000 7fd81a800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd81c0d29c0 0x7fd81c167be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:49392/0 (socket says 172.21.15.90:49392) 2024-04-25T01:18:05.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.475+0000 7fd81a800640 1 -- 172.21.15.90:0/1186959979 learned_addr learned my addr 172.21.15.90:0/1186959979 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:05.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.475+0000 7fd819e00640 1 --2- 172.21.15.90:0/1186959979 >> v2:172.21.15.103:3300/0 conn(0x7fd81c15e0e0 0x7fd81c0d0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:05.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.475+0000 7fd818a00640 1 -- 172.21.15.90:0/1186959979 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 136079636 0 0) 0x7fd81c15e4c0 con 0x7fd81c0d29c0 2024-04-25T01:18:05.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.475+0000 7fd818a00640 1 -- 172.21.15.90:0/1186959979 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7fc003520 con 0x7fd81c0d29c0 2024-04-25T01:18:05.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.476+0000 7fd818a00640 1 -- 172.21.15.90:0/1186959979 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 117511525 0 0) 0x7fd7fc003520 con 0x7fd81c0d29c0 2024-04-25T01:18:05.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.476+0000 7fd818a00640 1 -- 172.21.15.90:0/1186959979 >> v2:172.21.15.103:3300/0 conn(0x7fd81c15e0e0 msgr2=0x7fd81c0d0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:05.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.476+0000 7fd818a00640 1 --2- 172.21.15.90:0/1186959979 >> v2:172.21.15.103:3300/0 conn(0x7fd81c15e0e0 0x7fd81c0d0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:05.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.476+0000 7fd818a00640 1 -- 172.21.15.90:0/1186959979 >> v1:172.21.15.103:6789/0 conn(0x7fd81c0d49b0 legacy=0x7fd81c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:05.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.476+0000 7fd818a00640 1 -- 172.21.15.90:0/1186959979 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd81c0ceea0 con 0x7fd81c0d29c0 2024-04-25T01:18:05.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.476+0000 7fd818a00640 1 -- 172.21.15.90:0/1186959979 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7fd814001670 con 0x7fd81c0d29c0 2024-04-25T01:18:05.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.476+0000 7fd818a00640 1 -- 172.21.15.90:0/1186959979 >> v1:172.21.15.90:6789/0 conn(0x7fd81c0d29c0 legacy=0x7fd81c167be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:05.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.476+0000 7fd818a00640 1 --2- 172.21.15.90:0/1186959979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7fc003a00 0x7fd7fc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:05.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.476+0000 7fd818a00640 1 --2- 172.21.15.90:0/1186959979 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7fc004800 0x7fd7fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:05.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.476+0000 7fd818a00640 1 --2- 172.21.15.90:0/1186959979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7fc0051c0 0x7fd7fc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:05.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.476+0000 7fd818a00640 1 -- 172.21.15.90:0/1186959979 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd81c0cf4f0 con 0x7fd7fc0051c0 2024-04-25T01:18:05.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.476+0000 7fd818a00640 1 -- 172.21.15.90:0/1186959979 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd7fc00c0c0 con 0x7fd7fc003a00 2024-04-25T01:18:05.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.476+0000 7fd818a00640 1 -- 172.21.15.90:0/1186959979 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd7fc00c2c0 con 0x7fd7fc004800 2024-04-25T01:18:05.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.476+0000 7fd81a800640 1 --2- 172.21.15.90:0/1186959979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7fc0051c0 0x7fd7fc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:05.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.476+0000 7fd819e00640 1 --2- 172.21.15.90:0/1186959979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7fc003a00 0x7fd7fc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:05.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.476+0000 7fd81a800640 1 -- 172.21.15.90:0/1186959979 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7fc004800 msgr2=0x7fd7fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:05.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.476+0000 7fd81a800640 1 --2- 172.21.15.90:0/1186959979 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7fc004800 0x7fd7fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:05.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.476+0000 7fd81a800640 1 -- 172.21.15.90:0/1186959979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7fc003a00 msgr2=0x7fd7fc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:05.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.476+0000 7fd81a800640 1 --2- 172.21.15.90:0/1186959979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7fc003a00 0x7fd7fc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:05.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.476+0000 7fd81a800640 1 -- 172.21.15.90:0/1186959979 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7fc003520 con 0x7fd7fc0051c0 2024-04-25T01:18:05.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.477+0000 7fd819e00640 1 --2- 172.21.15.90:0/1186959979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7fc003a00 0x7fd7fc003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:05.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.477+0000 7fd81a800640 1 -- 172.21.15.90:0/1186959979 reap_dead start 2024-04-25T01:18:05.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.477+0000 7fd81a800640 1 --2- 172.21.15.90:0/1186959979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7fc0051c0 0x7fd7fc00b9c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd81400c0e0 tx=0x7fd7fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a055547882e6c5fc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:05.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.478+0000 7fd818a00640 1 -- 172.21.15.90:0/1186959979 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd814007c20 con 0x7fd7fc0051c0 2024-04-25T01:18:05.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.478+0000 7fd818a00640 1 -- 172.21.15.90:0/1186959979 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd814005af0 con 0x7fd7fc0051c0 2024-04-25T01:18:05.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.478+0000 7fd818a00640 1 -- 172.21.15.90:0/1186959979 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd814007070 con 0x7fd7fc0051c0 2024-04-25T01:18:05.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.478+0000 7fd81b200640 1 -- 172.21.15.90:0/1186959979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7fc0051c0 msgr2=0x7fd7fc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:05.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.478+0000 7fd81b200640 1 --2- 172.21.15.90:0/1186959979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7fc0051c0 0x7fd7fc00b9c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd81400c0e0 tx=0x7fd7fc005830 comp rx=0 tx=0).stop 2024-04-25T01:18:05.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.478+0000 7fd81b200640 1 -- 172.21.15.90:0/1186959979 shutdown_connections 2024-04-25T01:18:05.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.478+0000 7fd81b200640 1 --2- 172.21.15.90:0/1186959979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7fc0051c0 0x7fd7fc00b9c0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:05.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.478+0000 7fd81b200640 1 -- 172.21.15.90:0/1186959979 >> 172.21.15.90:0/1186959979 conn(0x7fd81c0d3cd0 msgr2=0x7fd81c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:05.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.479+0000 7fd81b200640 1 -- 172.21.15.90:0/1186959979 shutdown_connections 2024-04-25T01:18:05.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.479+0000 7fd81b200640 1 -- 172.21.15.90:0/1186959979 wait complete. 2024-04-25T01:18:05.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.480+0000 7fd81b200640 1 Processor -- start 2024-04-25T01:18:05.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.480+0000 7fd81b200640 1 -- start start 2024-04-25T01:18:05.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.481+0000 7fd81b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7fc0051c0 0x7fd81c0a0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:05.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.481+0000 7fd81b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd81c0a0c60 0x7fd81c0cbf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:05.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.481+0000 7fd81b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd81c0cc480 0x7fd81c1f3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:05.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.481+0000 7fd81b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd81c0ce960 con 0x7fd81c0cc480 2024-04-25T01:18:05.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.481+0000 7fd81b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd81c0ce7e0 con 0x7fd81c0a0c60 2024-04-25T01:18:05.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.481+0000 7fd81b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd81c0cf980 con 0x7fd7fc0051c0 2024-04-25T01:18:05.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.482+0000 7fd81a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd81c0cc480 0x7fd81c1f3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:05.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.482+0000 7fd81a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd81c0cc480 0x7fd81c1f3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33110/0 (socket says 172.21.15.90:33110) 2024-04-25T01:18:05.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.482+0000 7fd81a800640 1 -- 172.21.15.90:0/2213679919 learned_addr learned my addr 172.21.15.90:0/2213679919 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:05.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.482+0000 7fd819400640 1 --2- 172.21.15.90:0/2213679919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd81c0a0c60 0x7fd81c0cbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:05.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.482+0000 7fd819e00640 1 --2- 172.21.15.90:0/2213679919 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7fc0051c0 0x7fd81c0a0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:05.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.482+0000 7fd81a800640 1 -- 172.21.15.90:0/2213679919 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7fc0051c0 msgr2=0x7fd81c0a0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:05.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.482+0000 7fd81a800640 1 --2- 172.21.15.90:0/2213679919 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7fc0051c0 0x7fd81c0a0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:05.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.482+0000 7fd81a800640 1 -- 172.21.15.90:0/2213679919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd81c0a0c60 msgr2=0x7fd81c0cbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:05.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.482+0000 7fd81a800640 1 --2- 172.21.15.90:0/2213679919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd81c0a0c60 0x7fd81c0cbf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:05.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.482+0000 7fd81a800640 1 -- 172.21.15.90:0/2213679919 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd81c0cccb0 con 0x7fd81c0cc480 2024-04-25T01:18:05.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.482+0000 7fd81a800640 1 --2- 172.21.15.90:0/2213679919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd81c0cc480 0x7fd81c1f3e80 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fd814000c00 tx=0x7fd81c0ced90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:05.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.483+0000 7fd812000640 1 -- 172.21.15.90:0/2213679919 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd814001640 con 0x7fd81c0cc480 2024-04-25T01:18:05.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.483+0000 7fd812000640 1 -- 172.21.15.90:0/2213679919 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8140047e0 con 0x7fd81c0cc480 2024-04-25T01:18:05.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.483+0000 7fd812000640 1 -- 172.21.15.90:0/2213679919 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8140049b0 con 0x7fd81c0cc480 2024-04-25T01:18:05.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.483+0000 7fd81b200640 1 -- 172.21.15.90:0/2213679919 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd81c0ccf90 con 0x7fd81c0cc480 2024-04-25T01:18:05.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.483+0000 7fd81b200640 1 -- 172.21.15.90:0/2213679919 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd81c0cd550 con 0x7fd81c0cc480 2024-04-25T01:18:05.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.484+0000 7fd81b200640 1 -- 172.21.15.90:0/2213679919 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd81c075350 con 0x7fd81c0cc480 2024-04-25T01:18:05.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.485+0000 7fd812000640 1 -- 172.21.15.90:0/2213679919 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd814043020 con 0x7fd81c0cc480 2024-04-25T01:18:05.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.486+0000 7fd812000640 1 --2- 172.21.15.90:0/2213679919 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd7f806be30 0x7fd7f806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:05.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.487+0000 7fd812000640 1 -- 172.21.15.90:0/2213679919 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6101+0+0 (secure 0 0 0) 0x7fd8140aa250 con 0x7fd81c0cc480 2024-04-25T01:18:05.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.487+0000 7fd819e00640 1 --2- 172.21.15.90:0/2213679919 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd7f806be30 0x7fd7f806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:05.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.487+0000 7fd819e00640 1 --2- 172.21.15.90:0/2213679919 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd7f806be30 0x7fd7f806e2f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd80c009aa0 tx=0x7fd80c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:05.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.491+0000 7fd812000640 1 -- 172.21.15.90:0/2213679919 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd81404a020 con 0x7fd81c0cc480 2024-04-25T01:18:05.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:05.729+0000 7fd81b200640 1 -- 172.21.15.90:0/2213679919 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fd81c0d0ec0 con 0x7fd81c0cc480 2024-04-25T01:18:06.174 INFO:tasks.ceph.mds.c.smithi090.stderr:2024-04-25T01:18:06.191+0000 7f41d2a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T01:18:06.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.265+0000 7fd812000640 1 -- 172.21.15.90:0/2213679919 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v18) ==== 122+0+0 (secure 0 0 0) 0x7fd8140789d0 con 0x7fd81c0cc480 2024-04-25T01:18:06.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.269+0000 7fd81b200640 1 -- 172.21.15.90:0/2213679919 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd7f806be30 msgr2=0x7fd7f806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:06.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.269+0000 7fd81b200640 1 --2- 172.21.15.90:0/2213679919 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd7f806be30 0x7fd7f806e2f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd80c009aa0 tx=0x7fd80c008040 comp rx=0 tx=0).stop 2024-04-25T01:18:06.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.269+0000 7fd81b200640 1 -- 172.21.15.90:0/2213679919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd81c0cc480 msgr2=0x7fd81c1f3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:06.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.269+0000 7fd81b200640 1 --2- 172.21.15.90:0/2213679919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd81c0cc480 0x7fd81c1f3e80 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fd814000c00 tx=0x7fd81c0ced90 comp rx=0 tx=0).stop 2024-04-25T01:18:06.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.269+0000 7fd81b200640 1 -- 172.21.15.90:0/2213679919 shutdown_connections 2024-04-25T01:18:06.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.269+0000 7fd81b200640 1 --2- 172.21.15.90:0/2213679919 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd7f806be30 0x7fd7f806e2f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:06.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.269+0000 7fd81b200640 1 --2- 172.21.15.90:0/2213679919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd81c0cc480 0x7fd81c1f3e80 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:06.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.269+0000 7fd81b200640 1 --2- 172.21.15.90:0/2213679919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd81c0a0c60 0x7fd81c0cbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:06.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.269+0000 7fd81b200640 1 --2- 172.21.15.90:0/2213679919 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd7fc0051c0 0x7fd81c0a0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:06.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.269+0000 7fd81b200640 1 -- 172.21.15.90:0/2213679919 >> 172.21.15.90:0/2213679919 conn(0x7fd81c0d3cd0 msgr2=0x7fd81c15e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:06.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.269+0000 7fd81b200640 1 -- 172.21.15.90:0/2213679919 shutdown_connections 2024-04-25T01:18:06.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.269+0000 7fd81b200640 1 -- 172.21.15.90:0/2213679919 wait complete. 2024-04-25T01:18:06.273 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-25T01:18:06.273 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:18:06.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.479+0000 7eff3b000640 1 Processor -- start 2024-04-25T01:18:06.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.479+0000 7eff3b000640 1 -- start start 2024-04-25T01:18:06.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.481+0000 7eff3b000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7eff3406b120 0x7eff3406b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:06.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.481+0000 7eff3b000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7eff3415ca10 0x7eff3415cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:06.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.481+0000 7eff3b000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff3406ba40 con 0x7eff340ae730 2024-04-25T01:18:06.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.481+0000 7eff3b000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7eff340cef40 con 0x7eff3406b120 2024-04-25T01:18:06.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.481+0000 7eff3b000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7eff3415c580 con 0x7eff3415ca10 2024-04-25T01:18:06.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.481+0000 7eff3a600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7eff3415ca10 0x7eff3415cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:06.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.481+0000 7eff39c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7eff3406b120 0x7eff3406b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:06.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.481+0000 7eff3a600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7eff3415ca10 0x7eff3415cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.90:49348/0 (socket says 172.21.15.90:49348) 2024-04-25T01:18:06.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.481+0000 7eff3a600640 1 -- 172.21.15.90:0/1551966746 learned_addr learned my addr 172.21.15.90:0/1551966746 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:06.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.482+0000 7eff3a600640 1 -- 172.21.15.90:0/1551966746 >> v2:172.21.15.103:3300/0 conn(0x7eff3406b120 msgr2=0x7eff3406b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:06.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.482+0000 7eff3a600640 1 --2- 172.21.15.90:0/1551966746 >> v2:172.21.15.103:3300/0 conn(0x7eff3406b120 0x7eff3406b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:06.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.482+0000 7eff3a600640 1 -- 172.21.15.90:0/1551966746 >> v1:172.21.15.103:6789/0 conn(0x7eff340ae730 legacy=0x7eff340cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:06.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.482+0000 7eff3a600640 1 -- 172.21.15.90:0/1551966746 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff3415c220 con 0x7eff3415ca10 2024-04-25T01:18:06.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.484+0000 7eff3a600640 1 --2- 172.21.15.90:0/1551966746 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7eff3415ca10 0x7eff3415cea0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7eff2c009e50 tx=0x7eff2c02a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e92564158709b000 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:06.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.484+0000 7eff33e00640 1 -- 172.21.15.90:0/1551966746 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2c036070 con 0x7eff3415ca10 2024-04-25T01:18:06.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.484+0000 7eff33e00640 1 -- 172.21.15.90:0/1551966746 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff2c031600 con 0x7eff3415ca10 2024-04-25T01:18:06.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.484+0000 7eff33e00640 1 -- 172.21.15.90:0/1551966746 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2c0319d0 con 0x7eff3415ca10 2024-04-25T01:18:06.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.485+0000 7eff3b000640 1 -- 172.21.15.90:0/1551966746 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7eff3415ca10 msgr2=0x7eff3415cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:06.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.485+0000 7eff3b000640 1 --2- 172.21.15.90:0/1551966746 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7eff3415ca10 0x7eff3415cea0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7eff2c009e50 tx=0x7eff2c02a710 comp rx=0 tx=0).stop 2024-04-25T01:18:06.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.485+0000 7eff3b000640 1 -- 172.21.15.90:0/1551966746 shutdown_connections 2024-04-25T01:18:06.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.485+0000 7eff3b000640 1 --2- 172.21.15.90:0/1551966746 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7eff3415ca10 0x7eff3415cea0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:06.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.485+0000 7eff3b000640 1 --2- 172.21.15.90:0/1551966746 >> v2:172.21.15.103:3300/0 conn(0x7eff3406b120 0x7eff3406b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:06.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.485+0000 7eff3b000640 1 -- 172.21.15.90:0/1551966746 >> 172.21.15.90:0/1551966746 conn(0x7eff340b1660 msgr2=0x7eff340b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:06.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.485+0000 7eff3b000640 1 -- 172.21.15.90:0/1551966746 shutdown_connections 2024-04-25T01:18:06.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.485+0000 7eff3b000640 1 -- 172.21.15.90:0/1551966746 wait complete. 2024-04-25T01:18:06.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.487+0000 7eff3b000640 1 Processor -- start 2024-04-25T01:18:06.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.487+0000 7eff3b000640 1 -- start start 2024-04-25T01:18:06.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.488+0000 7eff3b000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff3406b120 0x7eff34145fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:06.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.488+0000 7eff3b000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff340ae730 0x7eff341f3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:06.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.488+0000 7eff3b000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7eff3415ca10 0x7eff34153320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:06.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.488+0000 7eff3b000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7eff340af240 con 0x7eff3406b120 2024-04-25T01:18:06.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.488+0000 7eff3b000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7eff340aef60 con 0x7eff340ae730 2024-04-25T01:18:06.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.488+0000 7eff3b000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7eff34153080 con 0x7eff3415ca10 2024-04-25T01:18:06.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.488+0000 7eff39c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff3406b120 0x7eff34145fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:06.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.488+0000 7eff39c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff3406b120 0x7eff34145fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33120/0 (socket says 172.21.15.90:33120) 2024-04-25T01:18:06.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.488+0000 7eff39c00640 1 -- 172.21.15.90:0/1269708466 learned_addr learned my addr 172.21.15.90:0/1269708466 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:06.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.488+0000 7eff39200640 1 --2- 172.21.15.90:0/1269708466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff340ae730 0x7eff341f3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:06.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.488+0000 7eff3a600640 1 --2- 172.21.15.90:0/1269708466 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7eff3415ca10 0x7eff34153320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:06.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.488+0000 7eff39c00640 1 -- 172.21.15.90:0/1269708466 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7eff3415ca10 msgr2=0x7eff34153320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:06.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.488+0000 7eff39c00640 1 --2- 172.21.15.90:0/1269708466 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7eff3415ca10 0x7eff34153320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:06.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.488+0000 7eff39c00640 1 -- 172.21.15.90:0/1269708466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff340ae730 msgr2=0x7eff341f3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:06.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.488+0000 7eff39c00640 1 --2- 172.21.15.90:0/1269708466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff340ae730 0x7eff341f3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:06.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.488+0000 7eff39c00640 1 -- 172.21.15.90:0/1269708466 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff341f4480 con 0x7eff3406b120 2024-04-25T01:18:06.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.488+0000 7eff39200640 1 --2- 172.21.15.90:0/1269708466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff340ae730 0x7eff341f3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:06.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.489+0000 7eff39c00640 1 --2- 172.21.15.90:0/1269708466 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff3406b120 0x7eff34145fb0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7eff2800e9b0 tx=0x7eff2800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:06.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.489+0000 7eff31600640 1 -- 172.21.15.90:0/1269708466 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff28017070 con 0x7eff3406b120 2024-04-25T01:18:06.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.489+0000 7eff31600640 1 -- 172.21.15.90:0/1269708466 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff2800c9a0 con 0x7eff3406b120 2024-04-25T01:18:06.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.489+0000 7eff31600640 1 -- 172.21.15.90:0/1269708466 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2800cd90 con 0x7eff3406b120 2024-04-25T01:18:06.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.489+0000 7eff3b000640 1 -- 172.21.15.90:0/1269708466 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff34153860 con 0x7eff3406b120 2024-04-25T01:18:06.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.489+0000 7eff3b000640 1 -- 172.21.15.90:0/1269708466 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff34156050 con 0x7eff3406b120 2024-04-25T01:18:06.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.490+0000 7eff3b000640 1 -- 172.21.15.90:0/1269708466 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff3406b500 con 0x7eff3406b120 2024-04-25T01:18:06.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.491+0000 7eff31600640 1 -- 172.21.15.90:0/1269708466 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7eff2801a070 con 0x7eff3406b120 2024-04-25T01:18:06.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.492+0000 7eff31600640 1 --2- 172.21.15.90:0/1269708466 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7eff1406be60 0x7eff1406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:06.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.492+0000 7eff31600640 1 -- 172.21.15.90:0/1269708466 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6101+0+0 (secure 0 0 0) 0x7eff2808aff0 con 0x7eff3406b120 2024-04-25T01:18:06.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.492+0000 7eff39200640 1 --2- 172.21.15.90:0/1269708466 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7eff1406be60 0x7eff1406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:06.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.493+0000 7eff39200640 1 --2- 172.21.15.90:0/1269708466 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7eff1406be60 0x7eff1406e320 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7eff34146d10 tx=0x7eff20009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:06.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.496+0000 7eff31600640 1 -- 172.21.15.90:0/1269708466 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7eff3406b500 con 0x7eff3406b120 2024-04-25T01:18:06.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.716+0000 7eff3b000640 1 -- 172.21.15.90:0/1269708466 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff3406d2d0 con 0x7eff3406b120 2024-04-25T01:18:06.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.717+0000 7eff31600640 1 -- 172.21.15.90:0/1269708466 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5184 (secure 0 0 0) 0x7eff28059800 con 0x7eff3406b120 2024-04-25T01:18:06.701 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:06.701 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4598,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1475874994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1475874994},{"type":"v1","addr":"172.21.15.90:6835","nonce":1475874994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4610,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6834/1993144745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1993144745},{"type":"v1","addr":"172.21.15.103:6834","nonce":1993144745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4840,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2376794333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6833","nonce":2376794333},{"type":"v1","addr":"172.21.15.103:6835","nonce":2376794333}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:18:02.701652+0000","modified":"2024-04-25T01:18:05.730947+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5334},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5334":{"gid":5334,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.90:6837/431390375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":431390375},{"type":"v1","addr":"172.21.15.90:6837","nonce":431390375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5334,"qdb_cluster":[5334]},"id":2}]} 2024-04-25T01:18:06.701 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 18 2024-04-25T01:18:06.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.719+0000 7eff3b000640 1 -- 172.21.15.90:0/1269708466 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7eff1406be60 msgr2=0x7eff1406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:06.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.720+0000 7eff3b000640 1 --2- 172.21.15.90:0/1269708466 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7eff1406be60 0x7eff1406e320 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7eff34146d10 tx=0x7eff20009040 comp rx=0 tx=0).stop 2024-04-25T01:18:06.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.720+0000 7eff3b000640 1 -- 172.21.15.90:0/1269708466 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff3406b120 msgr2=0x7eff34145fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:06.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.720+0000 7eff3b000640 1 --2- 172.21.15.90:0/1269708466 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff3406b120 0x7eff34145fb0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7eff2800e9b0 tx=0x7eff2800ee70 comp rx=0 tx=0).stop 2024-04-25T01:18:06.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.720+0000 7eff3b000640 1 -- 172.21.15.90:0/1269708466 shutdown_connections 2024-04-25T01:18:06.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.720+0000 7eff3b000640 1 --2- 172.21.15.90:0/1269708466 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7eff1406be60 0x7eff1406e320 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:06.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.720+0000 7eff3b000640 1 --2- 172.21.15.90:0/1269708466 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7eff3415ca10 0x7eff34153320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:06.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.720+0000 7eff3b000640 1 --2- 172.21.15.90:0/1269708466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff340ae730 0x7eff341f3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:06.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.720+0000 7eff3b000640 1 --2- 172.21.15.90:0/1269708466 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff3406b120 0x7eff34145fb0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:06.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.720+0000 7eff3b000640 1 -- 172.21.15.90:0/1269708466 >> 172.21.15.90:0/1269708466 conn(0x7eff340b1660 msgr2=0x7eff340af5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:06.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.720+0000 7eff3b000640 1 -- 172.21.15.90:0/1269708466 shutdown_connections 2024-04-25T01:18:06.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.720+0000 7eff3b000640 1 -- 172.21.15.90:0/1269708466 wait complete. 2024-04-25T01:18:06.721 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:18:06.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.938+0000 7fd8d4a00640 1 Processor -- start 2024-04-25T01:18:06.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.938+0000 7fd8d4a00640 1 -- start start 2024-04-25T01:18:06.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.940+0000 7fd8d4a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8d0145590 0x7fd8d00b4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:06.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.940+0000 7fd8d4a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8d00b4240 con 0x7fd8d015f9b0 2024-04-25T01:18:06.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.940+0000 7fd8d4a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8d00b4f60 con 0x7fd8d006e280 2024-04-25T01:18:06.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.940+0000 7fd8d4a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd8d0160830 con 0x7fd8d0145590 2024-04-25T01:18:06.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.940+0000 7fd8cd400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd8d015f9b0 0x7fd8d00ce110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:49404/0 (socket says 172.21.15.90:49404) 2024-04-25T01:18:06.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.940+0000 7fd8cd400640 1 -- 172.21.15.90:0/525216553 learned_addr learned my addr 172.21.15.90:0/525216553 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:06.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.940+0000 7fd8cde00640 1 --2- 172.21.15.90:0/525216553 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8d0145590 0x7fd8d00b4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:06.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.940+0000 7fd8cca00640 1 -- 172.21.15.90:0/525216553 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3254573399 0 0) 0x7fd8d00b4240 con 0x7fd8d015f9b0 2024-04-25T01:18:06.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.940+0000 7fd8cca00640 1 -- 172.21.15.90:0/525216553 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8b4003520 con 0x7fd8d015f9b0 2024-04-25T01:18:06.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.941+0000 7fd8cca00640 1 -- 172.21.15.90:0/525216553 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1818435083 0 0) 0x7fd8b4003520 con 0x7fd8d015f9b0 2024-04-25T01:18:06.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.941+0000 7fd8cca00640 1 -- 172.21.15.90:0/525216553 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8d0145590 msgr2=0x7fd8d00b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:06.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.941+0000 7fd8cca00640 1 --2- 172.21.15.90:0/525216553 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8d0145590 0x7fd8d00b4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:06.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.941+0000 7fd8cca00640 1 -- 172.21.15.90:0/525216553 >> v1:172.21.15.103:6789/0 conn(0x7fd8d006e280 legacy=0x7fd8d00ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:06.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.941+0000 7fd8cca00640 1 -- 172.21.15.90:0/525216553 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8d00b39a0 con 0x7fd8d015f9b0 2024-04-25T01:18:06.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.941+0000 7fd8cca00640 1 -- 172.21.15.90:0/525216553 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7fd8bc001670 con 0x7fd8d015f9b0 2024-04-25T01:18:06.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.941+0000 7fd8cca00640 1 -- 172.21.15.90:0/525216553 >> v1:172.21.15.90:6789/0 conn(0x7fd8d015f9b0 legacy=0x7fd8d00ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:06.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.941+0000 7fd8cca00640 1 --2- 172.21.15.90:0/525216553 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8b40039e0 0x7fd8b4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:06.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.941+0000 7fd8cca00640 1 --2- 172.21.15.90:0/525216553 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8d0145590 0x7fd8b4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:06.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.941+0000 7fd8cca00640 1 --2- 172.21.15.90:0/525216553 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8b4004d40 0x7fd8b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:06.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.941+0000 7fd8cca00640 1 -- 172.21.15.90:0/525216553 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b400bb50 con 0x7fd8b4004d40 2024-04-25T01:18:06.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.941+0000 7fd8cca00640 1 -- 172.21.15.90:0/525216553 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b400bcd0 con 0x7fd8b40039e0 2024-04-25T01:18:06.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.941+0000 7fd8cca00640 1 -- 172.21.15.90:0/525216553 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd8b400bea0 con 0x7fd8d0145590 2024-04-25T01:18:06.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.941+0000 7fd8ce800640 1 --2- 172.21.15.90:0/525216553 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8b4004d40 0x7fd8b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:06.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.942+0000 7fd8cd400640 1 --2- 172.21.15.90:0/525216553 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8d0145590 0x7fd8b4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:06.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.942+0000 7fd8cde00640 1 --2- 172.21.15.90:0/525216553 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8b40039e0 0x7fd8b4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:06.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.942+0000 7fd8ce800640 1 -- 172.21.15.90:0/525216553 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8d0145590 msgr2=0x7fd8b4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:06.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.942+0000 7fd8ce800640 1 --2- 172.21.15.90:0/525216553 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8d0145590 0x7fd8b4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:06.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.942+0000 7fd8ce800640 1 -- 172.21.15.90:0/525216553 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8b40039e0 msgr2=0x7fd8b4003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:06.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.942+0000 7fd8ce800640 1 --2- 172.21.15.90:0/525216553 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8b40039e0 0x7fd8b4003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:06.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.942+0000 7fd8ce800640 1 -- 172.21.15.90:0/525216553 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8b4003520 con 0x7fd8b4004d40 2024-04-25T01:18:06.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.942+0000 7fd8cde00640 1 --2- 172.21.15.90:0/525216553 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8b40039e0 0x7fd8b4003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:06.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.943+0000 7fd8ce800640 1 --2- 172.21.15.90:0/525216553 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8b4004d40 0x7fd8b40051c0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8001e50 tx=0x7fd8b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c80a482015e496ee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:06.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.943+0000 7fd8cca00640 1 -- 172.21.15.90:0/525216553 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c8001a70 con 0x7fd8b4004d40 2024-04-25T01:18:06.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.943+0000 7fd8cca00640 1 -- 172.21.15.90:0/525216553 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8c802ae30 con 0x7fd8b4004d40 2024-04-25T01:18:06.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.943+0000 7fd8cca00640 1 -- 172.21.15.90:0/525216553 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c8003b90 con 0x7fd8b4004d40 2024-04-25T01:18:06.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.944+0000 7fd8d4a00640 1 -- 172.21.15.90:0/525216553 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8b4004d40 msgr2=0x7fd8b40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:06.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.944+0000 7fd8d4a00640 1 --2- 172.21.15.90:0/525216553 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8b4004d40 0x7fd8b40051c0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8001e50 tx=0x7fd8b4005830 comp rx=0 tx=0).stop 2024-04-25T01:18:06.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.944+0000 7fd8ce800640 1 -- 172.21.15.90:0/525216553 reap_dead start 2024-04-25T01:18:06.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.944+0000 7fd8d4a00640 1 -- 172.21.15.90:0/525216553 shutdown_connections 2024-04-25T01:18:06.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.944+0000 7fd8d4a00640 1 -- 172.21.15.90:0/525216553 >> 172.21.15.90:0/525216553 conn(0x7fd8d00b1820 msgr2=0x7fd8d00b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:06.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.944+0000 7fd8d4a00640 1 -- 172.21.15.90:0/525216553 shutdown_connections 2024-04-25T01:18:06.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.944+0000 7fd8d4a00640 1 -- 172.21.15.90:0/525216553 wait complete. 2024-04-25T01:18:06.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.946+0000 7fd8d4a00640 1 Processor -- start 2024-04-25T01:18:06.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.946+0000 7fd8d4a00640 1 -- start start 2024-04-25T01:18:06.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.947+0000 7fd8d4a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8b4004d40 0x7fd8d013bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:06.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.947+0000 7fd8d4a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8d0141c00 0x7fd8d013a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:06.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.947+0000 7fd8d4a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8d0142080 0x7fd8d015af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:06.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.947+0000 7fd8d4a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d00b3f60 con 0x7fd8d0141c00 2024-04-25T01:18:06.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.947+0000 7fd8d4a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d006cde0 con 0x7fd8d0142080 2024-04-25T01:18:06.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.947+0000 7fd8d4a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd8d0145970 con 0x7fd8b4004d40 2024-04-25T01:18:06.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.947+0000 7fd8cd400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8d0141c00 0x7fd8d013a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:06.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.947+0000 7fd8cd400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8d0141c00 0x7fd8d013a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33146/0 (socket says 172.21.15.90:33146) 2024-04-25T01:18:06.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.947+0000 7fd8cd400640 1 -- 172.21.15.90:0/516510851 learned_addr learned my addr 172.21.15.90:0/516510851 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:06.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.948+0000 7fd8cde00640 1 --2- 172.21.15.90:0/516510851 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8b4004d40 0x7fd8d013bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:06.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.948+0000 7fd8cd400640 1 -- 172.21.15.90:0/516510851 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8b4004d40 msgr2=0x7fd8d013bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:06.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.948+0000 7fd8cd400640 1 --2- 172.21.15.90:0/516510851 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8b4004d40 0x7fd8d013bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:06.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.948+0000 7fd8ce800640 1 --2- 172.21.15.90:0/516510851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8d0142080 0x7fd8d015af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:06.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.948+0000 7fd8cd400640 1 -- 172.21.15.90:0/516510851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8d0142080 msgr2=0x7fd8d015af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:06.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.948+0000 7fd8cd400640 1 --2- 172.21.15.90:0/516510851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8d0142080 0x7fd8d015af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:06.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.948+0000 7fd8cd400640 1 -- 172.21.15.90:0/516510851 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8d0137800 con 0x7fd8d0141c00 2024-04-25T01:18:06.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.948+0000 7fd8cd400640 1 --2- 172.21.15.90:0/516510851 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8d0141c00 0x7fd8d013a7d0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc00e7f0 tx=0x7fd8bc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:06.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.948+0000 7fd8c6000640 1 -- 172.21.15.90:0/516510851 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc00c8d0 con 0x7fd8d0141c00 2024-04-25T01:18:06.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.948+0000 7fd8c6000640 1 -- 172.21.15.90:0/516510851 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8bc018070 con 0x7fd8d0141c00 2024-04-25T01:18:06.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.948+0000 7fd8c6000640 1 -- 172.21.15.90:0/516510851 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc00ccb0 con 0x7fd8d0141c00 2024-04-25T01:18:06.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.948+0000 7fd8d4a00640 1 -- 172.21.15.90:0/516510851 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8d013c260 con 0x7fd8d0141c00 2024-04-25T01:18:06.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.948+0000 7fd8d4a00640 1 -- 172.21.15.90:0/516510851 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8d013b550 con 0x7fd8d0141c00 2024-04-25T01:18:06.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.950+0000 7fd8d4a00640 1 -- 172.21.15.90:0/516510851 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8d0068fb0 con 0x7fd8d0141c00 2024-04-25T01:18:06.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.950+0000 7fd8c6000640 1 -- 172.21.15.90:0/516510851 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd8bc01d460 con 0x7fd8d0141c00 2024-04-25T01:18:06.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.951+0000 7fd8c6000640 1 --2- 172.21.15.90:0/516510851 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd8b006be60 0x7fd8b006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:06.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.951+0000 7fd8c6000640 1 -- 172.21.15.90:0/516510851 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6101+0+0 (secure 0 0 0) 0x7fd8bc08ae70 con 0x7fd8d0141c00 2024-04-25T01:18:06.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.952+0000 7fd8cde00640 1 --2- 172.21.15.90:0/516510851 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd8b006be60 0x7fd8b006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:06.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.952+0000 7fd8cde00640 1 --2- 172.21.15.90:0/516510851 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd8b006be60 0x7fd8b006e320 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd8c0006fd0 tx=0x7fd8c0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:06.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:06.956+0000 7fd8c6000640 1 -- 172.21.15.90:0/516510851 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd8bc059720 con 0x7fd8d0141c00 2024-04-25T01:18:07.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.152+0000 7fd8d4a00640 1 -- 172.21.15.90:0/516510851 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd8d006ea50 con 0x7fd8d0141c00 2024-04-25T01:18:07.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.155+0000 7fd8c6000640 1 -- 172.21.15.90:0/516510851 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) ==== 74+0+16906 (secure 0 0 0) 0x7fd8bc059020 con 0x7fd8d0141c00 2024-04-25T01:18:07.139 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:07.139 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":40,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:18:04.241731+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:18:00.504992+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T01:18:01.714313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:18:07.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.157+0000 7fd8d4a00640 1 -- 172.21.15.90:0/516510851 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd8b006be60 msgr2=0x7fd8b006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:07.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.157+0000 7fd8d4a00640 1 --2- 172.21.15.90:0/516510851 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd8b006be60 0x7fd8b006e320 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd8c0006fd0 tx=0x7fd8c0008040 comp rx=0 tx=0).stop 2024-04-25T01:18:07.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.157+0000 7fd8d4a00640 1 -- 172.21.15.90:0/516510851 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8d0141c00 msgr2=0x7fd8d013a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:07.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.157+0000 7fd8d4a00640 1 --2- 172.21.15.90:0/516510851 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8d0141c00 0x7fd8d013a7d0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc00e7f0 tx=0x7fd8bc00ecb0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.157+0000 7fd8d4a00640 1 -- 172.21.15.90:0/516510851 shutdown_connections 2024-04-25T01:18:07.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.157+0000 7fd8d4a00640 1 --2- 172.21.15.90:0/516510851 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd8b006be60 0x7fd8b006e320 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.157+0000 7fd8d4a00640 1 --2- 172.21.15.90:0/516510851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8d0142080 0x7fd8d015af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.157+0000 7fd8d4a00640 1 --2- 172.21.15.90:0/516510851 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8d0141c00 0x7fd8d013a7d0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.157+0000 7fd8d4a00640 1 --2- 172.21.15.90:0/516510851 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd8b4004d40 0x7fd8d013bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.157+0000 7fd8d4a00640 1 -- 172.21.15.90:0/516510851 >> 172.21.15.90:0/516510851 conn(0x7fd8d00b1820 msgr2=0x7fd8d0142540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:07.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.158+0000 7fd8d4a00640 1 -- 172.21.15.90:0/516510851 shutdown_connections 2024-04-25T01:18:07.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.158+0000 7fd8d4a00640 1 -- 172.21.15.90:0/516510851 wait complete. 2024-04-25T01:18:07.161 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T01:18:07.161 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:18:07.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.376+0000 7fc498a00640 1 Processor -- start 2024-04-25T01:18:07.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.377+0000 7fc498a00640 1 -- start start 2024-04-25T01:18:07.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.378+0000 7fc498a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc49415ad40 0x7fc49406b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:07.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.378+0000 7fc498a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc4940cf520 0x7fc4940ae730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:07.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.378+0000 7fc498a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc49406ba90 con 0x7fc49406d910 2024-04-25T01:18:07.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.378+0000 7fc498a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fc49406cb50 con 0x7fc49415ad40 2024-04-25T01:18:07.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.378+0000 7fc498a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc49406ce30 con 0x7fc4940cf520 2024-04-25T01:18:07.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.379+0000 7fc493400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc49415ad40 0x7fc49406b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:07.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.379+0000 7fc493400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc49415ad40 0x7fc49406b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33156/0 (socket says 172.21.15.90:33156) 2024-04-25T01:18:07.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.379+0000 7fc493400640 1 -- 172.21.15.90:0/3266314454 learned_addr learned my addr 172.21.15.90:0/3266314454 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:07.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.379+0000 7fc493400640 1 -- 172.21.15.90:0/3266314454 >> v2:172.21.15.103:3300/0 conn(0x7fc4940cf520 msgr2=0x7fc4940ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:07.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.379+0000 7fc48aa00640 1 --2- 172.21.15.90:0/3266314454 >> v2:172.21.15.103:3300/0 conn(0x7fc4940cf520 0x7fc4940ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:07.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.379+0000 7fc493400640 1 --2- 172.21.15.90:0/3266314454 >> v2:172.21.15.103:3300/0 conn(0x7fc4940cf520 0x7fc4940ae730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.379+0000 7fc493400640 1 -- 172.21.15.90:0/3266314454 >> v1:172.21.15.103:6789/0 conn(0x7fc49406d910 legacy=0x7fc49415e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:07.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.379+0000 7fc493400640 1 -- 172.21.15.90:0/3266314454 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc49406d110 con 0x7fc49415ad40 2024-04-25T01:18:07.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.380+0000 7fc493400640 1 --2- 172.21.15.90:0/3266314454 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc49415ad40 0x7fc49406b120 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc49406bf00 tx=0x7fc48402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=295c1bfa96f1f3a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:07.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.381+0000 7fc492a00640 1 -- 172.21.15.90:0/3266314454 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc484036070 con 0x7fc49415ad40 2024-04-25T01:18:07.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.381+0000 7fc492a00640 1 -- 172.21.15.90:0/3266314454 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc484031600 con 0x7fc49415ad40 2024-04-25T01:18:07.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.381+0000 7fc492a00640 1 -- 172.21.15.90:0/3266314454 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4840319d0 con 0x7fc49415ad40 2024-04-25T01:18:07.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.381+0000 7fc498a00640 1 -- 172.21.15.90:0/3266314454 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc49415ad40 msgr2=0x7fc49406b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:07.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.381+0000 7fc498a00640 1 --2- 172.21.15.90:0/3266314454 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc49415ad40 0x7fc49406b120 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc49406bf00 tx=0x7fc48402a6f0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.381+0000 7fc498a00640 1 -- 172.21.15.90:0/3266314454 shutdown_connections 2024-04-25T01:18:07.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.381+0000 7fc498a00640 1 --2- 172.21.15.90:0/3266314454 >> v2:172.21.15.103:3300/0 conn(0x7fc4940cf520 0x7fc4940ae730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.381+0000 7fc498a00640 1 --2- 172.21.15.90:0/3266314454 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc49415ad40 0x7fc49406b120 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.381+0000 7fc498a00640 1 -- 172.21.15.90:0/3266314454 >> 172.21.15.90:0/3266314454 conn(0x7fc4940b1660 msgr2=0x7fc4940b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:07.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.382+0000 7fc498a00640 1 -- 172.21.15.90:0/3266314454 shutdown_connections 2024-04-25T01:18:07.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.382+0000 7fc498a00640 1 -- 172.21.15.90:0/3266314454 wait complete. 2024-04-25T01:18:07.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.384+0000 7fc498a00640 1 Processor -- start 2024-04-25T01:18:07.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.384+0000 7fc498a00640 1 -- start start 2024-04-25T01:18:07.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.385+0000 7fc498a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc49406d910 0x7fc494144c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:07.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.385+0000 7fc498a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4940cf520 0x7fc494142e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:07.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.385+0000 7fc498a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc49415ad40 0x7fc494142460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:07.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.385+0000 7fc498a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc49406b7e0 con 0x7fc4940cf520 2024-04-25T01:18:07.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.385+0000 7fc498a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc49406b660 con 0x7fc49415ad40 2024-04-25T01:18:07.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.385+0000 7fc498a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc4940aec70 con 0x7fc49406d910 2024-04-25T01:18:07.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.385+0000 7fc48aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4940cf520 0x7fc494142e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:07.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.385+0000 7fc48aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4940cf520 0x7fc494142e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33162/0 (socket says 172.21.15.90:33162) 2024-04-25T01:18:07.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.385+0000 7fc48aa00640 1 -- 172.21.15.90:0/2396005032 learned_addr learned my addr 172.21.15.90:0/2396005032 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:07.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.386+0000 7fc493400640 1 --2- 172.21.15.90:0/2396005032 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc49406d910 0x7fc494144c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:07.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.386+0000 7fc48aa00640 1 -- 172.21.15.90:0/2396005032 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc49406d910 msgr2=0x7fc494144c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:07.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.386+0000 7fc48aa00640 1 --2- 172.21.15.90:0/2396005032 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc49406d910 0x7fc494144c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.386+0000 7fc48aa00640 1 -- 172.21.15.90:0/2396005032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc49415ad40 msgr2=0x7fc494142460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:07.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.386+0000 7fc493e00640 1 --2- 172.21.15.90:0/2396005032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc49415ad40 0x7fc494142460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:07.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.386+0000 7fc48aa00640 1 --2- 172.21.15.90:0/2396005032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc49415ad40 0x7fc494142460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.386+0000 7fc48aa00640 1 -- 172.21.15.90:0/2396005032 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4940af670 con 0x7fc4940cf520 2024-04-25T01:18:07.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.386+0000 7fc48aa00640 1 --2- 172.21.15.90:0/2396005032 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4940cf520 0x7fc494142e70 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc47c00c7f0 tx=0x7fc47c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:07.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.386+0000 7fc48be00640 1 -- 172.21.15.90:0/2396005032 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc47c015070 con 0x7fc4940cf520 2024-04-25T01:18:07.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.386+0000 7fc48be00640 1 -- 172.21.15.90:0/2396005032 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc47c004240 con 0x7fc4940cf520 2024-04-25T01:18:07.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.386+0000 7fc48be00640 1 -- 172.21.15.90:0/2396005032 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc47c007620 con 0x7fc4940cf520 2024-04-25T01:18:07.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.386+0000 7fc498a00640 1 -- 172.21.15.90:0/2396005032 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4940af950 con 0x7fc4940cf520 2024-04-25T01:18:07.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.387+0000 7fc498a00640 1 -- 172.21.15.90:0/2396005032 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4940aff10 con 0x7fc4940cf520 2024-04-25T01:18:07.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.388+0000 7fc498a00640 1 -- 172.21.15.90:0/2396005032 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc494075350 con 0x7fc4940cf520 2024-04-25T01:18:07.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.388+0000 7fc48be00640 1 -- 172.21.15.90:0/2396005032 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc47c0079d0 con 0x7fc4940cf520 2024-04-25T01:18:07.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.389+0000 7fc48be00640 1 --2- 172.21.15.90:0/2396005032 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc47006be30 0x7fc47006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:07.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.389+0000 7fc48be00640 1 -- 172.21.15.90:0/2396005032 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6101+0+0 (secure 0 0 0) 0x7fc47c08b440 con 0x7fc4940cf520 2024-04-25T01:18:07.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.390+0000 7fc493400640 1 --2- 172.21.15.90:0/2396005032 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc47006be30 0x7fc47006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:07.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.390+0000 7fc493400640 1 --2- 172.21.15.90:0/2396005032 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc47006be30 0x7fc47006e2f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc484033000 tx=0x7fc484009000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:07.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.394+0000 7fc48be00640 1 -- 172.21.15.90:0/2396005032 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc47c059c60 con 0x7fc4940cf520 2024-04-25T01:18:07.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.577+0000 7fc498a00640 1 -- 172.21.15.90:0/2396005032 --> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc494074c90 con 0x7fc47006be30 2024-04-25T01:18:07.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.597+0000 7fc48be00640 1 -- 172.21.15.90:0/2396005032 <== mgr.4104 v2:172.21.15.90:6832/2198086733 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251951 (secure 0 0 0) 0x7fc494074c90 con 0x7fc47006be30 2024-04-25T01:18:07.584 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:07.585 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-25T01:18:07.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.602+0000 7fc498a00640 1 -- 172.21.15.90:0/2396005032 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc47006be30 msgr2=0x7fc47006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:07.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.602+0000 7fc498a00640 1 --2- 172.21.15.90:0/2396005032 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc47006be30 0x7fc47006e2f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc484033000 tx=0x7fc484009000 comp rx=0 tx=0).stop 2024-04-25T01:18:07.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.603+0000 7fc498a00640 1 -- 172.21.15.90:0/2396005032 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4940cf520 msgr2=0x7fc494142e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:07.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.603+0000 7fc498a00640 1 --2- 172.21.15.90:0/2396005032 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4940cf520 0x7fc494142e70 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc47c00c7f0 tx=0x7fc47c00ccb0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.603+0000 7fc498a00640 1 -- 172.21.15.90:0/2396005032 shutdown_connections 2024-04-25T01:18:07.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.603+0000 7fc498a00640 1 --2- 172.21.15.90:0/2396005032 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc47006be30 0x7fc47006e2f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.603+0000 7fc498a00640 1 --2- 172.21.15.90:0/2396005032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc49415ad40 0x7fc494142460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.603+0000 7fc498a00640 1 --2- 172.21.15.90:0/2396005032 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4940cf520 0x7fc494142e70 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.603+0000 7fc498a00640 1 --2- 172.21.15.90:0/2396005032 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc49406d910 0x7fc494144c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.603+0000 7fc498a00640 1 -- 172.21.15.90:0/2396005032 >> 172.21.15.90:0/2396005032 conn(0x7fc4940b1660 msgr2=0x7fc4940b2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:07.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.603+0000 7fc498a00640 1 -- 172.21.15.90:0/2396005032 shutdown_connections 2024-04-25T01:18:07.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.603+0000 7fc498a00640 1 -- 172.21.15.90:0/2396005032 wait complete. 2024-04-25T01:18:07.604 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2024-04-25T01:18:05.859512+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":78,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220976,"kb_used_data":4796,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753744,"statfs":{"total":773094113280,"available":772867833856,"internally_reserved":0,"allocated":4911104,"data_stored":2755280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.646311"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.292064+0000","last_change":"2024-04-25T01:18:03.253664+0000","last_active":"2024-04-25T01:18:04.292064+0000","last_peered":"2024-04-25T01:18:04.292064+0000","last_clean":"2024-04-25T01:18:04.292064+0000","last_became_active":"2024-04-25T01:18:03.253362+0000","last_became_peered":"2024-04-25T01:18:03.253362+0000","last_unstale":"2024-04-25T01:18:04.292064+0000","last_undegraded":"2024-04-25T01:18:04.292064+0000","last_fullsized":"2024-04-25T01:18:04.292064+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:39.462581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.260823+0000","last_change":"2024-04-25T01:18:03.247432+0000","last_active":"2024-04-25T01:18:04.260823+0000","last_peered":"2024-04-25T01:18:04.260823+0000","last_clean":"2024-04-25T01:18:04.260823+0000","last_became_active":"2024-04-25T01:18:03.247132+0000","last_became_peered":"2024-04-25T01:18:03.247132+0000","last_unstale":"2024-04-25T01:18:04.260823+0000","last_undegraded":"2024-04-25T01:18:04.260823+0000","last_fullsized":"2024-04-25T01:18:04.260823+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:56.294412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291402+0000","last_change":"2024-04-25T01:18:03.267777+0000","last_active":"2024-04-25T01:18:04.291402+0000","last_peered":"2024-04-25T01:18:04.291402+0000","last_clean":"2024-04-25T01:18:04.291402+0000","last_became_active":"2024-04-25T01:18:03.267444+0000","last_became_peered":"2024-04-25T01:18:03.267444+0000","last_unstale":"2024-04-25T01:18:04.291402+0000","last_undegraded":"2024-04-25T01:18:04.291402+0000","last_fullsized":"2024-04-25T01:18:04.291402+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:38:56.106596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291936+0000","last_change":"2024-04-25T01:18:03.260302+0000","last_active":"2024-04-25T01:18:04.291936+0000","last_peered":"2024-04-25T01:18:04.291936+0000","last_clean":"2024-04-25T01:18:04.291936+0000","last_became_active":"2024-04-25T01:18:03.259967+0000","last_became_peered":"2024-04-25T01:18:03.259967+0000","last_unstale":"2024-04-25T01:18:04.291936+0000","last_undegraded":"2024-04-25T01:18:04.291936+0000","last_fullsized":"2024-04-25T01:18:04.291936+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:23:03.906624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.263671+0000","last_change":"2024-04-25T01:18:03.257452+0000","last_active":"2024-04-25T01:18:04.263671+0000","last_peered":"2024-04-25T01:18:04.263671+0000","last_clean":"2024-04-25T01:18:04.263671+0000","last_became_active":"2024-04-25T01:18:03.256443+0000","last_became_peered":"2024-04-25T01:18:03.256443+0000","last_unstale":"2024-04-25T01:18:04.263671+0000","last_undegraded":"2024-04-25T01:18:04.263671+0000","last_fullsized":"2024-04-25T01:18:04.263671+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:53:48.526435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.260073+0000","last_change":"2024-04-25T01:18:03.242630+0000","last_active":"2024-04-25T01:18:04.260073+0000","last_peered":"2024-04-25T01:18:04.260073+0000","last_clean":"2024-04-25T01:18:04.260073+0000","last_became_active":"2024-04-25T01:18:03.242276+0000","last_became_peered":"2024-04-25T01:18:03.242276+0000","last_unstale":"2024-04-25T01:18:04.260073+0000","last_undegraded":"2024-04-25T01:18:04.260073+0000","last_fullsized":"2024-04-25T01:18:04.260073+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:13.740974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.290456+0000","last_change":"2024-04-25T01:18:03.257363+0000","last_active":"2024-04-25T01:18:04.290456+0000","last_peered":"2024-04-25T01:18:04.290456+0000","last_clean":"2024-04-25T01:18:04.290456+0000","last_became_active":"2024-04-25T01:18:03.256685+0000","last_became_peered":"2024-04-25T01:18:03.256685+0000","last_unstale":"2024-04-25T01:18:04.290456+0000","last_undegraded":"2024-04-25T01:18:04.290456+0000","last_fullsized":"2024-04-25T01:18:04.290456+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:10:28.460600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.292081+0000","last_change":"2024-04-25T01:18:03.274858+0000","last_active":"2024-04-25T01:18:04.292081+0000","last_peered":"2024-04-25T01:18:04.292081+0000","last_clean":"2024-04-25T01:18:04.292081+0000","last_became_active":"2024-04-25T01:18:03.273775+0000","last_became_peered":"2024-04-25T01:18:03.273775+0000","last_unstale":"2024-04-25T01:18:04.292081+0000","last_undegraded":"2024-04-25T01:18:04.292081+0000","last_fullsized":"2024-04-25T01:18:04.292081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:01:09.052002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.263741+0000","last_change":"2024-04-25T01:18:03.265545+0000","last_active":"2024-04-25T01:18:04.263741+0000","last_peered":"2024-04-25T01:18:04.263741+0000","last_clean":"2024-04-25T01:18:04.263741+0000","last_became_active":"2024-04-25T01:18:03.265251+0000","last_became_peered":"2024-04-25T01:18:03.265251+0000","last_unstale":"2024-04-25T01:18:04.263741+0000","last_undegraded":"2024-04-25T01:18:04.263741+0000","last_fullsized":"2024-04-25T01:18:04.263741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:10:28.460600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.290660+0000","last_change":"2024-04-25T01:18:03.267089+0000","last_active":"2024-04-25T01:18:04.290660+0000","last_peered":"2024-04-25T01:18:04.290660+0000","last_clean":"2024-04-25T01:18:04.290660+0000","last_became_active":"2024-04-25T01:18:03.266799+0000","last_became_peered":"2024-04-25T01:18:03.266799+0000","last_unstale":"2024-04-25T01:18:04.290660+0000","last_undegraded":"2024-04-25T01:18:04.290660+0000","last_fullsized":"2024-04-25T01:18:04.290660+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:22:35.122721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.267246+0000","last_change":"2024-04-25T01:18:03.276878+0000","last_active":"2024-04-25T01:18:04.267246+0000","last_peered":"2024-04-25T01:18:04.267246+0000","last_clean":"2024-04-25T01:18:04.267246+0000","last_became_active":"2024-04-25T01:18:03.276396+0000","last_became_peered":"2024-04-25T01:18:03.276396+0000","last_unstale":"2024-04-25T01:18:04.267246+0000","last_undegraded":"2024-04-25T01:18:04.267246+0000","last_fullsized":"2024-04-25T01:18:04.267246+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:40.084475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.263218+0000","last_change":"2024-04-25T01:18:03.249998+0000","last_active":"2024-04-25T01:18:04.263218+0000","last_peered":"2024-04-25T01:18:04.263218+0000","last_clean":"2024-04-25T01:18:04.263218+0000","last_became_active":"2024-04-25T01:18:03.249714+0000","last_became_peered":"2024-04-25T01:18:03.249714+0000","last_unstale":"2024-04-25T01:18:04.263218+0000","last_undegraded":"2024-04-25T01:18:04.263218+0000","last_fullsized":"2024-04-25T01:18:04.263218+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:23:00.911957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.290448+0000","last_change":"2024-04-25T01:18:03.256824+0000","last_active":"2024-04-25T01:18:04.290448+0000","last_peered":"2024-04-25T01:18:04.290448+0000","last_clean":"2024-04-25T01:18:04.290448+0000","last_became_active":"2024-04-25T01:18:03.256477+0000","last_became_peered":"2024-04-25T01:18:03.256477+0000","last_unstale":"2024-04-25T01:18:04.290448+0000","last_undegraded":"2024-04-25T01:18:04.290448+0000","last_fullsized":"2024-04-25T01:18:04.290448+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:09:22.874364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.290651+0000","last_change":"2024-04-25T01:18:03.265116+0000","last_active":"2024-04-25T01:18:04.290651+0000","last_peered":"2024-04-25T01:18:04.290651+0000","last_clean":"2024-04-25T01:18:04.290651+0000","last_became_active":"2024-04-25T01:18:03.264863+0000","last_became_peered":"2024-04-25T01:18:03.264863+0000","last_unstale":"2024-04-25T01:18:04.290651+0000","last_undegraded":"2024-04-25T01:18:04.290651+0000","last_fullsized":"2024-04-25T01:18:04.290651+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:11.435894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.253875+0000","last_change":"2024-04-25T01:18:03.258504+0000","last_active":"2024-04-25T01:18:04.253875+0000","last_peered":"2024-04-25T01:18:04.253875+0000","last_clean":"2024-04-25T01:18:04.253875+0000","last_became_active":"2024-04-25T01:18:03.258242+0000","last_became_peered":"2024-04-25T01:18:03.258242+0000","last_unstale":"2024-04-25T01:18:04.253875+0000","last_undegraded":"2024-04-25T01:18:04.253875+0000","last_fullsized":"2024-04-25T01:18:04.253875+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:02:01.161670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.289891+0000","last_change":"2024-04-25T01:18:03.261774+0000","last_active":"2024-04-25T01:18:04.289891+0000","last_peered":"2024-04-25T01:18:04.289891+0000","last_clean":"2024-04-25T01:18:04.289891+0000","last_became_active":"2024-04-25T01:18:03.261428+0000","last_became_peered":"2024-04-25T01:18:03.261428+0000","last_unstale":"2024-04-25T01:18:04.289891+0000","last_undegraded":"2024-04-25T01:18:04.289891+0000","last_fullsized":"2024-04-25T01:18:04.289891+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:16.174537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.254754+0000","last_change":"2024-04-25T01:18:03.273618+0000","last_active":"2024-04-25T01:18:04.254754+0000","last_peered":"2024-04-25T01:18:04.254754+0000","last_clean":"2024-04-25T01:18:04.254754+0000","last_became_active":"2024-04-25T01:18:03.273161+0000","last_became_peered":"2024-04-25T01:18:03.273161+0000","last_unstale":"2024-04-25T01:18:04.254754+0000","last_undegraded":"2024-04-25T01:18:04.254754+0000","last_fullsized":"2024-04-25T01:18:04.254754+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:32:52.505647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.254583+0000","last_change":"2024-04-25T01:18:03.265079+0000","last_active":"2024-04-25T01:18:04.254583+0000","last_peered":"2024-04-25T01:18:04.254583+0000","last_clean":"2024-04-25T01:18:04.254583+0000","last_became_active":"2024-04-25T01:18:03.264708+0000","last_became_peered":"2024-04-25T01:18:03.264708+0000","last_unstale":"2024-04-25T01:18:04.254583+0000","last_undegraded":"2024-04-25T01:18:04.254583+0000","last_fullsized":"2024-04-25T01:18:04.254583+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:57:03.302298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.261281+0000","last_change":"2024-04-25T01:18:03.257519+0000","last_active":"2024-04-25T01:18:04.261281+0000","last_peered":"2024-04-25T01:18:04.261281+0000","last_clean":"2024-04-25T01:18:04.261281+0000","last_became_active":"2024-04-25T01:18:03.257225+0000","last_became_peered":"2024-04-25T01:18:03.257225+0000","last_unstale":"2024-04-25T01:18:04.261281+0000","last_undegraded":"2024-04-25T01:18:04.261281+0000","last_fullsized":"2024-04-25T01:18:04.261281+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:50:41.389115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.266754+0000","last_change":"2024-04-25T01:18:03.267873+0000","last_active":"2024-04-25T01:18:04.266754+0000","last_peered":"2024-04-25T01:18:04.266754+0000","last_clean":"2024-04-25T01:18:04.266754+0000","last_became_active":"2024-04-25T01:18:03.267601+0000","last_became_peered":"2024-04-25T01:18:03.267601+0000","last_unstale":"2024-04-25T01:18:04.266754+0000","last_undegraded":"2024-04-25T01:18:04.266754+0000","last_fullsized":"2024-04-25T01:18:04.266754+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:35:34.926511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.266736+0000","last_change":"2024-04-25T01:18:03.256260+0000","last_active":"2024-04-25T01:18:04.266736+0000","last_peered":"2024-04-25T01:18:04.266736+0000","last_clean":"2024-04-25T01:18:04.266736+0000","last_became_active":"2024-04-25T01:18:03.255488+0000","last_became_peered":"2024-04-25T01:18:03.255488+0000","last_unstale":"2024-04-25T01:18:04.266736+0000","last_undegraded":"2024-04-25T01:18:04.266736+0000","last_fullsized":"2024-04-25T01:18:04.266736+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:19.882784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.266737+0000","last_change":"2024-04-25T01:18:03.262912+0000","last_active":"2024-04-25T01:18:04.266737+0000","last_peered":"2024-04-25T01:18:04.266737+0000","last_clean":"2024-04-25T01:18:04.266737+0000","last_became_active":"2024-04-25T01:18:03.262598+0000","last_became_peered":"2024-04-25T01:18:03.262598+0000","last_unstale":"2024-04-25T01:18:04.266737+0000","last_undegraded":"2024-04-25T01:18:04.266737+0000","last_fullsized":"2024-04-25T01:18:04.266737+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:19.060283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.253929+0000","last_change":"2024-04-25T01:18:03.259084+0000","last_active":"2024-04-25T01:18:04.253929+0000","last_peered":"2024-04-25T01:18:04.253929+0000","last_clean":"2024-04-25T01:18:04.253929+0000","last_became_active":"2024-04-25T01:18:03.258651+0000","last_became_peered":"2024-04-25T01:18:03.258651+0000","last_unstale":"2024-04-25T01:18:04.253929+0000","last_undegraded":"2024-04-25T01:18:04.253929+0000","last_fullsized":"2024-04-25T01:18:04.253929+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:42:33.542272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291302+0000","last_change":"2024-04-25T01:18:03.270002+0000","last_active":"2024-04-25T01:18:04.291302+0000","last_peered":"2024-04-25T01:18:04.291302+0000","last_clean":"2024-04-25T01:18:04.291302+0000","last_became_active":"2024-04-25T01:18:03.269711+0000","last_became_peered":"2024-04-25T01:18:03.269711+0000","last_unstale":"2024-04-25T01:18:04.291302+0000","last_undegraded":"2024-04-25T01:18:04.291302+0000","last_fullsized":"2024-04-25T01:18:04.291302+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:06:25.097573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291532+0000","last_change":"2024-04-25T01:18:03.265266+0000","last_active":"2024-04-25T01:18:04.291532+0000","last_peered":"2024-04-25T01:18:04.291532+0000","last_clean":"2024-04-25T01:18:04.291532+0000","last_became_active":"2024-04-25T01:18:03.264973+0000","last_became_peered":"2024-04-25T01:18:03.264973+0000","last_unstale":"2024-04-25T01:18:04.291532+0000","last_undegraded":"2024-04-25T01:18:04.291532+0000","last_fullsized":"2024-04-25T01:18:04.291532+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:50:14.498011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291794+0000","last_change":"2024-04-25T01:18:03.263008+0000","last_active":"2024-04-25T01:18:04.291794+0000","last_peered":"2024-04-25T01:18:04.291794+0000","last_clean":"2024-04-25T01:18:04.291794+0000","last_became_active":"2024-04-25T01:18:03.262081+0000","last_became_peered":"2024-04-25T01:18:03.262081+0000","last_unstale":"2024-04-25T01:18:04.291794+0000","last_undegraded":"2024-04-25T01:18:04.291794+0000","last_fullsized":"2024-04-25T01:18:04.291794+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:03:36.196096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.267231+0000","last_change":"2024-04-25T01:18:03.263526+0000","last_active":"2024-04-25T01:18:04.267231+0000","last_peered":"2024-04-25T01:18:04.267231+0000","last_clean":"2024-04-25T01:18:04.267231+0000","last_became_active":"2024-04-25T01:18:03.258646+0000","last_became_peered":"2024-04-25T01:18:03.258646+0000","last_unstale":"2024-04-25T01:18:04.267231+0000","last_undegraded":"2024-04-25T01:18:04.267231+0000","last_fullsized":"2024-04-25T01:18:04.267231+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:30:00.416602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.290316+0000","last_change":"2024-04-25T01:18:03.267937+0000","last_active":"2024-04-25T01:18:04.290316+0000","last_peered":"2024-04-25T01:18:04.290316+0000","last_clean":"2024-04-25T01:18:04.290316+0000","last_became_active":"2024-04-25T01:18:03.266925+0000","last_became_peered":"2024-04-25T01:18:03.266925+0000","last_unstale":"2024-04-25T01:18:04.290316+0000","last_undegraded":"2024-04-25T01:18:04.290316+0000","last_fullsized":"2024-04-25T01:18:04.290316+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:53:48.526435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.254161+0000","last_change":"2024-04-25T01:18:03.253210+0000","last_active":"2024-04-25T01:18:04.254161+0000","last_peered":"2024-04-25T01:18:04.254161+0000","last_clean":"2024-04-25T01:18:04.254161+0000","last_became_active":"2024-04-25T01:18:03.252930+0000","last_became_peered":"2024-04-25T01:18:03.252930+0000","last_unstale":"2024-04-25T01:18:04.254161+0000","last_undegraded":"2024-04-25T01:18:04.254161+0000","last_fullsized":"2024-04-25T01:18:04.254161+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:02:03.876956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.254246+0000","last_change":"2024-04-25T01:18:03.255049+0000","last_active":"2024-04-25T01:18:04.254246+0000","last_peered":"2024-04-25T01:18:04.254246+0000","last_clean":"2024-04-25T01:18:04.254246+0000","last_became_active":"2024-04-25T01:18:03.254795+0000","last_became_peered":"2024-04-25T01:18:03.254795+0000","last_unstale":"2024-04-25T01:18:04.254246+0000","last_undegraded":"2024-04-25T01:18:04.254246+0000","last_fullsized":"2024-04-25T01:18:04.254246+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:39:30.588445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.290867+0000","last_change":"2024-04-25T01:18:03.257602+0000","last_active":"2024-04-25T01:18:04.290867+0000","last_peered":"2024-04-25T01:18:04.290867+0000","last_clean":"2024-04-25T01:18:04.290867+0000","last_became_active":"2024-04-25T01:18:03.257303+0000","last_became_peered":"2024-04-25T01:18:03.257303+0000","last_unstale":"2024-04-25T01:18:04.290867+0000","last_undegraded":"2024-04-25T01:18:04.290867+0000","last_fullsized":"2024-04-25T01:18:04.290867+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:20:18.432704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291746+0000","last_change":"2024-04-25T01:18:03.274018+0000","last_active":"2024-04-25T01:18:04.291746+0000","last_peered":"2024-04-25T01:18:04.291746+0000","last_clean":"2024-04-25T01:18:04.291746+0000","last_became_active":"2024-04-25T01:18:03.273719+0000","last_became_peered":"2024-04-25T01:18:03.273719+0000","last_unstale":"2024-04-25T01:18:04.291746+0000","last_undegraded":"2024-04-25T01:18:04.291746+0000","last_fullsized":"2024-04-25T01:18:04.291746+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:41:12.326642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.254726+0000","last_change":"2024-04-25T01:18:03.264199+0000","last_active":"2024-04-25T01:18:04.254726+0000","last_peered":"2024-04-25T01:18:04.254726+0000","last_clean":"2024-04-25T01:18:04.254726+0000","last_became_active":"2024-04-25T01:18:03.260827+0000","last_became_peered":"2024-04-25T01:18:03.260827+0000","last_unstale":"2024-04-25T01:18:04.254726+0000","last_undegraded":"2024-04-25T01:18:04.254726+0000","last_fullsized":"2024-04-25T01:18:04.254726+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:24:54.943447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"39'4","reported_seq":18,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291486+0000","last_change":"2024-04-25T01:18:02.263556+0000","last_active":"2024-04-25T01:18:04.291486+0000","last_peered":"2024-04-25T01:18:04.291486+0000","last_clean":"2024-04-25T01:18:04.291486+0000","last_became_active":"2024-04-25T01:18:02.263123+0000","last_became_peered":"2024-04-25T01:18:02.263123+0000","last_unstale":"2024-04-25T01:18:04.291486+0000","last_undegraded":"2024-04-25T01:18:04.291486+0000","last_fullsized":"2024-04-25T01:18:04.291486+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:09:21.858773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.261110+0000","last_change":"2024-04-25T01:18:03.269144+0000","last_active":"2024-04-25T01:18:04.261110+0000","last_peered":"2024-04-25T01:18:04.261110+0000","last_clean":"2024-04-25T01:18:04.261110+0000","last_became_active":"2024-04-25T01:18:03.268851+0000","last_became_peered":"2024-04-25T01:18:03.268851+0000","last_unstale":"2024-04-25T01:18:04.261110+0000","last_undegraded":"2024-04-25T01:18:04.261110+0000","last_fullsized":"2024-04-25T01:18:04.261110+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:50:41.389115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"39'1","reported_seq":15,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.261229+0000","last_change":"2024-04-25T01:18:02.255987+0000","last_active":"2024-04-25T01:18:04.261229+0000","last_peered":"2024-04-25T01:18:04.261229+0000","last_clean":"2024-04-25T01:18:04.261229+0000","last_became_active":"2024-04-25T01:18:02.255540+0000","last_became_peered":"2024-04-25T01:18:02.255540+0000","last_unstale":"2024-04-25T01:18:04.261229+0000","last_undegraded":"2024-04-25T01:18:04.261229+0000","last_fullsized":"2024-04-25T01:18:04.261229+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:38:12.422806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.254737+0000","last_change":"2024-04-25T01:18:03.260381+0000","last_active":"2024-04-25T01:18:04.254737+0000","last_peered":"2024-04-25T01:18:04.254737+0000","last_clean":"2024-04-25T01:18:04.254737+0000","last_became_active":"2024-04-25T01:18:03.260114+0000","last_became_peered":"2024-04-25T01:18:03.260114+0000","last_unstale":"2024-04-25T01:18:04.254737+0000","last_undegraded":"2024-04-25T01:18:04.254737+0000","last_fullsized":"2024-04-25T01:18:04.254737+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:11.366143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"39'2","reported_seq":16,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291743+0000","last_change":"2024-04-25T01:18:02.270338+0000","last_active":"2024-04-25T01:18:04.291743+0000","last_peered":"2024-04-25T01:18:04.291743+0000","last_clean":"2024-04-25T01:18:04.291743+0000","last_became_active":"2024-04-25T01:18:02.270063+0000","last_became_peered":"2024-04-25T01:18:02.270063+0000","last_unstale":"2024-04-25T01:18:04.291743+0000","last_undegraded":"2024-04-25T01:18:04.291743+0000","last_fullsized":"2024-04-25T01:18:04.291743+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:25:09.988571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.261521+0000","last_change":"2024-04-25T01:18:03.264063+0000","last_active":"2024-04-25T01:18:04.261521+0000","last_peered":"2024-04-25T01:18:04.261521+0000","last_clean":"2024-04-25T01:18:04.261521+0000","last_became_active":"2024-04-25T01:18:03.263708+0000","last_became_peered":"2024-04-25T01:18:03.263708+0000","last_unstale":"2024-04-25T01:18:04.261521+0000","last_undegraded":"2024-04-25T01:18:04.261521+0000","last_fullsized":"2024-04-25T01:18:04.261521+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:09:35.248123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"39'2","reported_seq":16,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291260+0000","last_change":"2024-04-25T01:18:02.263250+0000","last_active":"2024-04-25T01:18:04.291260+0000","last_peered":"2024-04-25T01:18:04.291260+0000","last_clean":"2024-04-25T01:18:04.291260+0000","last_became_active":"2024-04-25T01:18:02.262763+0000","last_became_peered":"2024-04-25T01:18:02.262763+0000","last_unstale":"2024-04-25T01:18:04.291260+0000","last_undegraded":"2024-04-25T01:18:04.291260+0000","last_fullsized":"2024-04-25T01:18:04.291260+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:22:34.107130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.292170+0000","last_change":"2024-04-25T01:18:03.250303+0000","last_active":"2024-04-25T01:18:04.292170+0000","last_peered":"2024-04-25T01:18:04.292170+0000","last_clean":"2024-04-25T01:18:04.292170+0000","last_became_active":"2024-04-25T01:18:03.249995+0000","last_became_peered":"2024-04-25T01:18:03.249995+0000","last_unstale":"2024-04-25T01:18:04.292170+0000","last_undegraded":"2024-04-25T01:18:04.292170+0000","last_fullsized":"2024-04-25T01:18:04.292170+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:29:13.129901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"39'1","reported_seq":15,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291012+0000","last_change":"2024-04-25T01:18:02.270642+0000","last_active":"2024-04-25T01:18:04.291012+0000","last_peered":"2024-04-25T01:18:04.291012+0000","last_clean":"2024-04-25T01:18:04.291012+0000","last_became_active":"2024-04-25T01:18:02.270383+0000","last_became_peered":"2024-04-25T01:18:02.270383+0000","last_unstale":"2024-04-25T01:18:04.291012+0000","last_undegraded":"2024-04-25T01:18:04.291012+0000","last_fullsized":"2024-04-25T01:18:04.291012+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:20:17.417113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.266557+0000","last_change":"2024-04-25T01:18:03.262229+0000","last_active":"2024-04-25T01:18:04.266557+0000","last_peered":"2024-04-25T01:18:04.266557+0000","last_clean":"2024-04-25T01:18:04.266557+0000","last_became_active":"2024-04-25T01:18:03.256356+0000","last_became_peered":"2024-04-25T01:18:03.256356+0000","last_unstale":"2024-04-25T01:18:04.266557+0000","last_undegraded":"2024-04-25T01:18:04.266557+0000","last_fullsized":"2024-04-25T01:18:04.266557+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:55:31.639411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"39'1","reported_seq":15,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.290079+0000","last_change":"2024-04-25T01:18:02.267951+0000","last_active":"2024-04-25T01:18:04.290079+0000","last_peered":"2024-04-25T01:18:04.290079+0000","last_clean":"2024-04-25T01:18:04.290079+0000","last_became_active":"2024-04-25T01:18:02.267086+0000","last_became_peered":"2024-04-25T01:18:02.267086+0000","last_unstale":"2024-04-25T01:18:04.290079+0000","last_undegraded":"2024-04-25T01:18:04.290079+0000","last_fullsized":"2024-04-25T01:18:04.290079+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:03:28.616338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.254418+0000","last_change":"2024-04-25T01:18:03.265044+0000","last_active":"2024-04-25T01:18:04.254418+0000","last_peered":"2024-04-25T01:18:04.254418+0000","last_clean":"2024-04-25T01:18:04.254418+0000","last_became_active":"2024-04-25T01:18:03.264645+0000","last_became_peered":"2024-04-25T01:18:03.264645+0000","last_unstale":"2024-04-25T01:18:04.254418+0000","last_undegraded":"2024-04-25T01:18:04.254418+0000","last_fullsized":"2024-04-25T01:18:04.254418+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:20:51.212419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.290999+0000","last_change":"2024-04-25T01:18:02.262809+0000","last_active":"2024-04-25T01:18:04.290999+0000","last_peered":"2024-04-25T01:18:04.290999+0000","last_clean":"2024-04-25T01:18:04.290999+0000","last_became_active":"2024-04-25T01:18:02.261620+0000","last_became_peered":"2024-04-25T01:18:02.261620+0000","last_unstale":"2024-04-25T01:18:04.290999+0000","last_undegraded":"2024-04-25T01:18:04.290999+0000","last_fullsized":"2024-04-25T01:18:04.290999+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:12.817891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.261149+0000","last_change":"2024-04-25T01:18:03.261875+0000","last_active":"2024-04-25T01:18:04.261149+0000","last_peered":"2024-04-25T01:18:04.261149+0000","last_clean":"2024-04-25T01:18:04.261149+0000","last_became_active":"2024-04-25T01:18:03.261413+0000","last_became_peered":"2024-04-25T01:18:03.261413+0000","last_unstale":"2024-04-25T01:18:04.261149+0000","last_undegraded":"2024-04-25T01:18:04.261149+0000","last_fullsized":"2024-04-25T01:18:04.261149+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:58:30.612559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.289931+0000","last_change":"2024-04-25T01:18:02.257390+0000","last_active":"2024-04-25T01:18:04.289931+0000","last_peered":"2024-04-25T01:18:04.289931+0000","last_clean":"2024-04-25T01:18:04.289931+0000","last_became_active":"2024-04-25T01:18:02.257118+0000","last_became_peered":"2024-04-25T01:18:02.257118+0000","last_unstale":"2024-04-25T01:18:04.289931+0000","last_undegraded":"2024-04-25T01:18:04.289931+0000","last_fullsized":"2024-04-25T01:18:04.289931+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:49:10.075668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291638+0000","last_change":"2024-04-25T01:18:03.260224+0000","last_active":"2024-04-25T01:18:04.291638+0000","last_peered":"2024-04-25T01:18:04.291638+0000","last_clean":"2024-04-25T01:18:04.291638+0000","last_became_active":"2024-04-25T01:18:03.259886+0000","last_became_peered":"2024-04-25T01:18:03.259886+0000","last_unstale":"2024-04-25T01:18:04.291638+0000","last_undegraded":"2024-04-25T01:18:04.291638+0000","last_fullsized":"2024-04-25T01:18:04.291638+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:10:58.007657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.292274+0000","last_change":"2024-04-25T01:18:02.265416+0000","last_active":"2024-04-25T01:18:04.292274+0000","last_peered":"2024-04-25T01:18:04.292274+0000","last_clean":"2024-04-25T01:18:04.292274+0000","last_became_active":"2024-04-25T01:18:02.265123+0000","last_became_peered":"2024-04-25T01:18:02.265123+0000","last_unstale":"2024-04-25T01:18:04.292274+0000","last_undegraded":"2024-04-25T01:18:04.292274+0000","last_fullsized":"2024-04-25T01:18:04.292274+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:18.369587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291157+0000","last_change":"2024-04-25T01:18:03.279931+0000","last_active":"2024-04-25T01:18:04.291157+0000","last_peered":"2024-04-25T01:18:04.291157+0000","last_clean":"2024-04-25T01:18:04.291157+0000","last_became_active":"2024-04-25T01:18:03.279643+0000","last_became_peered":"2024-04-25T01:18:03.279643+0000","last_unstale":"2024-04-25T01:18:04.291157+0000","last_undegraded":"2024-04-25T01:18:04.291157+0000","last_fullsized":"2024-04-25T01:18:04.291157+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:19.385178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.254696+0000","last_change":"2024-04-25T01:18:02.256098+0000","last_active":"2024-04-25T01:18:04.254696+0000","last_peered":"2024-04-25T01:18:04.254696+0000","last_clean":"2024-04-25T01:18:04.254696+0000","last_became_active":"2024-04-25T01:18:02.255609+0000","last_became_peered":"2024-04-25T01:18:02.255609+0000","last_unstale":"2024-04-25T01:18:04.254696+0000","last_undegraded":"2024-04-25T01:18:04.254696+0000","last_fullsized":"2024-04-25T01:18:04.254696+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:46:28.843042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.292099+0000","last_change":"2024-04-25T01:18:03.274406+0000","last_active":"2024-04-25T01:18:04.292099+0000","last_peered":"2024-04-25T01:18:04.292099+0000","last_clean":"2024-04-25T01:18:04.292099+0000","last_became_active":"2024-04-25T01:18:03.274186+0000","last_became_peered":"2024-04-25T01:18:03.274186+0000","last_unstale":"2024-04-25T01:18:04.292099+0000","last_undegraded":"2024-04-25T01:18:04.292099+0000","last_fullsized":"2024-04-25T01:18:04.292099+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:34:33.950222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"39'3","reported_seq":17,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.254272+0000","last_change":"2024-04-25T01:18:02.263678+0000","last_active":"2024-04-25T01:18:04.254272+0000","last_peered":"2024-04-25T01:18:04.254272+0000","last_clean":"2024-04-25T01:18:04.254272+0000","last_became_active":"2024-04-25T01:18:02.263328+0000","last_became_peered":"2024-04-25T01:18:02.263328+0000","last_unstale":"2024-04-25T01:18:04.254272+0000","last_undegraded":"2024-04-25T01:18:04.254272+0000","last_fullsized":"2024-04-25T01:18:04.254272+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:30:37.768204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291123+0000","last_change":"2024-04-25T01:18:03.257743+0000","last_active":"2024-04-25T01:18:04.291123+0000","last_peered":"2024-04-25T01:18:04.291123+0000","last_clean":"2024-04-25T01:18:04.291123+0000","last_became_active":"2024-04-25T01:18:03.257421+0000","last_became_peered":"2024-04-25T01:18:03.257421+0000","last_unstale":"2024-04-25T01:18:04.291123+0000","last_undegraded":"2024-04-25T01:18:04.291123+0000","last_fullsized":"2024-04-25T01:18:04.291123+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:02:29.348596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.14","version":"39'6","reported_seq":20,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.292088+0000","last_change":"2024-04-25T01:18:02.269681+0000","last_active":"2024-04-25T01:18:04.292088+0000","last_peered":"2024-04-25T01:18:04.292088+0000","last_clean":"2024-04-25T01:18:04.292088+0000","last_became_active":"2024-04-25T01:18:02.268702+0000","last_became_peered":"2024-04-25T01:18:02.268702+0000","last_unstale":"2024-04-25T01:18:04.292088+0000","last_undegraded":"2024-04-25T01:18:04.292088+0000","last_fullsized":"2024-04-25T01:18:04.292088+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:50:13.482420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.292471+0000","last_change":"2024-04-25T01:18:03.269967+0000","last_active":"2024-04-25T01:18:04.292471+0000","last_peered":"2024-04-25T01:18:04.292471+0000","last_clean":"2024-04-25T01:18:04.292471+0000","last_became_active":"2024-04-25T01:18:03.269663+0000","last_became_peered":"2024-04-25T01:18:03.269663+0000","last_unstale":"2024-04-25T01:18:04.292471+0000","last_undegraded":"2024-04-25T01:18:04.292471+0000","last_fullsized":"2024-04-25T01:18:04.292471+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:53:56.635142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"39'2","reported_seq":16,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291062+0000","last_change":"2024-04-25T01:18:02.272092+0000","last_active":"2024-04-25T01:18:04.291062+0000","last_peered":"2024-04-25T01:18:04.291062+0000","last_clean":"2024-04-25T01:18:04.291062+0000","last_became_active":"2024-04-25T01:18:02.271767+0000","last_became_peered":"2024-04-25T01:18:02.271767+0000","last_unstale":"2024-04-25T01:18:04.291062+0000","last_undegraded":"2024-04-25T01:18:04.291062+0000","last_fullsized":"2024-04-25T01:18:04.291062+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:10.420303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.261347+0000","last_change":"2024-04-25T01:18:03.258055+0000","last_active":"2024-04-25T01:18:04.261347+0000","last_peered":"2024-04-25T01:18:04.261347+0000","last_clean":"2024-04-25T01:18:04.261347+0000","last_became_active":"2024-04-25T01:18:03.257517+0000","last_became_peered":"2024-04-25T01:18:03.257517+0000","last_unstale":"2024-04-25T01:18:04.261347+0000","last_undegraded":"2024-04-25T01:18:04.261347+0000","last_fullsized":"2024-04-25T01:18:04.261347+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:14:54.012578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.260652+0000","last_change":"2024-04-25T01:18:02.258795+0000","last_active":"2024-04-25T01:18:04.260652+0000","last_peered":"2024-04-25T01:18:04.260652+0000","last_clean":"2024-04-25T01:18:04.260652+0000","last_became_active":"2024-04-25T01:18:02.258494+0000","last_became_peered":"2024-04-25T01:18:02.258494+0000","last_unstale":"2024-04-25T01:18:04.260652+0000","last_undegraded":"2024-04-25T01:18:04.260652+0000","last_fullsized":"2024-04-25T01:18:04.260652+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:16:48.693012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.266960+0000","last_change":"2024-04-25T01:18:03.254569+0000","last_active":"2024-04-25T01:18:04.266960+0000","last_peered":"2024-04-25T01:18:04.266960+0000","last_clean":"2024-04-25T01:18:04.266960+0000","last_became_active":"2024-04-25T01:18:03.254299+0000","last_became_peered":"2024-04-25T01:18:03.254299+0000","last_unstale":"2024-04-25T01:18:04.266960+0000","last_undegraded":"2024-04-25T01:18:04.266960+0000","last_fullsized":"2024-04-25T01:18:04.266960+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:50:36.431482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.254147+0000","last_change":"2024-04-25T01:18:02.254663+0000","last_active":"2024-04-25T01:18:04.254147+0000","last_peered":"2024-04-25T01:18:04.254147+0000","last_clean":"2024-04-25T01:18:04.254147+0000","last_became_active":"2024-04-25T01:18:02.254405+0000","last_became_peered":"2024-04-25T01:18:02.254405+0000","last_unstale":"2024-04-25T01:18:04.254147+0000","last_undegraded":"2024-04-25T01:18:04.254147+0000","last_fullsized":"2024-04-25T01:18:04.254147+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:00:56.792493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.263363+0000","last_change":"2024-04-25T01:18:03.252241+0000","last_active":"2024-04-25T01:18:04.263363+0000","last_peered":"2024-04-25T01:18:04.263363+0000","last_clean":"2024-04-25T01:18:04.263363+0000","last_became_active":"2024-04-25T01:18:03.251874+0000","last_became_peered":"2024-04-25T01:18:03.251874+0000","last_unstale":"2024-04-25T01:18:04.263363+0000","last_undegraded":"2024-04-25T01:18:04.263363+0000","last_fullsized":"2024-04-25T01:18:04.263363+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:16.174537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"39'8","reported_seq":22,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.290580+0000","last_change":"2024-04-25T01:18:02.258502+0000","last_active":"2024-04-25T01:18:04.290580+0000","last_peered":"2024-04-25T01:18:04.290580+0000","last_clean":"2024-04-25T01:18:04.290580+0000","last_became_active":"2024-04-25T01:18:02.258205+0000","last_became_peered":"2024-04-25T01:18:02.258205+0000","last_unstale":"2024-04-25T01:18:04.290580+0000","last_undegraded":"2024-04-25T01:18:04.290580+0000","last_fullsized":"2024-04-25T01:18:04.290580+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:20:28.341160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.260451+0000","last_change":"2024-04-25T01:18:03.275449+0000","last_active":"2024-04-25T01:18:04.260451+0000","last_peered":"2024-04-25T01:18:04.260451+0000","last_clean":"2024-04-25T01:18:04.260451+0000","last_became_active":"2024-04-25T01:18:03.275006+0000","last_became_peered":"2024-04-25T01:18:03.275006+0000","last_unstale":"2024-04-25T01:18:04.260451+0000","last_undegraded":"2024-04-25T01:18:04.260451+0000","last_fullsized":"2024-04-25T01:18:04.260451+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:13.737432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"39'1","reported_seq":15,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291123+0000","last_change":"2024-04-25T01:18:02.259736+0000","last_active":"2024-04-25T01:18:04.291123+0000","last_peered":"2024-04-25T01:18:04.291123+0000","last_clean":"2024-04-25T01:18:04.291123+0000","last_became_active":"2024-04-25T01:18:02.259127+0000","last_became_peered":"2024-04-25T01:18:02.259127+0000","last_unstale":"2024-04-25T01:18:04.291123+0000","last_undegraded":"2024-04-25T01:18:04.291123+0000","last_fullsized":"2024-04-25T01:18:04.291123+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:58:04.484125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.290937+0000","last_change":"2024-04-25T01:18:03.270127+0000","last_active":"2024-04-25T01:18:04.290937+0000","last_peered":"2024-04-25T01:18:04.290937+0000","last_clean":"2024-04-25T01:18:04.290937+0000","last_became_active":"2024-04-25T01:18:03.267471+0000","last_became_peered":"2024-04-25T01:18:03.267471+0000","last_unstale":"2024-04-25T01:18:04.290937+0000","last_undegraded":"2024-04-25T01:18:04.290937+0000","last_fullsized":"2024-04-25T01:18:04.290937+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:30:08.111624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"39'2","reported_seq":16,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.254928+0000","last_change":"2024-04-25T01:18:02.267703+0000","last_active":"2024-04-25T01:18:04.254928+0000","last_peered":"2024-04-25T01:18:04.254928+0000","last_clean":"2024-04-25T01:18:04.254928+0000","last_became_active":"2024-04-25T01:18:02.267415+0000","last_became_peered":"2024-04-25T01:18:02.267415+0000","last_unstale":"2024-04-25T01:18:04.254928+0000","last_undegraded":"2024-04-25T01:18:04.254928+0000","last_fullsized":"2024-04-25T01:18:04.254928+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:09:47.374684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.290879+0000","last_change":"2024-04-25T01:18:03.266233+0000","last_active":"2024-04-25T01:18:04.290879+0000","last_peered":"2024-04-25T01:18:04.290879+0000","last_clean":"2024-04-25T01:18:04.290879+0000","last_became_active":"2024-04-25T01:18:03.265929+0000","last_became_peered":"2024-04-25T01:18:03.265929+0000","last_unstale":"2024-04-25T01:18:04.290879+0000","last_undegraded":"2024-04-25T01:18:04.290879+0000","last_fullsized":"2024-04-25T01:18:04.290879+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:20:29.356751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"39'1","reported_seq":15,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.262922+0000","last_change":"2024-04-25T01:18:02.267511+0000","last_active":"2024-04-25T01:18:04.262922+0000","last_peered":"2024-04-25T01:18:04.262922+0000","last_clean":"2024-04-25T01:18:04.262922+0000","last_became_active":"2024-04-25T01:18:02.266732+0000","last_became_peered":"2024-04-25T01:18:02.266732+0000","last_unstale":"2024-04-25T01:18:04.262922+0000","last_undegraded":"2024-04-25T01:18:04.262922+0000","last_fullsized":"2024-04-25T01:18:04.262922+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:58:04.484125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.262182+0000","last_change":"2024-04-25T01:18:03.255121+0000","last_active":"2024-04-25T01:18:04.262182+0000","last_peered":"2024-04-25T01:18:04.262182+0000","last_clean":"2024-04-25T01:18:04.262182+0000","last_became_active":"2024-04-25T01:18:03.254831+0000","last_became_peered":"2024-04-25T01:18:03.254831+0000","last_unstale":"2024-04-25T01:18:04.262182+0000","last_undegraded":"2024-04-25T01:18:04.262182+0000","last_fullsized":"2024-04-25T01:18:04.262182+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:14:54.012578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.260403+0000","last_change":"2024-04-25T01:18:02.259323+0000","last_active":"2024-04-25T01:18:04.260403+0000","last_peered":"2024-04-25T01:18:04.260403+0000","last_clean":"2024-04-25T01:18:04.260403+0000","last_became_active":"2024-04-25T01:18:02.258937+0000","last_became_peered":"2024-04-25T01:18:02.258937+0000","last_unstale":"2024-04-25T01:18:04.260403+0000","last_undegraded":"2024-04-25T01:18:04.260403+0000","last_fullsized":"2024-04-25T01:18:04.260403+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:12.725383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.253843+0000","last_change":"2024-04-25T01:18:03.253199+0000","last_active":"2024-04-25T01:18:04.253843+0000","last_peered":"2024-04-25T01:18:04.253843+0000","last_clean":"2024-04-25T01:18:04.253843+0000","last_became_active":"2024-04-25T01:18:03.252912+0000","last_became_peered":"2024-04-25T01:18:03.252912+0000","last_unstale":"2024-04-25T01:18:04.253843+0000","last_undegraded":"2024-04-25T01:18:04.253843+0000","last_fullsized":"2024-04-25T01:18:04.253843+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:57:14.448582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"39'2","reported_seq":16,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.259796+0000","last_change":"2024-04-25T01:18:02.267229+0000","last_active":"2024-04-25T01:18:04.259796+0000","last_peered":"2024-04-25T01:18:04.259796+0000","last_clean":"2024-04-25T01:18:04.259796+0000","last_became_active":"2024-04-25T01:18:02.266605+0000","last_became_peered":"2024-04-25T01:18:02.266605+0000","last_unstale":"2024-04-25T01:18:04.259796+0000","last_undegraded":"2024-04-25T01:18:04.259796+0000","last_fullsized":"2024-04-25T01:18:04.259796+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:34:33.983053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291049+0000","last_change":"2024-04-25T01:18:03.274242+0000","last_active":"2024-04-25T01:18:04.291049+0000","last_peered":"2024-04-25T01:18:04.291049+0000","last_clean":"2024-04-25T01:18:04.291049+0000","last_became_active":"2024-04-25T01:18:03.273857+0000","last_became_peered":"2024-04-25T01:18:03.273857+0000","last_unstale":"2024-04-25T01:18:04.291049+0000","last_undegraded":"2024-04-25T01:18:04.291049+0000","last_fullsized":"2024-04-25T01:18:04.291049+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:09:48.390275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.260370+0000","last_change":"2024-04-25T01:18:02.259159+0000","last_active":"2024-04-25T01:18:04.260370+0000","last_peered":"2024-04-25T01:18:04.260370+0000","last_clean":"2024-04-25T01:18:04.260370+0000","last_became_active":"2024-04-25T01:18:02.257389+0000","last_became_peered":"2024-04-25T01:18:02.257389+0000","last_unstale":"2024-04-25T01:18:04.260370+0000","last_undegraded":"2024-04-25T01:18:04.260370+0000","last_fullsized":"2024-04-25T01:18:04.260370+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:43.394014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.267005+0000","last_change":"2024-04-25T01:18:03.274693+0000","last_active":"2024-04-25T01:18:04.267005+0000","last_peered":"2024-04-25T01:18:04.267005+0000","last_clean":"2024-04-25T01:18:04.267005+0000","last_became_active":"2024-04-25T01:18:03.274419+0000","last_became_peered":"2024-04-25T01:18:03.274419+0000","last_unstale":"2024-04-25T01:18:04.267005+0000","last_undegraded":"2024-04-25T01:18:04.267005+0000","last_fullsized":"2024-04-25T01:18:04.267005+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:57:10.276501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291126+0000","last_change":"2024-04-25T01:18:02.254033+0000","last_active":"2024-04-25T01:18:04.291126+0000","last_peered":"2024-04-25T01:18:04.291126+0000","last_clean":"2024-04-25T01:18:04.291126+0000","last_became_active":"2024-04-25T01:18:02.253731+0000","last_became_peered":"2024-04-25T01:18:02.253731+0000","last_unstale":"2024-04-25T01:18:04.291126+0000","last_undegraded":"2024-04-25T01:18:04.291126+0000","last_fullsized":"2024-04-25T01:18:04.291126+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:00:15.697767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.290988+0000","last_change":"2024-04-25T01:18:03.258959+0000","last_active":"2024-04-25T01:18:04.290988+0000","last_peered":"2024-04-25T01:18:04.290988+0000","last_clean":"2024-04-25T01:18:04.290988+0000","last_became_active":"2024-04-25T01:18:03.258462+0000","last_became_peered":"2024-04-25T01:18:03.258462+0000","last_unstale":"2024-04-25T01:18:04.290988+0000","last_undegraded":"2024-04-25T01:18:04.290988+0000","last_fullsized":"2024-04-25T01:18:04.290988+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:46:29.858633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.259349+0000","last_change":"2024-04-25T01:18:02.266874+0000","last_active":"2024-04-25T01:18:04.259349+0000","last_peered":"2024-04-25T01:18:04.259349+0000","last_clean":"2024-04-25T01:18:04.259349+0000","last_became_active":"2024-04-25T01:18:02.266188+0000","last_became_peered":"2024-04-25T01:18:02.266188+0000","last_unstale":"2024-04-25T01:18:04.259349+0000","last_undegraded":"2024-04-25T01:18:04.259349+0000","last_fullsized":"2024-04-25T01:18:04.259349+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:12:59.564111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.263283+0000","last_change":"2024-04-25T01:18:03.269115+0000","last_active":"2024-04-25T01:18:04.263283+0000","last_peered":"2024-04-25T01:18:04.263283+0000","last_clean":"2024-04-25T01:18:04.263283+0000","last_became_active":"2024-04-25T01:18:03.268774+0000","last_became_peered":"2024-04-25T01:18:03.268774+0000","last_unstale":"2024-04-25T01:18:04.263283+0000","last_undegraded":"2024-04-25T01:18:04.263283+0000","last_fullsized":"2024-04-25T01:18:04.263283+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:03:34.263185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"39'2","reported_seq":16,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.254836+0000","last_change":"2024-04-25T01:18:02.257314+0000","last_active":"2024-04-25T01:18:04.254836+0000","last_peered":"2024-04-25T01:18:04.254836+0000","last_clean":"2024-04-25T01:18:04.254836+0000","last_became_active":"2024-04-25T01:18:02.256973+0000","last_became_peered":"2024-04-25T01:18:02.256973+0000","last_unstale":"2024-04-25T01:18:04.254836+0000","last_undegraded":"2024-04-25T01:18:04.254836+0000","last_fullsized":"2024-04-25T01:18:04.254836+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:06:24.081982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.267318+0000","last_change":"2024-04-25T01:18:03.276655+0000","last_active":"2024-04-25T01:18:04.267318+0000","last_peered":"2024-04-25T01:18:04.267318+0000","last_clean":"2024-04-25T01:18:04.267318+0000","last_became_active":"2024-04-25T01:18:03.276023+0000","last_became_peered":"2024-04-25T01:18:03.276023+0000","last_unstale":"2024-04-25T01:18:04.267318+0000","last_undegraded":"2024-04-25T01:18:04.267318+0000","last_fullsized":"2024-04-25T01:18:04.267318+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:00:57.808084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"40'1","reported_seq":15,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.266060+0000","last_change":"2024-04-25T01:18:02.265987+0000","last_active":"2024-04-25T01:18:04.266060+0000","last_peered":"2024-04-25T01:18:04.266060+0000","last_clean":"2024-04-25T01:18:04.266060+0000","last_became_active":"2024-04-25T01:18:02.265704+0000","last_became_peered":"2024-04-25T01:18:02.265704+0000","last_unstale":"2024-04-25T01:18:04.266060+0000","last_undegraded":"2024-04-25T01:18:04.266060+0000","last_fullsized":"2024-04-25T01:18:04.266060+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:03:28.616338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.261930+0000","last_change":"2024-04-25T01:18:03.260610+0000","last_active":"2024-04-25T01:18:04.261930+0000","last_peered":"2024-04-25T01:18:04.261930+0000","last_clean":"2024-04-25T01:18:04.261930+0000","last_became_active":"2024-04-25T01:18:03.259487+0000","last_became_peered":"2024-04-25T01:18:03.259487+0000","last_unstale":"2024-04-25T01:18:04.261930+0000","last_undegraded":"2024-04-25T01:18:04.261930+0000","last_fullsized":"2024-04-25T01:18:04.261930+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:58:30.612559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.263133+0000","last_change":"2024-04-25T01:18:02.269139+0000","last_active":"2024-04-25T01:18:04.263133+0000","last_peered":"2024-04-25T01:18:04.263133+0000","last_clean":"2024-04-25T01:18:04.263133+0000","last_became_active":"2024-04-25T01:18:02.268747+0000","last_became_peered":"2024-04-25T01:18:02.268747+0000","last_unstale":"2024-04-25T01:18:04.263133+0000","last_undegraded":"2024-04-25T01:18:04.263133+0000","last_fullsized":"2024-04-25T01:18:04.263133+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:49:10.075668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.262688+0000","last_change":"2024-04-25T01:18:03.254793+0000","last_active":"2024-04-25T01:18:04.262688+0000","last_peered":"2024-04-25T01:18:04.262688+0000","last_clean":"2024-04-25T01:18:04.262688+0000","last_became_active":"2024-04-25T01:18:03.253467+0000","last_became_peered":"2024-04-25T01:18:03.253467+0000","last_unstale":"2024-04-25T01:18:04.262688+0000","last_undegraded":"2024-04-25T01:18:04.262688+0000","last_fullsized":"2024-04-25T01:18:04.262688+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:54:36.224430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291357+0000","last_change":"2024-04-25T01:18:02.269410+0000","last_active":"2024-04-25T01:18:04.291357+0000","last_peered":"2024-04-25T01:18:04.291357+0000","last_clean":"2024-04-25T01:18:04.291357+0000","last_became_active":"2024-04-25T01:18:02.268924+0000","last_became_peered":"2024-04-25T01:18:02.268924+0000","last_unstale":"2024-04-25T01:18:04.291357+0000","last_undegraded":"2024-04-25T01:18:04.291357+0000","last_fullsized":"2024-04-25T01:18:04.291357+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:19.010506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.260535+0000","last_change":"2024-04-25T01:18:03.261409+0000","last_active":"2024-04-25T01:18:04.260535+0000","last_peered":"2024-04-25T01:18:04.260535+0000","last_clean":"2024-04-25T01:18:04.260535+0000","last_became_active":"2024-04-25T01:18:03.261126+0000","last_became_peered":"2024-04-25T01:18:03.261126+0000","last_unstale":"2024-04-25T01:18:04.260535+0000","last_undegraded":"2024-04-25T01:18:04.260535+0000","last_fullsized":"2024-04-25T01:18:04.260535+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:53:46.580659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.267157+0000","last_change":"2024-04-25T01:18:02.267491+0000","last_active":"2024-04-25T01:18:04.267157+0000","last_peered":"2024-04-25T01:18:04.267157+0000","last_clean":"2024-04-25T01:18:04.267157+0000","last_became_active":"2024-04-25T01:18:02.267044+0000","last_became_peered":"2024-04-25T01:18:02.267044+0000","last_unstale":"2024-04-25T01:18:04.267157+0000","last_undegraded":"2024-04-25T01:18:04.267157+0000","last_fullsized":"2024-04-25T01:18:04.267157+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:30:07.096033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.254437+0000","last_change":"2024-04-25T01:18:03.254489+0000","last_active":"2024-04-25T01:18:04.254437+0000","last_peered":"2024-04-25T01:18:04.254437+0000","last_clean":"2024-04-25T01:18:04.254437+0000","last_became_active":"2024-04-25T01:18:03.254164+0000","last_became_peered":"2024-04-25T01:18:03.254164+0000","last_unstale":"2024-04-25T01:18:04.254437+0000","last_undegraded":"2024-04-25T01:18:04.254437+0000","last_fullsized":"2024-04-25T01:18:04.254437+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:52:20.724622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.254663+0000","last_change":"2024-04-25T01:18:02.263853+0000","last_active":"2024-04-25T01:18:04.254663+0000","last_peered":"2024-04-25T01:18:04.254663+0000","last_clean":"2024-04-25T01:18:04.254663+0000","last_became_active":"2024-04-25T01:18:02.263501+0000","last_became_peered":"2024-04-25T01:18:02.263501+0000","last_unstale":"2024-04-25T01:18:04.254663+0000","last_undegraded":"2024-04-25T01:18:04.254663+0000","last_fullsized":"2024-04-25T01:18:04.254663+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:06:33.412784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.266916+0000","last_change":"2024-04-25T01:18:03.256275+0000","last_active":"2024-04-25T01:18:04.266916+0000","last_peered":"2024-04-25T01:18:04.266916+0000","last_clean":"2024-04-25T01:18:04.266916+0000","last_became_active":"2024-04-25T01:18:03.255950+0000","last_became_peered":"2024-04-25T01:18:03.255950+0000","last_unstale":"2024-04-25T01:18:04.266916+0000","last_undegraded":"2024-04-25T01:18:04.266916+0000","last_fullsized":"2024-04-25T01:18:04.266916+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:18:05.301920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.292552+0000","last_change":"2024-04-25T01:18:02.269077+0000","last_active":"2024-04-25T01:18:04.292552+0000","last_peered":"2024-04-25T01:18:04.292552+0000","last_clean":"2024-04-25T01:18:04.292552+0000","last_became_active":"2024-04-25T01:18:02.268795+0000","last_became_peered":"2024-04-25T01:18:02.268795+0000","last_unstale":"2024-04-25T01:18:04.292552+0000","last_undegraded":"2024-04-25T01:18:04.292552+0000","last_fullsized":"2024-04-25T01:18:04.292552+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:16.123119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.254719+0000","last_change":"2024-04-25T01:18:03.253285+0000","last_active":"2024-04-25T01:18:04.254719+0000","last_peered":"2024-04-25T01:18:04.254719+0000","last_clean":"2024-04-25T01:18:04.254719+0000","last_became_active":"2024-04-25T01:18:03.252973+0000","last_became_peered":"2024-04-25T01:18:03.252973+0000","last_unstale":"2024-04-25T01:18:04.254719+0000","last_undegraded":"2024-04-25T01:18:04.254719+0000","last_fullsized":"2024-04-25T01:18:04.254719+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:20:54.301682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.266877+0000","last_change":"2024-04-25T01:18:02.267166+0000","last_active":"2024-04-25T01:18:04.266877+0000","last_peered":"2024-04-25T01:18:04.266877+0000","last_clean":"2024-04-25T01:18:04.266877+0000","last_became_active":"2024-04-25T01:18:02.266866+0000","last_became_peered":"2024-04-25T01:18:02.266866+0000","last_unstale":"2024-04-25T01:18:04.266877+0000","last_undegraded":"2024-04-25T01:18:04.266877+0000","last_fullsized":"2024-04-25T01:18:04.266877+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:20:28.341160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":204,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.289984+0000","last_change":"2024-04-25T01:16:17.466569+0000","last_active":"2024-04-25T01:18:04.289984+0000","last_peered":"2024-04-25T01:18:04.289984+0000","last_clean":"2024-04-25T01:18:04.289984+0000","last_became_active":"2024-04-25T01:16:17.465687+0000","last_became_peered":"2024-04-25T01:16:17.465687+0000","last_unstale":"2024-04-25T01:18:04.289984+0000","last_undegraded":"2024-04-25T01:18:04.289984+0000","last_fullsized":"2024-04-25T01:18:04.289984+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:16.440559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:16.440559+0000","last_clean_scrub_stamp":"2024-04-25T01:16:16.440559+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:18:17.493555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542169,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":462848,"data_stored":197122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542168,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":462848,"data_stored":197086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542169,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28052,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343788,"statfs":{"total":96636764160,"available":96608038912,"internally_reserved":0,"allocated":1056768,"data_stored":788002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542168,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27484,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344356,"statfs":{"total":96636764160,"available":96608620544,"internally_reserved":0,"allocated":466944,"data_stored":192096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542168,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":462848,"data_stored":197142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542170,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27484,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344356,"statfs":{"total":96636764160,"available":96608620544,"internally_reserved":0,"allocated":475136,"data_stored":199246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542168,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28060,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343780,"statfs":{"total":96636764160,"available":96608030720,"internally_reserved":0,"allocated":1056768,"data_stored":787442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542169,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":466944,"data_stored":197144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:18:07.609 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:18:07.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.865+0000 7f2b95800640 1 Processor -- start 2024-04-25T01:18:07.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.866+0000 7f2b95800640 1 -- start start 2024-04-25T01:18:07.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.867+0000 7f2b95800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2b90161760 0x7f2b900d2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:07.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.867+0000 7f2b95800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2b900b5340 0x7f2b900b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:07.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.867+0000 7f2b95800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b900cebb0 con 0x7f2b900d0850 2024-04-25T01:18:07.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.867+0000 7f2b95800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f2b900d46d0 con 0x7f2b90161760 2024-04-25T01:18:07.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.867+0000 7f2b95800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f2b900d49b0 con 0x7f2b900b5340 2024-04-25T01:18:07.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.867+0000 7f2b8fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2b90161760 0x7f2b900d2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:07.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.867+0000 7f2b8fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2b90161760 0x7f2b900d2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33174/0 (socket says 172.21.15.90:33174) 2024-04-25T01:18:07.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.867+0000 7f2b8fe00640 1 -- 172.21.15.90:0/423034124 learned_addr learned my addr 172.21.15.90:0/423034124 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:07.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.867+0000 7f2b8f400640 1 --2- 172.21.15.90:0/423034124 >> v2:172.21.15.103:3300/0 conn(0x7f2b900b5340 0x7f2b900b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:07.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.867+0000 7f2b8fe00640 1 -- 172.21.15.90:0/423034124 >> v2:172.21.15.103:3300/0 conn(0x7f2b900b5340 msgr2=0x7f2b900b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:07.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.868+0000 7f2b8fe00640 1 --2- 172.21.15.90:0/423034124 >> v2:172.21.15.103:3300/0 conn(0x7f2b900b5340 0x7f2b900b3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.868+0000 7f2b8fe00640 1 -- 172.21.15.90:0/423034124 >> v1:172.21.15.103:6789/0 conn(0x7f2b900d0850 legacy=0x7f2b900cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:07.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.868+0000 7f2b8fe00640 1 -- 172.21.15.90:0/423034124 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b900cee50 con 0x7f2b90161760 2024-04-25T01:18:07.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.868+0000 7f2b8f400640 1 --2- 172.21.15.90:0/423034124 >> v2:172.21.15.103:3300/0 conn(0x7f2b900b5340 0x7f2b900b3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:07.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.869+0000 7f2b8fe00640 1 --2- 172.21.15.90:0/423034124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b90161760 0x7f2b900d2b10 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f2b9006dfe0 tx=0x7f2b8002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20af95514c2a413d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:07.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.869+0000 7f2b8ea00640 1 -- 172.21.15.90:0/423034124 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b80037070 con 0x7f2b90161760 2024-04-25T01:18:07.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.869+0000 7f2b8ea00640 1 -- 172.21.15.90:0/423034124 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b80032440 con 0x7f2b90161760 2024-04-25T01:18:07.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.869+0000 7f2b8ea00640 1 -- 172.21.15.90:0/423034124 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b80032810 con 0x7f2b90161760 2024-04-25T01:18:07.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.869+0000 7f2b95800640 1 -- 172.21.15.90:0/423034124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b90161760 msgr2=0x7f2b900d2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:07.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.869+0000 7f2b95800640 1 --2- 172.21.15.90:0/423034124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b90161760 0x7f2b900d2b10 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f2b9006dfe0 tx=0x7f2b8002a180 comp rx=0 tx=0).stop 2024-04-25T01:18:07.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.870+0000 7f2b95800640 1 -- 172.21.15.90:0/423034124 shutdown_connections 2024-04-25T01:18:07.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.870+0000 7f2b95800640 1 --2- 172.21.15.90:0/423034124 >> v2:172.21.15.103:3300/0 conn(0x7f2b900b5340 0x7f2b900b3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.870+0000 7f2b95800640 1 --2- 172.21.15.90:0/423034124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b90161760 0x7f2b900d2b10 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.870+0000 7f2b95800640 1 -- 172.21.15.90:0/423034124 >> 172.21.15.90:0/423034124 conn(0x7f2b9009d170 msgr2=0x7f2b900d3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:07.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.870+0000 7f2b95800640 1 -- 172.21.15.90:0/423034124 shutdown_connections 2024-04-25T01:18:07.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.870+0000 7f2b95800640 1 -- 172.21.15.90:0/423034124 wait complete. 2024-04-25T01:18:07.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.871+0000 7f2b95800640 1 Processor -- start 2024-04-25T01:18:07.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.871+0000 7f2b95800640 1 -- start start 2024-04-25T01:18:07.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.872+0000 7f2b95800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b900b5340 0x7f2b900cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:07.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.872+0000 7f2b95800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b900d0850 0x7f2b900c7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:07.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.872+0000 7f2b95800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b90161760 0x7f2b9015d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:07.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.872+0000 7f2b95800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2b90161b40 con 0x7f2b900b5340 2024-04-25T01:18:07.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.872+0000 7f2b95800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2b900d0ca0 con 0x7f2b900d0850 2024-04-25T01:18:07.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.872+0000 7f2b95800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2b9015d820 con 0x7f2b90161760 2024-04-25T01:18:07.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.873+0000 7f2b8fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b900b5340 0x7f2b900cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:07.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.873+0000 7f2b8fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b900b5340 0x7f2b900cdb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33186/0 (socket says 172.21.15.90:33186) 2024-04-25T01:18:07.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.873+0000 7f2b8fe00640 1 -- 172.21.15.90:0/1688968037 learned_addr learned my addr 172.21.15.90:0/1688968037 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:07.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.873+0000 7f2b8fe00640 1 -- 172.21.15.90:0/1688968037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b90161760 msgr2=0x7f2b9015d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:07.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.873+0000 7f2b8fe00640 1 --2- 172.21.15.90:0/1688968037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b90161760 0x7f2b9015d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.873+0000 7f2b8fe00640 1 -- 172.21.15.90:0/1688968037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b900d0850 msgr2=0x7f2b900c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:07.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.873+0000 7f2b8fe00640 1 --2- 172.21.15.90:0/1688968037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b900d0850 0x7f2b900c7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:07.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.873+0000 7f2b8fe00640 1 -- 172.21.15.90:0/1688968037 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b900b7af0 con 0x7f2b900b5340 2024-04-25T01:18:07.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.873+0000 7f2b8fe00640 1 --2- 172.21.15.90:0/1688968037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b900b5340 0x7f2b900cdb70 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f2b80000c00 tx=0x7f2b800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:07.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.874+0000 7f2b87e00640 1 -- 172.21.15.90:0/1688968037 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b80041020 con 0x7f2b900b5340 2024-04-25T01:18:07.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.874+0000 7f2b87e00640 1 -- 172.21.15.90:0/1688968037 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b80030970 con 0x7f2b900b5340 2024-04-25T01:18:07.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.874+0000 7f2b87e00640 1 -- 172.21.15.90:0/1688968037 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b80037040 con 0x7f2b900b5340 2024-04-25T01:18:07.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.874+0000 7f2b95800640 1 -- 172.21.15.90:0/1688968037 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b900b7dd0 con 0x7f2b900b5340 2024-04-25T01:18:07.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.874+0000 7f2b95800640 1 -- 172.21.15.90:0/1688968037 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b900b8390 con 0x7f2b900b5340 2024-04-25T01:18:07.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.875+0000 7f2b95800640 1 -- 172.21.15.90:0/1688968037 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b90075350 con 0x7f2b900b5340 2024-04-25T01:18:07.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.876+0000 7f2b87e00640 1 -- 172.21.15.90:0/1688968037 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2b80044020 con 0x7f2b900b5340 2024-04-25T01:18:07.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.877+0000 7f2b87e00640 1 --2- 172.21.15.90:0/1688968037 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2b7006be60 0x7f2b7006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:07.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.877+0000 7f2b87e00640 1 -- 172.21.15.90:0/1688968037 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6101+0+0 (secure 0 0 0) 0x7f2b800ab9c0 con 0x7f2b900b5340 2024-04-25T01:18:07.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.877+0000 7f2b8f400640 1 --2- 172.21.15.90:0/1688968037 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2b7006be60 0x7f2b7006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:07.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.878+0000 7f2b8f400640 1 --2- 172.21.15.90:0/1688968037 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2b7006be60 0x7f2b7006e320 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c006fd0 tx=0x7f2b7c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:07.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:07.881+0000 7f2b87e00640 1 -- 172.21.15.90:0/1688968037 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2b8004b030 con 0x7f2b900b5340 2024-04-25T01:18:08.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.079+0000 7f2b95800640 1 -- 172.21.15.90:0/1688968037 --> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2b900678f0 con 0x7f2b7006be60 2024-04-25T01:18:08.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.099+0000 7f2b87e00640 1 -- 172.21.15.90:0/1688968037 <== mgr.4104 v2:172.21.15.90:6832/2198086733 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251951 (secure 0 0 0) 0x7f2b900678f0 con 0x7f2b7006be60 2024-04-25T01:18:08.085 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:08.086 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-25T01:18:08.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.104+0000 7f2b95800640 1 -- 172.21.15.90:0/1688968037 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2b7006be60 msgr2=0x7f2b7006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:08.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.104+0000 7f2b95800640 1 --2- 172.21.15.90:0/1688968037 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2b7006be60 0x7f2b7006e320 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c006fd0 tx=0x7f2b7c008040 comp rx=0 tx=0).stop 2024-04-25T01:18:08.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.104+0000 7f2b95800640 1 -- 172.21.15.90:0/1688968037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b900b5340 msgr2=0x7f2b900cdb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:08.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.104+0000 7f2b95800640 1 --2- 172.21.15.90:0/1688968037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b900b5340 0x7f2b900cdb70 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f2b80000c00 tx=0x7f2b800046c0 comp rx=0 tx=0).stop 2024-04-25T01:18:08.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.104+0000 7f2b95800640 1 -- 172.21.15.90:0/1688968037 shutdown_connections 2024-04-25T01:18:08.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.104+0000 7f2b95800640 1 --2- 172.21.15.90:0/1688968037 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2b7006be60 0x7f2b7006e320 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:08.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.104+0000 7f2b95800640 1 --2- 172.21.15.90:0/1688968037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2b90161760 0x7f2b9015d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:08.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.104+0000 7f2b95800640 1 --2- 172.21.15.90:0/1688968037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b900d0850 0x7f2b900c7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:08.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.104+0000 7f2b95800640 1 --2- 172.21.15.90:0/1688968037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b900b5340 0x7f2b900cdb70 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:08.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.104+0000 7f2b95800640 1 -- 172.21.15.90:0/1688968037 >> 172.21.15.90:0/1688968037 conn(0x7f2b9009d170 msgr2=0x7f2b9015fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:08.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.104+0000 7f2b95800640 1 -- 172.21.15.90:0/1688968037 shutdown_connections 2024-04-25T01:18:08.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.104+0000 7f2b95800640 1 -- 172.21.15.90:0/1688968037 wait complete. 2024-04-25T01:18:08.107 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2024-04-25T01:18:07.860343+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":78,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220976,"kb_used_data":4796,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753744,"statfs":{"total":773094113280,"available":772867833856,"internally_reserved":0,"allocated":4911104,"data_stored":2755280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002001"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.292064+0000","last_change":"2024-04-25T01:18:03.253664+0000","last_active":"2024-04-25T01:18:04.292064+0000","last_peered":"2024-04-25T01:18:04.292064+0000","last_clean":"2024-04-25T01:18:04.292064+0000","last_became_active":"2024-04-25T01:18:03.253362+0000","last_became_peered":"2024-04-25T01:18:03.253362+0000","last_unstale":"2024-04-25T01:18:04.292064+0000","last_undegraded":"2024-04-25T01:18:04.292064+0000","last_fullsized":"2024-04-25T01:18:04.292064+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:39.462581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.260823+0000","last_change":"2024-04-25T01:18:03.247432+0000","last_active":"2024-04-25T01:18:04.260823+0000","last_peered":"2024-04-25T01:18:04.260823+0000","last_clean":"2024-04-25T01:18:04.260823+0000","last_became_active":"2024-04-25T01:18:03.247132+0000","last_became_peered":"2024-04-25T01:18:03.247132+0000","last_unstale":"2024-04-25T01:18:04.260823+0000","last_undegraded":"2024-04-25T01:18:04.260823+0000","last_fullsized":"2024-04-25T01:18:04.260823+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:56.294412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291402+0000","last_change":"2024-04-25T01:18:03.267777+0000","last_active":"2024-04-25T01:18:04.291402+0000","last_peered":"2024-04-25T01:18:04.291402+0000","last_clean":"2024-04-25T01:18:04.291402+0000","last_became_active":"2024-04-25T01:18:03.267444+0000","last_became_peered":"2024-04-25T01:18:03.267444+0000","last_unstale":"2024-04-25T01:18:04.291402+0000","last_undegraded":"2024-04-25T01:18:04.291402+0000","last_fullsized":"2024-04-25T01:18:04.291402+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:38:56.106596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291936+0000","last_change":"2024-04-25T01:18:03.260302+0000","last_active":"2024-04-25T01:18:04.291936+0000","last_peered":"2024-04-25T01:18:04.291936+0000","last_clean":"2024-04-25T01:18:04.291936+0000","last_became_active":"2024-04-25T01:18:03.259967+0000","last_became_peered":"2024-04-25T01:18:03.259967+0000","last_unstale":"2024-04-25T01:18:04.291936+0000","last_undegraded":"2024-04-25T01:18:04.291936+0000","last_fullsized":"2024-04-25T01:18:04.291936+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:23:03.906624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.263671+0000","last_change":"2024-04-25T01:18:03.257452+0000","last_active":"2024-04-25T01:18:04.263671+0000","last_peered":"2024-04-25T01:18:04.263671+0000","last_clean":"2024-04-25T01:18:04.263671+0000","last_became_active":"2024-04-25T01:18:03.256443+0000","last_became_peered":"2024-04-25T01:18:03.256443+0000","last_unstale":"2024-04-25T01:18:04.263671+0000","last_undegraded":"2024-04-25T01:18:04.263671+0000","last_fullsized":"2024-04-25T01:18:04.263671+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:53:48.526435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.260073+0000","last_change":"2024-04-25T01:18:03.242630+0000","last_active":"2024-04-25T01:18:04.260073+0000","last_peered":"2024-04-25T01:18:04.260073+0000","last_clean":"2024-04-25T01:18:04.260073+0000","last_became_active":"2024-04-25T01:18:03.242276+0000","last_became_peered":"2024-04-25T01:18:03.242276+0000","last_unstale":"2024-04-25T01:18:04.260073+0000","last_undegraded":"2024-04-25T01:18:04.260073+0000","last_fullsized":"2024-04-25T01:18:04.260073+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:13.740974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.290456+0000","last_change":"2024-04-25T01:18:03.257363+0000","last_active":"2024-04-25T01:18:04.290456+0000","last_peered":"2024-04-25T01:18:04.290456+0000","last_clean":"2024-04-25T01:18:04.290456+0000","last_became_active":"2024-04-25T01:18:03.256685+0000","last_became_peered":"2024-04-25T01:18:03.256685+0000","last_unstale":"2024-04-25T01:18:04.290456+0000","last_undegraded":"2024-04-25T01:18:04.290456+0000","last_fullsized":"2024-04-25T01:18:04.290456+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:10:28.460600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.292081+0000","last_change":"2024-04-25T01:18:03.274858+0000","last_active":"2024-04-25T01:18:04.292081+0000","last_peered":"2024-04-25T01:18:04.292081+0000","last_clean":"2024-04-25T01:18:04.292081+0000","last_became_active":"2024-04-25T01:18:03.273775+0000","last_became_peered":"2024-04-25T01:18:03.273775+0000","last_unstale":"2024-04-25T01:18:04.292081+0000","last_undegraded":"2024-04-25T01:18:04.292081+0000","last_fullsized":"2024-04-25T01:18:04.292081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:01:09.052002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.263741+0000","last_change":"2024-04-25T01:18:03.265545+0000","last_active":"2024-04-25T01:18:04.263741+0000","last_peered":"2024-04-25T01:18:04.263741+0000","last_clean":"2024-04-25T01:18:04.263741+0000","last_became_active":"2024-04-25T01:18:03.265251+0000","last_became_peered":"2024-04-25T01:18:03.265251+0000","last_unstale":"2024-04-25T01:18:04.263741+0000","last_undegraded":"2024-04-25T01:18:04.263741+0000","last_fullsized":"2024-04-25T01:18:04.263741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:10:28.460600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.290660+0000","last_change":"2024-04-25T01:18:03.267089+0000","last_active":"2024-04-25T01:18:04.290660+0000","last_peered":"2024-04-25T01:18:04.290660+0000","last_clean":"2024-04-25T01:18:04.290660+0000","last_became_active":"2024-04-25T01:18:03.266799+0000","last_became_peered":"2024-04-25T01:18:03.266799+0000","last_unstale":"2024-04-25T01:18:04.290660+0000","last_undegraded":"2024-04-25T01:18:04.290660+0000","last_fullsized":"2024-04-25T01:18:04.290660+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:22:35.122721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.267246+0000","last_change":"2024-04-25T01:18:03.276878+0000","last_active":"2024-04-25T01:18:04.267246+0000","last_peered":"2024-04-25T01:18:04.267246+0000","last_clean":"2024-04-25T01:18:04.267246+0000","last_became_active":"2024-04-25T01:18:03.276396+0000","last_became_peered":"2024-04-25T01:18:03.276396+0000","last_unstale":"2024-04-25T01:18:04.267246+0000","last_undegraded":"2024-04-25T01:18:04.267246+0000","last_fullsized":"2024-04-25T01:18:04.267246+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:40.084475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.263218+0000","last_change":"2024-04-25T01:18:03.249998+0000","last_active":"2024-04-25T01:18:04.263218+0000","last_peered":"2024-04-25T01:18:04.263218+0000","last_clean":"2024-04-25T01:18:04.263218+0000","last_became_active":"2024-04-25T01:18:03.249714+0000","last_became_peered":"2024-04-25T01:18:03.249714+0000","last_unstale":"2024-04-25T01:18:04.263218+0000","last_undegraded":"2024-04-25T01:18:04.263218+0000","last_fullsized":"2024-04-25T01:18:04.263218+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:23:00.911957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.290448+0000","last_change":"2024-04-25T01:18:03.256824+0000","last_active":"2024-04-25T01:18:04.290448+0000","last_peered":"2024-04-25T01:18:04.290448+0000","last_clean":"2024-04-25T01:18:04.290448+0000","last_became_active":"2024-04-25T01:18:03.256477+0000","last_became_peered":"2024-04-25T01:18:03.256477+0000","last_unstale":"2024-04-25T01:18:04.290448+0000","last_undegraded":"2024-04-25T01:18:04.290448+0000","last_fullsized":"2024-04-25T01:18:04.290448+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:09:22.874364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.290651+0000","last_change":"2024-04-25T01:18:03.265116+0000","last_active":"2024-04-25T01:18:04.290651+0000","last_peered":"2024-04-25T01:18:04.290651+0000","last_clean":"2024-04-25T01:18:04.290651+0000","last_became_active":"2024-04-25T01:18:03.264863+0000","last_became_peered":"2024-04-25T01:18:03.264863+0000","last_unstale":"2024-04-25T01:18:04.290651+0000","last_undegraded":"2024-04-25T01:18:04.290651+0000","last_fullsized":"2024-04-25T01:18:04.290651+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:11.435894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.253875+0000","last_change":"2024-04-25T01:18:03.258504+0000","last_active":"2024-04-25T01:18:04.253875+0000","last_peered":"2024-04-25T01:18:04.253875+0000","last_clean":"2024-04-25T01:18:04.253875+0000","last_became_active":"2024-04-25T01:18:03.258242+0000","last_became_peered":"2024-04-25T01:18:03.258242+0000","last_unstale":"2024-04-25T01:18:04.253875+0000","last_undegraded":"2024-04-25T01:18:04.253875+0000","last_fullsized":"2024-04-25T01:18:04.253875+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:02:01.161670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.289891+0000","last_change":"2024-04-25T01:18:03.261774+0000","last_active":"2024-04-25T01:18:04.289891+0000","last_peered":"2024-04-25T01:18:04.289891+0000","last_clean":"2024-04-25T01:18:04.289891+0000","last_became_active":"2024-04-25T01:18:03.261428+0000","last_became_peered":"2024-04-25T01:18:03.261428+0000","last_unstale":"2024-04-25T01:18:04.289891+0000","last_undegraded":"2024-04-25T01:18:04.289891+0000","last_fullsized":"2024-04-25T01:18:04.289891+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:16.174537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.254754+0000","last_change":"2024-04-25T01:18:03.273618+0000","last_active":"2024-04-25T01:18:04.254754+0000","last_peered":"2024-04-25T01:18:04.254754+0000","last_clean":"2024-04-25T01:18:04.254754+0000","last_became_active":"2024-04-25T01:18:03.273161+0000","last_became_peered":"2024-04-25T01:18:03.273161+0000","last_unstale":"2024-04-25T01:18:04.254754+0000","last_undegraded":"2024-04-25T01:18:04.254754+0000","last_fullsized":"2024-04-25T01:18:04.254754+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:32:52.505647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.254583+0000","last_change":"2024-04-25T01:18:03.265079+0000","last_active":"2024-04-25T01:18:04.254583+0000","last_peered":"2024-04-25T01:18:04.254583+0000","last_clean":"2024-04-25T01:18:04.254583+0000","last_became_active":"2024-04-25T01:18:03.264708+0000","last_became_peered":"2024-04-25T01:18:03.264708+0000","last_unstale":"2024-04-25T01:18:04.254583+0000","last_undegraded":"2024-04-25T01:18:04.254583+0000","last_fullsized":"2024-04-25T01:18:04.254583+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:57:03.302298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.261281+0000","last_change":"2024-04-25T01:18:03.257519+0000","last_active":"2024-04-25T01:18:04.261281+0000","last_peered":"2024-04-25T01:18:04.261281+0000","last_clean":"2024-04-25T01:18:04.261281+0000","last_became_active":"2024-04-25T01:18:03.257225+0000","last_became_peered":"2024-04-25T01:18:03.257225+0000","last_unstale":"2024-04-25T01:18:04.261281+0000","last_undegraded":"2024-04-25T01:18:04.261281+0000","last_fullsized":"2024-04-25T01:18:04.261281+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:50:41.389115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.266754+0000","last_change":"2024-04-25T01:18:03.267873+0000","last_active":"2024-04-25T01:18:04.266754+0000","last_peered":"2024-04-25T01:18:04.266754+0000","last_clean":"2024-04-25T01:18:04.266754+0000","last_became_active":"2024-04-25T01:18:03.267601+0000","last_became_peered":"2024-04-25T01:18:03.267601+0000","last_unstale":"2024-04-25T01:18:04.266754+0000","last_undegraded":"2024-04-25T01:18:04.266754+0000","last_fullsized":"2024-04-25T01:18:04.266754+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:35:34.926511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.266736+0000","last_change":"2024-04-25T01:18:03.256260+0000","last_active":"2024-04-25T01:18:04.266736+0000","last_peered":"2024-04-25T01:18:04.266736+0000","last_clean":"2024-04-25T01:18:04.266736+0000","last_became_active":"2024-04-25T01:18:03.255488+0000","last_became_peered":"2024-04-25T01:18:03.255488+0000","last_unstale":"2024-04-25T01:18:04.266736+0000","last_undegraded":"2024-04-25T01:18:04.266736+0000","last_fullsized":"2024-04-25T01:18:04.266736+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:19.882784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.266737+0000","last_change":"2024-04-25T01:18:03.262912+0000","last_active":"2024-04-25T01:18:04.266737+0000","last_peered":"2024-04-25T01:18:04.266737+0000","last_clean":"2024-04-25T01:18:04.266737+0000","last_became_active":"2024-04-25T01:18:03.262598+0000","last_became_peered":"2024-04-25T01:18:03.262598+0000","last_unstale":"2024-04-25T01:18:04.266737+0000","last_undegraded":"2024-04-25T01:18:04.266737+0000","last_fullsized":"2024-04-25T01:18:04.266737+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:19.060283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.253929+0000","last_change":"2024-04-25T01:18:03.259084+0000","last_active":"2024-04-25T01:18:04.253929+0000","last_peered":"2024-04-25T01:18:04.253929+0000","last_clean":"2024-04-25T01:18:04.253929+0000","last_became_active":"2024-04-25T01:18:03.258651+0000","last_became_peered":"2024-04-25T01:18:03.258651+0000","last_unstale":"2024-04-25T01:18:04.253929+0000","last_undegraded":"2024-04-25T01:18:04.253929+0000","last_fullsized":"2024-04-25T01:18:04.253929+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:42:33.542272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291302+0000","last_change":"2024-04-25T01:18:03.270002+0000","last_active":"2024-04-25T01:18:04.291302+0000","last_peered":"2024-04-25T01:18:04.291302+0000","last_clean":"2024-04-25T01:18:04.291302+0000","last_became_active":"2024-04-25T01:18:03.269711+0000","last_became_peered":"2024-04-25T01:18:03.269711+0000","last_unstale":"2024-04-25T01:18:04.291302+0000","last_undegraded":"2024-04-25T01:18:04.291302+0000","last_fullsized":"2024-04-25T01:18:04.291302+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:06:25.097573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291532+0000","last_change":"2024-04-25T01:18:03.265266+0000","last_active":"2024-04-25T01:18:04.291532+0000","last_peered":"2024-04-25T01:18:04.291532+0000","last_clean":"2024-04-25T01:18:04.291532+0000","last_became_active":"2024-04-25T01:18:03.264973+0000","last_became_peered":"2024-04-25T01:18:03.264973+0000","last_unstale":"2024-04-25T01:18:04.291532+0000","last_undegraded":"2024-04-25T01:18:04.291532+0000","last_fullsized":"2024-04-25T01:18:04.291532+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:50:14.498011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291794+0000","last_change":"2024-04-25T01:18:03.263008+0000","last_active":"2024-04-25T01:18:04.291794+0000","last_peered":"2024-04-25T01:18:04.291794+0000","last_clean":"2024-04-25T01:18:04.291794+0000","last_became_active":"2024-04-25T01:18:03.262081+0000","last_became_peered":"2024-04-25T01:18:03.262081+0000","last_unstale":"2024-04-25T01:18:04.291794+0000","last_undegraded":"2024-04-25T01:18:04.291794+0000","last_fullsized":"2024-04-25T01:18:04.291794+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:03:36.196096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.267231+0000","last_change":"2024-04-25T01:18:03.263526+0000","last_active":"2024-04-25T01:18:04.267231+0000","last_peered":"2024-04-25T01:18:04.267231+0000","last_clean":"2024-04-25T01:18:04.267231+0000","last_became_active":"2024-04-25T01:18:03.258646+0000","last_became_peered":"2024-04-25T01:18:03.258646+0000","last_unstale":"2024-04-25T01:18:04.267231+0000","last_undegraded":"2024-04-25T01:18:04.267231+0000","last_fullsized":"2024-04-25T01:18:04.267231+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:30:00.416602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.290316+0000","last_change":"2024-04-25T01:18:03.267937+0000","last_active":"2024-04-25T01:18:04.290316+0000","last_peered":"2024-04-25T01:18:04.290316+0000","last_clean":"2024-04-25T01:18:04.290316+0000","last_became_active":"2024-04-25T01:18:03.266925+0000","last_became_peered":"2024-04-25T01:18:03.266925+0000","last_unstale":"2024-04-25T01:18:04.290316+0000","last_undegraded":"2024-04-25T01:18:04.290316+0000","last_fullsized":"2024-04-25T01:18:04.290316+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:53:48.526435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.254161+0000","last_change":"2024-04-25T01:18:03.253210+0000","last_active":"2024-04-25T01:18:04.254161+0000","last_peered":"2024-04-25T01:18:04.254161+0000","last_clean":"2024-04-25T01:18:04.254161+0000","last_became_active":"2024-04-25T01:18:03.252930+0000","last_became_peered":"2024-04-25T01:18:03.252930+0000","last_unstale":"2024-04-25T01:18:04.254161+0000","last_undegraded":"2024-04-25T01:18:04.254161+0000","last_fullsized":"2024-04-25T01:18:04.254161+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:02:03.876956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.254246+0000","last_change":"2024-04-25T01:18:03.255049+0000","last_active":"2024-04-25T01:18:04.254246+0000","last_peered":"2024-04-25T01:18:04.254246+0000","last_clean":"2024-04-25T01:18:04.254246+0000","last_became_active":"2024-04-25T01:18:03.254795+0000","last_became_peered":"2024-04-25T01:18:03.254795+0000","last_unstale":"2024-04-25T01:18:04.254246+0000","last_undegraded":"2024-04-25T01:18:04.254246+0000","last_fullsized":"2024-04-25T01:18:04.254246+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:39:30.588445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.290867+0000","last_change":"2024-04-25T01:18:03.257602+0000","last_active":"2024-04-25T01:18:04.290867+0000","last_peered":"2024-04-25T01:18:04.290867+0000","last_clean":"2024-04-25T01:18:04.290867+0000","last_became_active":"2024-04-25T01:18:03.257303+0000","last_became_peered":"2024-04-25T01:18:03.257303+0000","last_unstale":"2024-04-25T01:18:04.290867+0000","last_undegraded":"2024-04-25T01:18:04.290867+0000","last_fullsized":"2024-04-25T01:18:04.290867+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:20:18.432704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291746+0000","last_change":"2024-04-25T01:18:03.274018+0000","last_active":"2024-04-25T01:18:04.291746+0000","last_peered":"2024-04-25T01:18:04.291746+0000","last_clean":"2024-04-25T01:18:04.291746+0000","last_became_active":"2024-04-25T01:18:03.273719+0000","last_became_peered":"2024-04-25T01:18:03.273719+0000","last_unstale":"2024-04-25T01:18:04.291746+0000","last_undegraded":"2024-04-25T01:18:04.291746+0000","last_fullsized":"2024-04-25T01:18:04.291746+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:41:12.326642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.254726+0000","last_change":"2024-04-25T01:18:03.264199+0000","last_active":"2024-04-25T01:18:04.254726+0000","last_peered":"2024-04-25T01:18:04.254726+0000","last_clean":"2024-04-25T01:18:04.254726+0000","last_became_active":"2024-04-25T01:18:03.260827+0000","last_became_peered":"2024-04-25T01:18:03.260827+0000","last_unstale":"2024-04-25T01:18:04.254726+0000","last_undegraded":"2024-04-25T01:18:04.254726+0000","last_fullsized":"2024-04-25T01:18:04.254726+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:24:54.943447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"39'4","reported_seq":18,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291486+0000","last_change":"2024-04-25T01:18:02.263556+0000","last_active":"2024-04-25T01:18:04.291486+0000","last_peered":"2024-04-25T01:18:04.291486+0000","last_clean":"2024-04-25T01:18:04.291486+0000","last_became_active":"2024-04-25T01:18:02.263123+0000","last_became_peered":"2024-04-25T01:18:02.263123+0000","last_unstale":"2024-04-25T01:18:04.291486+0000","last_undegraded":"2024-04-25T01:18:04.291486+0000","last_fullsized":"2024-04-25T01:18:04.291486+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:09:21.858773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.261110+0000","last_change":"2024-04-25T01:18:03.269144+0000","last_active":"2024-04-25T01:18:04.261110+0000","last_peered":"2024-04-25T01:18:04.261110+0000","last_clean":"2024-04-25T01:18:04.261110+0000","last_became_active":"2024-04-25T01:18:03.268851+0000","last_became_peered":"2024-04-25T01:18:03.268851+0000","last_unstale":"2024-04-25T01:18:04.261110+0000","last_undegraded":"2024-04-25T01:18:04.261110+0000","last_fullsized":"2024-04-25T01:18:04.261110+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:50:41.389115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"39'1","reported_seq":15,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.261229+0000","last_change":"2024-04-25T01:18:02.255987+0000","last_active":"2024-04-25T01:18:04.261229+0000","last_peered":"2024-04-25T01:18:04.261229+0000","last_clean":"2024-04-25T01:18:04.261229+0000","last_became_active":"2024-04-25T01:18:02.255540+0000","last_became_peered":"2024-04-25T01:18:02.255540+0000","last_unstale":"2024-04-25T01:18:04.261229+0000","last_undegraded":"2024-04-25T01:18:04.261229+0000","last_fullsized":"2024-04-25T01:18:04.261229+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:38:12.422806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.254737+0000","last_change":"2024-04-25T01:18:03.260381+0000","last_active":"2024-04-25T01:18:04.254737+0000","last_peered":"2024-04-25T01:18:04.254737+0000","last_clean":"2024-04-25T01:18:04.254737+0000","last_became_active":"2024-04-25T01:18:03.260114+0000","last_became_peered":"2024-04-25T01:18:03.260114+0000","last_unstale":"2024-04-25T01:18:04.254737+0000","last_undegraded":"2024-04-25T01:18:04.254737+0000","last_fullsized":"2024-04-25T01:18:04.254737+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:11.366143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"39'2","reported_seq":16,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291743+0000","last_change":"2024-04-25T01:18:02.270338+0000","last_active":"2024-04-25T01:18:04.291743+0000","last_peered":"2024-04-25T01:18:04.291743+0000","last_clean":"2024-04-25T01:18:04.291743+0000","last_became_active":"2024-04-25T01:18:02.270063+0000","last_became_peered":"2024-04-25T01:18:02.270063+0000","last_unstale":"2024-04-25T01:18:04.291743+0000","last_undegraded":"2024-04-25T01:18:04.291743+0000","last_fullsized":"2024-04-25T01:18:04.291743+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:25:09.988571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.261521+0000","last_change":"2024-04-25T01:18:03.264063+0000","last_active":"2024-04-25T01:18:04.261521+0000","last_peered":"2024-04-25T01:18:04.261521+0000","last_clean":"2024-04-25T01:18:04.261521+0000","last_became_active":"2024-04-25T01:18:03.263708+0000","last_became_peered":"2024-04-25T01:18:03.263708+0000","last_unstale":"2024-04-25T01:18:04.261521+0000","last_undegraded":"2024-04-25T01:18:04.261521+0000","last_fullsized":"2024-04-25T01:18:04.261521+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:09:35.248123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"39'2","reported_seq":16,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291260+0000","last_change":"2024-04-25T01:18:02.263250+0000","last_active":"2024-04-25T01:18:04.291260+0000","last_peered":"2024-04-25T01:18:04.291260+0000","last_clean":"2024-04-25T01:18:04.291260+0000","last_became_active":"2024-04-25T01:18:02.262763+0000","last_became_peered":"2024-04-25T01:18:02.262763+0000","last_unstale":"2024-04-25T01:18:04.291260+0000","last_undegraded":"2024-04-25T01:18:04.291260+0000","last_fullsized":"2024-04-25T01:18:04.291260+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:22:34.107130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.292170+0000","last_change":"2024-04-25T01:18:03.250303+0000","last_active":"2024-04-25T01:18:04.292170+0000","last_peered":"2024-04-25T01:18:04.292170+0000","last_clean":"2024-04-25T01:18:04.292170+0000","last_became_active":"2024-04-25T01:18:03.249995+0000","last_became_peered":"2024-04-25T01:18:03.249995+0000","last_unstale":"2024-04-25T01:18:04.292170+0000","last_undegraded":"2024-04-25T01:18:04.292170+0000","last_fullsized":"2024-04-25T01:18:04.292170+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:29:13.129901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"39'1","reported_seq":15,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291012+0000","last_change":"2024-04-25T01:18:02.270642+0000","last_active":"2024-04-25T01:18:04.291012+0000","last_peered":"2024-04-25T01:18:04.291012+0000","last_clean":"2024-04-25T01:18:04.291012+0000","last_became_active":"2024-04-25T01:18:02.270383+0000","last_became_peered":"2024-04-25T01:18:02.270383+0000","last_unstale":"2024-04-25T01:18:04.291012+0000","last_undegraded":"2024-04-25T01:18:04.291012+0000","last_fullsized":"2024-04-25T01:18:04.291012+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:20:17.417113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.266557+0000","last_change":"2024-04-25T01:18:03.262229+0000","last_active":"2024-04-25T01:18:04.266557+0000","last_peered":"2024-04-25T01:18:04.266557+0000","last_clean":"2024-04-25T01:18:04.266557+0000","last_became_active":"2024-04-25T01:18:03.256356+0000","last_became_peered":"2024-04-25T01:18:03.256356+0000","last_unstale":"2024-04-25T01:18:04.266557+0000","last_undegraded":"2024-04-25T01:18:04.266557+0000","last_fullsized":"2024-04-25T01:18:04.266557+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:55:31.639411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"39'1","reported_seq":15,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.290079+0000","last_change":"2024-04-25T01:18:02.267951+0000","last_active":"2024-04-25T01:18:04.290079+0000","last_peered":"2024-04-25T01:18:04.290079+0000","last_clean":"2024-04-25T01:18:04.290079+0000","last_became_active":"2024-04-25T01:18:02.267086+0000","last_became_peered":"2024-04-25T01:18:02.267086+0000","last_unstale":"2024-04-25T01:18:04.290079+0000","last_undegraded":"2024-04-25T01:18:04.290079+0000","last_fullsized":"2024-04-25T01:18:04.290079+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:03:28.616338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.254418+0000","last_change":"2024-04-25T01:18:03.265044+0000","last_active":"2024-04-25T01:18:04.254418+0000","last_peered":"2024-04-25T01:18:04.254418+0000","last_clean":"2024-04-25T01:18:04.254418+0000","last_became_active":"2024-04-25T01:18:03.264645+0000","last_became_peered":"2024-04-25T01:18:03.264645+0000","last_unstale":"2024-04-25T01:18:04.254418+0000","last_undegraded":"2024-04-25T01:18:04.254418+0000","last_fullsized":"2024-04-25T01:18:04.254418+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:20:51.212419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.290999+0000","last_change":"2024-04-25T01:18:02.262809+0000","last_active":"2024-04-25T01:18:04.290999+0000","last_peered":"2024-04-25T01:18:04.290999+0000","last_clean":"2024-04-25T01:18:04.290999+0000","last_became_active":"2024-04-25T01:18:02.261620+0000","last_became_peered":"2024-04-25T01:18:02.261620+0000","last_unstale":"2024-04-25T01:18:04.290999+0000","last_undegraded":"2024-04-25T01:18:04.290999+0000","last_fullsized":"2024-04-25T01:18:04.290999+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:12.817891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.261149+0000","last_change":"2024-04-25T01:18:03.261875+0000","last_active":"2024-04-25T01:18:04.261149+0000","last_peered":"2024-04-25T01:18:04.261149+0000","last_clean":"2024-04-25T01:18:04.261149+0000","last_became_active":"2024-04-25T01:18:03.261413+0000","last_became_peered":"2024-04-25T01:18:03.261413+0000","last_unstale":"2024-04-25T01:18:04.261149+0000","last_undegraded":"2024-04-25T01:18:04.261149+0000","last_fullsized":"2024-04-25T01:18:04.261149+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:58:30.612559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.289931+0000","last_change":"2024-04-25T01:18:02.257390+0000","last_active":"2024-04-25T01:18:04.289931+0000","last_peered":"2024-04-25T01:18:04.289931+0000","last_clean":"2024-04-25T01:18:04.289931+0000","last_became_active":"2024-04-25T01:18:02.257118+0000","last_became_peered":"2024-04-25T01:18:02.257118+0000","last_unstale":"2024-04-25T01:18:04.289931+0000","last_undegraded":"2024-04-25T01:18:04.289931+0000","last_fullsized":"2024-04-25T01:18:04.289931+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:49:10.075668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291638+0000","last_change":"2024-04-25T01:18:03.260224+0000","last_active":"2024-04-25T01:18:04.291638+0000","last_peered":"2024-04-25T01:18:04.291638+0000","last_clean":"2024-04-25T01:18:04.291638+0000","last_became_active":"2024-04-25T01:18:03.259886+0000","last_became_peered":"2024-04-25T01:18:03.259886+0000","last_unstale":"2024-04-25T01:18:04.291638+0000","last_undegraded":"2024-04-25T01:18:04.291638+0000","last_fullsized":"2024-04-25T01:18:04.291638+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:10:58.007657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.292274+0000","last_change":"2024-04-25T01:18:02.265416+0000","last_active":"2024-04-25T01:18:04.292274+0000","last_peered":"2024-04-25T01:18:04.292274+0000","last_clean":"2024-04-25T01:18:04.292274+0000","last_became_active":"2024-04-25T01:18:02.265123+0000","last_became_peered":"2024-04-25T01:18:02.265123+0000","last_unstale":"2024-04-25T01:18:04.292274+0000","last_undegraded":"2024-04-25T01:18:04.292274+0000","last_fullsized":"2024-04-25T01:18:04.292274+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:18.369587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291157+0000","last_change":"2024-04-25T01:18:03.279931+0000","last_active":"2024-04-25T01:18:04.291157+0000","last_peered":"2024-04-25T01:18:04.291157+0000","last_clean":"2024-04-25T01:18:04.291157+0000","last_became_active":"2024-04-25T01:18:03.279643+0000","last_became_peered":"2024-04-25T01:18:03.279643+0000","last_unstale":"2024-04-25T01:18:04.291157+0000","last_undegraded":"2024-04-25T01:18:04.291157+0000","last_fullsized":"2024-04-25T01:18:04.291157+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:19.385178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.254696+0000","last_change":"2024-04-25T01:18:02.256098+0000","last_active":"2024-04-25T01:18:04.254696+0000","last_peered":"2024-04-25T01:18:04.254696+0000","last_clean":"2024-04-25T01:18:04.254696+0000","last_became_active":"2024-04-25T01:18:02.255609+0000","last_became_peered":"2024-04-25T01:18:02.255609+0000","last_unstale":"2024-04-25T01:18:04.254696+0000","last_undegraded":"2024-04-25T01:18:04.254696+0000","last_fullsized":"2024-04-25T01:18:04.254696+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:46:28.843042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.292099+0000","last_change":"2024-04-25T01:18:03.274406+0000","last_active":"2024-04-25T01:18:04.292099+0000","last_peered":"2024-04-25T01:18:04.292099+0000","last_clean":"2024-04-25T01:18:04.292099+0000","last_became_active":"2024-04-25T01:18:03.274186+0000","last_became_peered":"2024-04-25T01:18:03.274186+0000","last_unstale":"2024-04-25T01:18:04.292099+0000","last_undegraded":"2024-04-25T01:18:04.292099+0000","last_fullsized":"2024-04-25T01:18:04.292099+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:34:33.950222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"39'3","reported_seq":17,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.254272+0000","last_change":"2024-04-25T01:18:02.263678+0000","last_active":"2024-04-25T01:18:04.254272+0000","last_peered":"2024-04-25T01:18:04.254272+0000","last_clean":"2024-04-25T01:18:04.254272+0000","last_became_active":"2024-04-25T01:18:02.263328+0000","last_became_peered":"2024-04-25T01:18:02.263328+0000","last_unstale":"2024-04-25T01:18:04.254272+0000","last_undegraded":"2024-04-25T01:18:04.254272+0000","last_fullsized":"2024-04-25T01:18:04.254272+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:30:37.768204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291123+0000","last_change":"2024-04-25T01:18:03.257743+0000","last_active":"2024-04-25T01:18:04.291123+0000","last_peered":"2024-04-25T01:18:04.291123+0000","last_clean":"2024-04-25T01:18:04.291123+0000","last_became_active":"2024-04-25T01:18:03.257421+0000","last_became_peered":"2024-04-25T01:18:03.257421+0000","last_unstale":"2024-04-25T01:18:04.291123+0000","last_undegraded":"2024-04-25T01:18:04.291123+0000","last_fullsized":"2024-04-25T01:18:04.291123+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:02:29.348596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.14","version":"39'6","reported_seq":20,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.292088+0000","last_change":"2024-04-25T01:18:02.269681+0000","last_active":"2024-04-25T01:18:04.292088+0000","last_peered":"2024-04-25T01:18:04.292088+0000","last_clean":"2024-04-25T01:18:04.292088+0000","last_became_active":"2024-04-25T01:18:02.268702+0000","last_became_peered":"2024-04-25T01:18:02.268702+0000","last_unstale":"2024-04-25T01:18:04.292088+0000","last_undegraded":"2024-04-25T01:18:04.292088+0000","last_fullsized":"2024-04-25T01:18:04.292088+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:50:13.482420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.292471+0000","last_change":"2024-04-25T01:18:03.269967+0000","last_active":"2024-04-25T01:18:04.292471+0000","last_peered":"2024-04-25T01:18:04.292471+0000","last_clean":"2024-04-25T01:18:04.292471+0000","last_became_active":"2024-04-25T01:18:03.269663+0000","last_became_peered":"2024-04-25T01:18:03.269663+0000","last_unstale":"2024-04-25T01:18:04.292471+0000","last_undegraded":"2024-04-25T01:18:04.292471+0000","last_fullsized":"2024-04-25T01:18:04.292471+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:53:56.635142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"39'2","reported_seq":16,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291062+0000","last_change":"2024-04-25T01:18:02.272092+0000","last_active":"2024-04-25T01:18:04.291062+0000","last_peered":"2024-04-25T01:18:04.291062+0000","last_clean":"2024-04-25T01:18:04.291062+0000","last_became_active":"2024-04-25T01:18:02.271767+0000","last_became_peered":"2024-04-25T01:18:02.271767+0000","last_unstale":"2024-04-25T01:18:04.291062+0000","last_undegraded":"2024-04-25T01:18:04.291062+0000","last_fullsized":"2024-04-25T01:18:04.291062+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:10.420303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.261347+0000","last_change":"2024-04-25T01:18:03.258055+0000","last_active":"2024-04-25T01:18:04.261347+0000","last_peered":"2024-04-25T01:18:04.261347+0000","last_clean":"2024-04-25T01:18:04.261347+0000","last_became_active":"2024-04-25T01:18:03.257517+0000","last_became_peered":"2024-04-25T01:18:03.257517+0000","last_unstale":"2024-04-25T01:18:04.261347+0000","last_undegraded":"2024-04-25T01:18:04.261347+0000","last_fullsized":"2024-04-25T01:18:04.261347+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:14:54.012578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.260652+0000","last_change":"2024-04-25T01:18:02.258795+0000","last_active":"2024-04-25T01:18:04.260652+0000","last_peered":"2024-04-25T01:18:04.260652+0000","last_clean":"2024-04-25T01:18:04.260652+0000","last_became_active":"2024-04-25T01:18:02.258494+0000","last_became_peered":"2024-04-25T01:18:02.258494+0000","last_unstale":"2024-04-25T01:18:04.260652+0000","last_undegraded":"2024-04-25T01:18:04.260652+0000","last_fullsized":"2024-04-25T01:18:04.260652+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:16:48.693012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.266960+0000","last_change":"2024-04-25T01:18:03.254569+0000","last_active":"2024-04-25T01:18:04.266960+0000","last_peered":"2024-04-25T01:18:04.266960+0000","last_clean":"2024-04-25T01:18:04.266960+0000","last_became_active":"2024-04-25T01:18:03.254299+0000","last_became_peered":"2024-04-25T01:18:03.254299+0000","last_unstale":"2024-04-25T01:18:04.266960+0000","last_undegraded":"2024-04-25T01:18:04.266960+0000","last_fullsized":"2024-04-25T01:18:04.266960+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:50:36.431482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.254147+0000","last_change":"2024-04-25T01:18:02.254663+0000","last_active":"2024-04-25T01:18:04.254147+0000","last_peered":"2024-04-25T01:18:04.254147+0000","last_clean":"2024-04-25T01:18:04.254147+0000","last_became_active":"2024-04-25T01:18:02.254405+0000","last_became_peered":"2024-04-25T01:18:02.254405+0000","last_unstale":"2024-04-25T01:18:04.254147+0000","last_undegraded":"2024-04-25T01:18:04.254147+0000","last_fullsized":"2024-04-25T01:18:04.254147+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:00:56.792493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.263363+0000","last_change":"2024-04-25T01:18:03.252241+0000","last_active":"2024-04-25T01:18:04.263363+0000","last_peered":"2024-04-25T01:18:04.263363+0000","last_clean":"2024-04-25T01:18:04.263363+0000","last_became_active":"2024-04-25T01:18:03.251874+0000","last_became_peered":"2024-04-25T01:18:03.251874+0000","last_unstale":"2024-04-25T01:18:04.263363+0000","last_undegraded":"2024-04-25T01:18:04.263363+0000","last_fullsized":"2024-04-25T01:18:04.263363+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:16.174537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"39'8","reported_seq":22,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.290580+0000","last_change":"2024-04-25T01:18:02.258502+0000","last_active":"2024-04-25T01:18:04.290580+0000","last_peered":"2024-04-25T01:18:04.290580+0000","last_clean":"2024-04-25T01:18:04.290580+0000","last_became_active":"2024-04-25T01:18:02.258205+0000","last_became_peered":"2024-04-25T01:18:02.258205+0000","last_unstale":"2024-04-25T01:18:04.290580+0000","last_undegraded":"2024-04-25T01:18:04.290580+0000","last_fullsized":"2024-04-25T01:18:04.290580+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:20:28.341160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.260451+0000","last_change":"2024-04-25T01:18:03.275449+0000","last_active":"2024-04-25T01:18:04.260451+0000","last_peered":"2024-04-25T01:18:04.260451+0000","last_clean":"2024-04-25T01:18:04.260451+0000","last_became_active":"2024-04-25T01:18:03.275006+0000","last_became_peered":"2024-04-25T01:18:03.275006+0000","last_unstale":"2024-04-25T01:18:04.260451+0000","last_undegraded":"2024-04-25T01:18:04.260451+0000","last_fullsized":"2024-04-25T01:18:04.260451+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:13.737432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"39'1","reported_seq":15,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291123+0000","last_change":"2024-04-25T01:18:02.259736+0000","last_active":"2024-04-25T01:18:04.291123+0000","last_peered":"2024-04-25T01:18:04.291123+0000","last_clean":"2024-04-25T01:18:04.291123+0000","last_became_active":"2024-04-25T01:18:02.259127+0000","last_became_peered":"2024-04-25T01:18:02.259127+0000","last_unstale":"2024-04-25T01:18:04.291123+0000","last_undegraded":"2024-04-25T01:18:04.291123+0000","last_fullsized":"2024-04-25T01:18:04.291123+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:58:04.484125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.290937+0000","last_change":"2024-04-25T01:18:03.270127+0000","last_active":"2024-04-25T01:18:04.290937+0000","last_peered":"2024-04-25T01:18:04.290937+0000","last_clean":"2024-04-25T01:18:04.290937+0000","last_became_active":"2024-04-25T01:18:03.267471+0000","last_became_peered":"2024-04-25T01:18:03.267471+0000","last_unstale":"2024-04-25T01:18:04.290937+0000","last_undegraded":"2024-04-25T01:18:04.290937+0000","last_fullsized":"2024-04-25T01:18:04.290937+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:30:08.111624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"39'2","reported_seq":16,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.254928+0000","last_change":"2024-04-25T01:18:02.267703+0000","last_active":"2024-04-25T01:18:04.254928+0000","last_peered":"2024-04-25T01:18:04.254928+0000","last_clean":"2024-04-25T01:18:04.254928+0000","last_became_active":"2024-04-25T01:18:02.267415+0000","last_became_peered":"2024-04-25T01:18:02.267415+0000","last_unstale":"2024-04-25T01:18:04.254928+0000","last_undegraded":"2024-04-25T01:18:04.254928+0000","last_fullsized":"2024-04-25T01:18:04.254928+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:09:47.374684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.290879+0000","last_change":"2024-04-25T01:18:03.266233+0000","last_active":"2024-04-25T01:18:04.290879+0000","last_peered":"2024-04-25T01:18:04.290879+0000","last_clean":"2024-04-25T01:18:04.290879+0000","last_became_active":"2024-04-25T01:18:03.265929+0000","last_became_peered":"2024-04-25T01:18:03.265929+0000","last_unstale":"2024-04-25T01:18:04.290879+0000","last_undegraded":"2024-04-25T01:18:04.290879+0000","last_fullsized":"2024-04-25T01:18:04.290879+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:20:29.356751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"39'1","reported_seq":15,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.262922+0000","last_change":"2024-04-25T01:18:02.267511+0000","last_active":"2024-04-25T01:18:04.262922+0000","last_peered":"2024-04-25T01:18:04.262922+0000","last_clean":"2024-04-25T01:18:04.262922+0000","last_became_active":"2024-04-25T01:18:02.266732+0000","last_became_peered":"2024-04-25T01:18:02.266732+0000","last_unstale":"2024-04-25T01:18:04.262922+0000","last_undegraded":"2024-04-25T01:18:04.262922+0000","last_fullsized":"2024-04-25T01:18:04.262922+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:58:04.484125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.262182+0000","last_change":"2024-04-25T01:18:03.255121+0000","last_active":"2024-04-25T01:18:04.262182+0000","last_peered":"2024-04-25T01:18:04.262182+0000","last_clean":"2024-04-25T01:18:04.262182+0000","last_became_active":"2024-04-25T01:18:03.254831+0000","last_became_peered":"2024-04-25T01:18:03.254831+0000","last_unstale":"2024-04-25T01:18:04.262182+0000","last_undegraded":"2024-04-25T01:18:04.262182+0000","last_fullsized":"2024-04-25T01:18:04.262182+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:14:54.012578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.260403+0000","last_change":"2024-04-25T01:18:02.259323+0000","last_active":"2024-04-25T01:18:04.260403+0000","last_peered":"2024-04-25T01:18:04.260403+0000","last_clean":"2024-04-25T01:18:04.260403+0000","last_became_active":"2024-04-25T01:18:02.258937+0000","last_became_peered":"2024-04-25T01:18:02.258937+0000","last_unstale":"2024-04-25T01:18:04.260403+0000","last_undegraded":"2024-04-25T01:18:04.260403+0000","last_fullsized":"2024-04-25T01:18:04.260403+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:12.725383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.253843+0000","last_change":"2024-04-25T01:18:03.253199+0000","last_active":"2024-04-25T01:18:04.253843+0000","last_peered":"2024-04-25T01:18:04.253843+0000","last_clean":"2024-04-25T01:18:04.253843+0000","last_became_active":"2024-04-25T01:18:03.252912+0000","last_became_peered":"2024-04-25T01:18:03.252912+0000","last_unstale":"2024-04-25T01:18:04.253843+0000","last_undegraded":"2024-04-25T01:18:04.253843+0000","last_fullsized":"2024-04-25T01:18:04.253843+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:57:14.448582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"39'2","reported_seq":16,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.259796+0000","last_change":"2024-04-25T01:18:02.267229+0000","last_active":"2024-04-25T01:18:04.259796+0000","last_peered":"2024-04-25T01:18:04.259796+0000","last_clean":"2024-04-25T01:18:04.259796+0000","last_became_active":"2024-04-25T01:18:02.266605+0000","last_became_peered":"2024-04-25T01:18:02.266605+0000","last_unstale":"2024-04-25T01:18:04.259796+0000","last_undegraded":"2024-04-25T01:18:04.259796+0000","last_fullsized":"2024-04-25T01:18:04.259796+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:34:33.983053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291049+0000","last_change":"2024-04-25T01:18:03.274242+0000","last_active":"2024-04-25T01:18:04.291049+0000","last_peered":"2024-04-25T01:18:04.291049+0000","last_clean":"2024-04-25T01:18:04.291049+0000","last_became_active":"2024-04-25T01:18:03.273857+0000","last_became_peered":"2024-04-25T01:18:03.273857+0000","last_unstale":"2024-04-25T01:18:04.291049+0000","last_undegraded":"2024-04-25T01:18:04.291049+0000","last_fullsized":"2024-04-25T01:18:04.291049+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:09:48.390275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.260370+0000","last_change":"2024-04-25T01:18:02.259159+0000","last_active":"2024-04-25T01:18:04.260370+0000","last_peered":"2024-04-25T01:18:04.260370+0000","last_clean":"2024-04-25T01:18:04.260370+0000","last_became_active":"2024-04-25T01:18:02.257389+0000","last_became_peered":"2024-04-25T01:18:02.257389+0000","last_unstale":"2024-04-25T01:18:04.260370+0000","last_undegraded":"2024-04-25T01:18:04.260370+0000","last_fullsized":"2024-04-25T01:18:04.260370+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:43.394014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.267005+0000","last_change":"2024-04-25T01:18:03.274693+0000","last_active":"2024-04-25T01:18:04.267005+0000","last_peered":"2024-04-25T01:18:04.267005+0000","last_clean":"2024-04-25T01:18:04.267005+0000","last_became_active":"2024-04-25T01:18:03.274419+0000","last_became_peered":"2024-04-25T01:18:03.274419+0000","last_unstale":"2024-04-25T01:18:04.267005+0000","last_undegraded":"2024-04-25T01:18:04.267005+0000","last_fullsized":"2024-04-25T01:18:04.267005+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:57:10.276501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291126+0000","last_change":"2024-04-25T01:18:02.254033+0000","last_active":"2024-04-25T01:18:04.291126+0000","last_peered":"2024-04-25T01:18:04.291126+0000","last_clean":"2024-04-25T01:18:04.291126+0000","last_became_active":"2024-04-25T01:18:02.253731+0000","last_became_peered":"2024-04-25T01:18:02.253731+0000","last_unstale":"2024-04-25T01:18:04.291126+0000","last_undegraded":"2024-04-25T01:18:04.291126+0000","last_fullsized":"2024-04-25T01:18:04.291126+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:00:15.697767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.290988+0000","last_change":"2024-04-25T01:18:03.258959+0000","last_active":"2024-04-25T01:18:04.290988+0000","last_peered":"2024-04-25T01:18:04.290988+0000","last_clean":"2024-04-25T01:18:04.290988+0000","last_became_active":"2024-04-25T01:18:03.258462+0000","last_became_peered":"2024-04-25T01:18:03.258462+0000","last_unstale":"2024-04-25T01:18:04.290988+0000","last_undegraded":"2024-04-25T01:18:04.290988+0000","last_fullsized":"2024-04-25T01:18:04.290988+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:46:29.858633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.259349+0000","last_change":"2024-04-25T01:18:02.266874+0000","last_active":"2024-04-25T01:18:04.259349+0000","last_peered":"2024-04-25T01:18:04.259349+0000","last_clean":"2024-04-25T01:18:04.259349+0000","last_became_active":"2024-04-25T01:18:02.266188+0000","last_became_peered":"2024-04-25T01:18:02.266188+0000","last_unstale":"2024-04-25T01:18:04.259349+0000","last_undegraded":"2024-04-25T01:18:04.259349+0000","last_fullsized":"2024-04-25T01:18:04.259349+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:12:59.564111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.263283+0000","last_change":"2024-04-25T01:18:03.269115+0000","last_active":"2024-04-25T01:18:04.263283+0000","last_peered":"2024-04-25T01:18:04.263283+0000","last_clean":"2024-04-25T01:18:04.263283+0000","last_became_active":"2024-04-25T01:18:03.268774+0000","last_became_peered":"2024-04-25T01:18:03.268774+0000","last_unstale":"2024-04-25T01:18:04.263283+0000","last_undegraded":"2024-04-25T01:18:04.263283+0000","last_fullsized":"2024-04-25T01:18:04.263283+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:03:34.263185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"39'2","reported_seq":16,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.254836+0000","last_change":"2024-04-25T01:18:02.257314+0000","last_active":"2024-04-25T01:18:04.254836+0000","last_peered":"2024-04-25T01:18:04.254836+0000","last_clean":"2024-04-25T01:18:04.254836+0000","last_became_active":"2024-04-25T01:18:02.256973+0000","last_became_peered":"2024-04-25T01:18:02.256973+0000","last_unstale":"2024-04-25T01:18:04.254836+0000","last_undegraded":"2024-04-25T01:18:04.254836+0000","last_fullsized":"2024-04-25T01:18:04.254836+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:06:24.081982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.267318+0000","last_change":"2024-04-25T01:18:03.276655+0000","last_active":"2024-04-25T01:18:04.267318+0000","last_peered":"2024-04-25T01:18:04.267318+0000","last_clean":"2024-04-25T01:18:04.267318+0000","last_became_active":"2024-04-25T01:18:03.276023+0000","last_became_peered":"2024-04-25T01:18:03.276023+0000","last_unstale":"2024-04-25T01:18:04.267318+0000","last_undegraded":"2024-04-25T01:18:04.267318+0000","last_fullsized":"2024-04-25T01:18:04.267318+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:00:57.808084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"40'1","reported_seq":15,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.266060+0000","last_change":"2024-04-25T01:18:02.265987+0000","last_active":"2024-04-25T01:18:04.266060+0000","last_peered":"2024-04-25T01:18:04.266060+0000","last_clean":"2024-04-25T01:18:04.266060+0000","last_became_active":"2024-04-25T01:18:02.265704+0000","last_became_peered":"2024-04-25T01:18:02.265704+0000","last_unstale":"2024-04-25T01:18:04.266060+0000","last_undegraded":"2024-04-25T01:18:04.266060+0000","last_fullsized":"2024-04-25T01:18:04.266060+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:03:28.616338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.261930+0000","last_change":"2024-04-25T01:18:03.260610+0000","last_active":"2024-04-25T01:18:04.261930+0000","last_peered":"2024-04-25T01:18:04.261930+0000","last_clean":"2024-04-25T01:18:04.261930+0000","last_became_active":"2024-04-25T01:18:03.259487+0000","last_became_peered":"2024-04-25T01:18:03.259487+0000","last_unstale":"2024-04-25T01:18:04.261930+0000","last_undegraded":"2024-04-25T01:18:04.261930+0000","last_fullsized":"2024-04-25T01:18:04.261930+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:58:30.612559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.263133+0000","last_change":"2024-04-25T01:18:02.269139+0000","last_active":"2024-04-25T01:18:04.263133+0000","last_peered":"2024-04-25T01:18:04.263133+0000","last_clean":"2024-04-25T01:18:04.263133+0000","last_became_active":"2024-04-25T01:18:02.268747+0000","last_became_peered":"2024-04-25T01:18:02.268747+0000","last_unstale":"2024-04-25T01:18:04.263133+0000","last_undegraded":"2024-04-25T01:18:04.263133+0000","last_fullsized":"2024-04-25T01:18:04.263133+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:49:10.075668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.262688+0000","last_change":"2024-04-25T01:18:03.254793+0000","last_active":"2024-04-25T01:18:04.262688+0000","last_peered":"2024-04-25T01:18:04.262688+0000","last_clean":"2024-04-25T01:18:04.262688+0000","last_became_active":"2024-04-25T01:18:03.253467+0000","last_became_peered":"2024-04-25T01:18:03.253467+0000","last_unstale":"2024-04-25T01:18:04.262688+0000","last_undegraded":"2024-04-25T01:18:04.262688+0000","last_fullsized":"2024-04-25T01:18:04.262688+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:54:36.224430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.291357+0000","last_change":"2024-04-25T01:18:02.269410+0000","last_active":"2024-04-25T01:18:04.291357+0000","last_peered":"2024-04-25T01:18:04.291357+0000","last_clean":"2024-04-25T01:18:04.291357+0000","last_became_active":"2024-04-25T01:18:02.268924+0000","last_became_peered":"2024-04-25T01:18:02.268924+0000","last_unstale":"2024-04-25T01:18:04.291357+0000","last_undegraded":"2024-04-25T01:18:04.291357+0000","last_fullsized":"2024-04-25T01:18:04.291357+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:19.010506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.260535+0000","last_change":"2024-04-25T01:18:03.261409+0000","last_active":"2024-04-25T01:18:04.260535+0000","last_peered":"2024-04-25T01:18:04.260535+0000","last_clean":"2024-04-25T01:18:04.260535+0000","last_became_active":"2024-04-25T01:18:03.261126+0000","last_became_peered":"2024-04-25T01:18:03.261126+0000","last_unstale":"2024-04-25T01:18:04.260535+0000","last_undegraded":"2024-04-25T01:18:04.260535+0000","last_fullsized":"2024-04-25T01:18:04.260535+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:53:46.580659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.267157+0000","last_change":"2024-04-25T01:18:02.267491+0000","last_active":"2024-04-25T01:18:04.267157+0000","last_peered":"2024-04-25T01:18:04.267157+0000","last_clean":"2024-04-25T01:18:04.267157+0000","last_became_active":"2024-04-25T01:18:02.267044+0000","last_became_peered":"2024-04-25T01:18:02.267044+0000","last_unstale":"2024-04-25T01:18:04.267157+0000","last_undegraded":"2024-04-25T01:18:04.267157+0000","last_fullsized":"2024-04-25T01:18:04.267157+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:30:07.096033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.254437+0000","last_change":"2024-04-25T01:18:03.254489+0000","last_active":"2024-04-25T01:18:04.254437+0000","last_peered":"2024-04-25T01:18:04.254437+0000","last_clean":"2024-04-25T01:18:04.254437+0000","last_became_active":"2024-04-25T01:18:03.254164+0000","last_became_peered":"2024-04-25T01:18:03.254164+0000","last_unstale":"2024-04-25T01:18:04.254437+0000","last_undegraded":"2024-04-25T01:18:04.254437+0000","last_fullsized":"2024-04-25T01:18:04.254437+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:52:20.724622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.254663+0000","last_change":"2024-04-25T01:18:02.263853+0000","last_active":"2024-04-25T01:18:04.254663+0000","last_peered":"2024-04-25T01:18:04.254663+0000","last_clean":"2024-04-25T01:18:04.254663+0000","last_became_active":"2024-04-25T01:18:02.263501+0000","last_became_peered":"2024-04-25T01:18:02.263501+0000","last_unstale":"2024-04-25T01:18:04.254663+0000","last_undegraded":"2024-04-25T01:18:04.254663+0000","last_fullsized":"2024-04-25T01:18:04.254663+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:06:33.412784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.266916+0000","last_change":"2024-04-25T01:18:03.256275+0000","last_active":"2024-04-25T01:18:04.266916+0000","last_peered":"2024-04-25T01:18:04.266916+0000","last_clean":"2024-04-25T01:18:04.266916+0000","last_became_active":"2024-04-25T01:18:03.255950+0000","last_became_peered":"2024-04-25T01:18:03.255950+0000","last_unstale":"2024-04-25T01:18:04.266916+0000","last_undegraded":"2024-04-25T01:18:04.266916+0000","last_fullsized":"2024-04-25T01:18:04.266916+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:18:05.301920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.292552+0000","last_change":"2024-04-25T01:18:02.269077+0000","last_active":"2024-04-25T01:18:04.292552+0000","last_peered":"2024-04-25T01:18:04.292552+0000","last_clean":"2024-04-25T01:18:04.292552+0000","last_became_active":"2024-04-25T01:18:02.268795+0000","last_became_peered":"2024-04-25T01:18:02.268795+0000","last_unstale":"2024-04-25T01:18:04.292552+0000","last_undegraded":"2024-04-25T01:18:04.292552+0000","last_fullsized":"2024-04-25T01:18:04.292552+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:16.123119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.254719+0000","last_change":"2024-04-25T01:18:03.253285+0000","last_active":"2024-04-25T01:18:04.254719+0000","last_peered":"2024-04-25T01:18:04.254719+0000","last_clean":"2024-04-25T01:18:04.254719+0000","last_became_active":"2024-04-25T01:18:03.252973+0000","last_became_peered":"2024-04-25T01:18:03.252973+0000","last_unstale":"2024-04-25T01:18:04.254719+0000","last_undegraded":"2024-04-25T01:18:04.254719+0000","last_fullsized":"2024-04-25T01:18:04.254719+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:20:54.301682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.266877+0000","last_change":"2024-04-25T01:18:02.267166+0000","last_active":"2024-04-25T01:18:04.266877+0000","last_peered":"2024-04-25T01:18:04.266877+0000","last_clean":"2024-04-25T01:18:04.266877+0000","last_became_active":"2024-04-25T01:18:02.266866+0000","last_became_peered":"2024-04-25T01:18:02.266866+0000","last_unstale":"2024-04-25T01:18:04.266877+0000","last_undegraded":"2024-04-25T01:18:04.266877+0000","last_fullsized":"2024-04-25T01:18:04.266877+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:20:28.341160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":204,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T01:18:04.289984+0000","last_change":"2024-04-25T01:16:17.466569+0000","last_active":"2024-04-25T01:18:04.289984+0000","last_peered":"2024-04-25T01:18:04.289984+0000","last_clean":"2024-04-25T01:18:04.289984+0000","last_became_active":"2024-04-25T01:16:17.465687+0000","last_became_peered":"2024-04-25T01:16:17.465687+0000","last_unstale":"2024-04-25T01:18:04.289984+0000","last_undegraded":"2024-04-25T01:18:04.289984+0000","last_fullsized":"2024-04-25T01:18:04.289984+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:16.440559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:16.440559+0000","last_clean_scrub_stamp":"2024-04-25T01:16:16.440559+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:18:17.493555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542169,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":462848,"data_stored":197122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542168,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":462848,"data_stored":197086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542169,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28052,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343788,"statfs":{"total":96636764160,"available":96608038912,"internally_reserved":0,"allocated":1056768,"data_stored":788002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542168,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27484,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344356,"statfs":{"total":96636764160,"available":96608620544,"internally_reserved":0,"allocated":466944,"data_stored":192096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542168,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":462848,"data_stored":197142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542170,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27484,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344356,"statfs":{"total":96636764160,"available":96608620544,"internally_reserved":0,"allocated":475136,"data_stored":199246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542168,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28060,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343780,"statfs":{"total":96636764160,"available":96608030720,"internally_reserved":0,"allocated":1056768,"data_stored":787442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542169,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":466944,"data_stored":197144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:18:08.111 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T01:18:08.112 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-25T01:18:08.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.368+0000 7f44d3e00640 1 Processor -- start 2024-04-25T01:18:08.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.368+0000 7f44d3e00640 1 -- start start 2024-04-25T01:18:08.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.370+0000 7f44d3e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f44d40aee00 0x7f44d40b4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:08.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.370+0000 7f44d3e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f44d415ff20 0x7f44d40b4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:08.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.370+0000 7f44d3e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44d4160580 con 0x7f44d406d700 2024-04-25T01:18:08.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.370+0000 7f44d3e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f44d406c9d0 con 0x7f44d415ff20 2024-04-25T01:18:08.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.370+0000 7f44d3e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f44d406cc20 con 0x7f44d40aee00 2024-04-25T01:18:08.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.370+0000 7f44d3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f44d415ff20 0x7f44d40b4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:08.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.370+0000 7f44d3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f44d415ff20 0x7f44d40b4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33198/0 (socket says 172.21.15.90:33198) 2024-04-25T01:18:08.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.370+0000 7f44d3400640 1 -- 172.21.15.90:0/861765713 learned_addr learned my addr 172.21.15.90:0/861765713 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:08.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.370+0000 7f44d2a00640 1 --2- 172.21.15.90:0/861765713 >> v2:172.21.15.103:3300/0 conn(0x7f44d40aee00 0x7f44d40b4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:08.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.371+0000 7f44d3400640 1 -- 172.21.15.90:0/861765713 >> v2:172.21.15.103:3300/0 conn(0x7f44d40aee00 msgr2=0x7f44d40b4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:08.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.371+0000 7f44d3400640 1 --2- 172.21.15.90:0/861765713 >> v2:172.21.15.103:3300/0 conn(0x7f44d40aee00 0x7f44d40b4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:08.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.371+0000 7f44d3400640 1 -- 172.21.15.90:0/861765713 >> v1:172.21.15.103:6789/0 conn(0x7f44d406d700 legacy=0x7f44d415f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:08.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.371+0000 7f44d3400640 1 -- 172.21.15.90:0/861765713 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44d406cf00 con 0x7f44d415ff20 2024-04-25T01:18:08.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.372+0000 7f44d3400640 1 --2- 172.21.15.90:0/861765713 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44d415ff20 0x7f44d40b4850 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f44d4072c80 tx=0x7f44cc02a5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b190b5a5e0cb3750 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:08.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.372+0000 7f44d1600640 1 -- 172.21.15.90:0/861765713 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44cc036070 con 0x7f44d415ff20 2024-04-25T01:18:08.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.372+0000 7f44d1600640 1 -- 172.21.15.90:0/861765713 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44cc031600 con 0x7f44d415ff20 2024-04-25T01:18:08.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.372+0000 7f44d1600640 1 -- 172.21.15.90:0/861765713 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44cc0319d0 con 0x7f44d415ff20 2024-04-25T01:18:08.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.373+0000 7f44d3e00640 1 -- 172.21.15.90:0/861765713 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44d415ff20 msgr2=0x7f44d40b4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:08.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.373+0000 7f44d3e00640 1 --2- 172.21.15.90:0/861765713 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44d415ff20 0x7f44d40b4850 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f44d4072c80 tx=0x7f44cc02a5b0 comp rx=0 tx=0).stop 2024-04-25T01:18:08.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.373+0000 7f44d3e00640 1 -- 172.21.15.90:0/861765713 shutdown_connections 2024-04-25T01:18:08.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.373+0000 7f44d3e00640 1 --2- 172.21.15.90:0/861765713 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44d415ff20 0x7f44d40b4850 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:08.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.373+0000 7f44d3e00640 1 --2- 172.21.15.90:0/861765713 >> v2:172.21.15.103:3300/0 conn(0x7f44d40aee00 0x7f44d40b4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:08.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.373+0000 7f44d3e00640 1 -- 172.21.15.90:0/861765713 >> 172.21.15.90:0/861765713 conn(0x7f44d406b120 msgr2=0x7f44d406b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:08.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.373+0000 7f44d3e00640 1 -- 172.21.15.90:0/861765713 shutdown_connections 2024-04-25T01:18:08.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.373+0000 7f44d3e00640 1 -- 172.21.15.90:0/861765713 wait complete. 2024-04-25T01:18:08.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.375+0000 7f44d3e00640 1 Processor -- start 2024-04-25T01:18:08.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.376+0000 7f44d3e00640 1 -- start start 2024-04-25T01:18:08.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.377+0000 7f44d3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44d406d700 0x7f44d4145720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:08.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.377+0000 7f44d3e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f44d40aee00 0x7f44d41554f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:08.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.377+0000 7f44d3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44d415ff20 0x7f44d415d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:08.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.377+0000 7f44d3e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f44d415d370 con 0x7f44d406d700 2024-04-25T01:18:08.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.377+0000 7f44d3e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f44d40af1e0 con 0x7f44d415ff20 2024-04-25T01:18:08.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.377+0000 7f44d3e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f44d415d4f0 con 0x7f44d40aee00 2024-04-25T01:18:08.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.377+0000 7f44d2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44d406d700 0x7f44d4145720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:08.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.377+0000 7f44d2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44d406d700 0x7f44d4145720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33210/0 (socket says 172.21.15.90:33210) 2024-04-25T01:18:08.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.377+0000 7f44d2a00640 1 -- 172.21.15.90:0/1394840082 learned_addr learned my addr 172.21.15.90:0/1394840082 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:08.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.377+0000 7f44d2a00640 1 -- 172.21.15.90:0/1394840082 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f44d40aee00 msgr2=0x7f44d41554f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:08.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.377+0000 7f44d2a00640 1 --2- 172.21.15.90:0/1394840082 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f44d40aee00 0x7f44d41554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:08.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.377+0000 7f44d2a00640 1 -- 172.21.15.90:0/1394840082 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44d415ff20 msgr2=0x7f44d415d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:08.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.377+0000 7f44d3400640 1 --2- 172.21.15.90:0/1394840082 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44d415ff20 0x7f44d415d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:08.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.378+0000 7f44d2a00640 1 --2- 172.21.15.90:0/1394840082 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44d415ff20 0x7f44d415d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:08.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.378+0000 7f44d2a00640 1 -- 172.21.15.90:0/1394840082 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44d4145c60 con 0x7f44d406d700 2024-04-25T01:18:08.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.378+0000 7f44d2a00640 1 --2- 172.21.15.90:0/1394840082 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44d406d700 0x7f44d4145720 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f44c400c820 tx=0x7f44c400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:08.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.378+0000 7f44caa00640 1 -- 172.21.15.90:0/1394840082 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c4015070 con 0x7f44d406d700 2024-04-25T01:18:08.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.378+0000 7f44caa00640 1 -- 172.21.15.90:0/1394840082 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44c4004240 con 0x7f44d406d700 2024-04-25T01:18:08.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.378+0000 7f44caa00640 1 -- 172.21.15.90:0/1394840082 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c400ce10 con 0x7f44d406d700 2024-04-25T01:18:08.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.378+0000 7f44d3e00640 1 -- 172.21.15.90:0/1394840082 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44d41f1fd0 con 0x7f44d406d700 2024-04-25T01:18:08.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.378+0000 7f44d3e00640 1 -- 172.21.15.90:0/1394840082 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44d41f2590 con 0x7f44d406d700 2024-04-25T01:18:08.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.379+0000 7f44d3e00640 1 -- 172.21.15.90:0/1394840082 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44d40753c0 con 0x7f44d406d700 2024-04-25T01:18:08.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.380+0000 7f44caa00640 1 -- 172.21.15.90:0/1394840082 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f44c4007830 con 0x7f44d406d700 2024-04-25T01:18:08.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.381+0000 7f44caa00640 1 --2- 172.21.15.90:0/1394840082 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f44b406be30 0x7f44b406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:08.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.381+0000 7f44caa00640 1 -- 172.21.15.90:0/1394840082 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6101+0+0 (secure 0 0 0) 0x7f44c408aee0 con 0x7f44d406d700 2024-04-25T01:18:08.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.381+0000 7f44d2000640 1 --2- 172.21.15.90:0/1394840082 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f44b406be30 0x7f44b406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:08.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.382+0000 7f44d2000640 1 --2- 172.21.15.90:0/1394840082 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f44b406be30 0x7f44b406e2f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f44d415e6a0 tx=0x7f44c0007990 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:08.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.385+0000 7f44caa00640 1 -- 172.21.15.90:0/1394840082 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f44c4059700 con 0x7f44d406d700 2024-04-25T01:18:08.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.654+0000 7f44d3e00640 1 -- 172.21.15.90:0/1394840082 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f44d406b910 con 0x7f44d406d700 2024-04-25T01:18:08.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.658+0000 7f44caa00640 1 -- 172.21.15.90:0/1394840082 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) ==== 190+0+147 (secure 0 0 0) 0x7f44c4059000 con 0x7f44d406d700 2024-04-25T01:18:08.641 INFO:teuthology.orchestra.run.smithi090.stdout:[client.0] 2024-04-25T01:18:08.642 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQCsrilmTsovDhAAVqcIHshdGrSHncSLyuueoQ== 2024-04-25T01:18:08.642 INFO:teuthology.orchestra.run.smithi090.stdout: caps mds = "allow" 2024-04-25T01:18:08.642 INFO:teuthology.orchestra.run.smithi090.stdout: caps mon = "allow r" 2024-04-25T01:18:08.642 INFO:teuthology.orchestra.run.smithi090.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-25T01:18:08.642 INFO:teuthology.orchestra.run.smithi090.stderr:updated caps for client.0 2024-04-25T01:18:08.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.661+0000 7f44d3e00640 1 -- 172.21.15.90:0/1394840082 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f44b406be30 msgr2=0x7f44b406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:08.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.661+0000 7f44d3e00640 1 --2- 172.21.15.90:0/1394840082 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f44b406be30 0x7f44b406e2f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f44d415e6a0 tx=0x7f44c0007990 comp rx=0 tx=0).stop 2024-04-25T01:18:08.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.661+0000 7f44d3e00640 1 -- 172.21.15.90:0/1394840082 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44d406d700 msgr2=0x7f44d4145720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:08.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.661+0000 7f44d3e00640 1 --2- 172.21.15.90:0/1394840082 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44d406d700 0x7f44d4145720 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f44c400c820 tx=0x7f44c400cce0 comp rx=0 tx=0).stop 2024-04-25T01:18:08.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.661+0000 7f44d3e00640 1 -- 172.21.15.90:0/1394840082 shutdown_connections 2024-04-25T01:18:08.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.661+0000 7f44d3e00640 1 --2- 172.21.15.90:0/1394840082 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f44b406be30 0x7f44b406e2f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:08.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.661+0000 7f44d3e00640 1 --2- 172.21.15.90:0/1394840082 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44d415ff20 0x7f44d415d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:08.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.661+0000 7f44d3e00640 1 --2- 172.21.15.90:0/1394840082 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f44d40aee00 0x7f44d41554f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:08.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.661+0000 7f44d3e00640 1 --2- 172.21.15.90:0/1394840082 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44d406d700 0x7f44d4145720 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:08.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.661+0000 7f44d3e00640 1 -- 172.21.15.90:0/1394840082 >> 172.21.15.90:0/1394840082 conn(0x7f44d406b120 msgr2=0x7f44d415ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:08.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.662+0000 7f44d3e00640 1 -- 172.21.15.90:0/1394840082 shutdown_connections 2024-04-25T01:18:08.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.662+0000 7f44d3e00640 1 -- 172.21.15.90:0/1394840082 wait complete. 2024-04-25T01:18:08.664 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:18:08.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.880+0000 7f4dc8400640 1 Processor -- start 2024-04-25T01:18:08.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.880+0000 7f4dc8400640 1 -- start start 2024-04-25T01:18:08.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.881+0000 7f4dc8400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4dc0145590 0x7f4dc00b4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:08.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.882+0000 7f4dc8400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4dc015f9b0 0x7f4dc015fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:08.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.882+0000 7f4dc8400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4dc00b4240 con 0x7f4dc00ce110 2024-04-25T01:18:08.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.882+0000 7f4dc8400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f4dc0160830 con 0x7f4dc015f9b0 2024-04-25T01:18:08.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.882+0000 7f4dc8400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4dc0160350 con 0x7f4dc0145590 2024-04-25T01:18:08.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.882+0000 7f4dc5000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4dc015f9b0 0x7f4dc015fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:08.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.882+0000 7f4dc5000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4dc015f9b0 0x7f4dc015fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33216/0 (socket says 172.21.15.90:33216) 2024-04-25T01:18:08.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.882+0000 7f4dc5000640 1 -- 172.21.15.90:0/2361422499 learned_addr learned my addr 172.21.15.90:0/2361422499 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:08.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.882+0000 7f4dc5a00640 1 --2- 172.21.15.90:0/2361422499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4dc0145590 0x7f4dc00b4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:08.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.882+0000 7f4dc5000640 1 -- 172.21.15.90:0/2361422499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4dc0145590 msgr2=0x7f4dc00b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:08.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.882+0000 7f4dc5000640 1 --2- 172.21.15.90:0/2361422499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4dc0145590 0x7f4dc00b4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:08.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.882+0000 7f4dc5000640 1 -- 172.21.15.90:0/2361422499 >> v1:172.21.15.103:6789/0 conn(0x7f4dc00ce110 legacy=0x7f4dc00ce5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:08.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.882+0000 7f4dc5000640 1 -- 172.21.15.90:0/2361422499 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dc00b4f60 con 0x7f4dc015f9b0 2024-04-25T01:18:08.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.882+0000 7f4dc5a00640 1 --2- 172.21.15.90:0/2361422499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4dc0145590 0x7f4dc00b4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:08.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.883+0000 7f4dc5000640 1 --2- 172.21.15.90:0/2361422499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dc015f9b0 0x7f4dc015fe10 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f4dac009870 tx=0x7f4dac02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=620d444da1cb13b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:08.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.884+0000 7f4dbfe00640 1 -- 172.21.15.90:0/2361422499 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dac037070 con 0x7f4dc015f9b0 2024-04-25T01:18:08.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.884+0000 7f4dbfe00640 1 -- 172.21.15.90:0/2361422499 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dac032440 con 0x7f4dc015f9b0 2024-04-25T01:18:08.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.884+0000 7f4dbfe00640 1 -- 172.21.15.90:0/2361422499 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dac032810 con 0x7f4dc015f9b0 2024-04-25T01:18:08.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.884+0000 7f4dc8400640 1 -- 172.21.15.90:0/2361422499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dc015f9b0 msgr2=0x7f4dc015fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:08.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.884+0000 7f4dc8400640 1 --2- 172.21.15.90:0/2361422499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dc015f9b0 0x7f4dc015fe10 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f4dac009870 tx=0x7f4dac02a210 comp rx=0 tx=0).stop 2024-04-25T01:18:08.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.884+0000 7f4dc8400640 1 -- 172.21.15.90:0/2361422499 shutdown_connections 2024-04-25T01:18:08.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.884+0000 7f4dc8400640 1 --2- 172.21.15.90:0/2361422499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dc015f9b0 0x7f4dc015fe10 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:08.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.884+0000 7f4dc8400640 1 --2- 172.21.15.90:0/2361422499 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4dc0145590 0x7f4dc00b4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:08.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.884+0000 7f4dc8400640 1 -- 172.21.15.90:0/2361422499 >> 172.21.15.90:0/2361422499 conn(0x7f4dc00b1820 msgr2=0x7f4dc00b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:08.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.885+0000 7f4dc8400640 1 -- 172.21.15.90:0/2361422499 shutdown_connections 2024-04-25T01:18:08.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.885+0000 7f4dc8400640 1 -- 172.21.15.90:0/2361422499 wait complete. 2024-04-25T01:18:08.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.886+0000 7f4dc8400640 1 Processor -- start 2024-04-25T01:18:08.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.886+0000 7f4dc8400640 1 -- start start 2024-04-25T01:18:08.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.887+0000 7f4dc8400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dc00ce110 0x7f4dc0072170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:08.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.887+0000 7f4dc8400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4dc0145590 0x7f4dc0156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:08.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.887+0000 7f4dc8400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4dc015f9b0 0x7f4dc0156db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:08.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.887+0000 7f4dc8400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc00b3c80 con 0x7f4dc00ce110 2024-04-25T01:18:08.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.887+0000 7f4dc8400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc006cde0 con 0x7f4dc0145590 2024-04-25T01:18:08.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.887+0000 7f4dc8400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4dc00b3f60 con 0x7f4dc015f9b0 2024-04-25T01:18:08.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.888+0000 7f4dc5a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dc00ce110 0x7f4dc0072170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:08.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.888+0000 7f4dc5a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dc00ce110 0x7f4dc0072170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33230/0 (socket says 172.21.15.90:33230) 2024-04-25T01:18:08.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.888+0000 7f4dc5a00640 1 -- 172.21.15.90:0/3520675559 learned_addr learned my addr 172.21.15.90:0/3520675559 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:08.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.888+0000 7f4dc5000640 1 --2- 172.21.15.90:0/3520675559 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4dc0145590 0x7f4dc0156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:08.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.888+0000 7f4dc6400640 1 --2- 172.21.15.90:0/3520675559 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4dc015f9b0 0x7f4dc0156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:08.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.888+0000 7f4dc5a00640 1 -- 172.21.15.90:0/3520675559 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4dc015f9b0 msgr2=0x7f4dc0156db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:08.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.888+0000 7f4dc5a00640 1 --2- 172.21.15.90:0/3520675559 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4dc015f9b0 0x7f4dc0156db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:08.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.888+0000 7f4dc5a00640 1 -- 172.21.15.90:0/3520675559 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4dc0145590 msgr2=0x7f4dc0156870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:08.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.888+0000 7f4dc5a00640 1 --2- 172.21.15.90:0/3520675559 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4dc0145590 0x7f4dc0156870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:08.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.888+0000 7f4dc5a00640 1 -- 172.21.15.90:0/3520675559 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dc015b6e0 con 0x7f4dc00ce110 2024-04-25T01:18:08.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.889+0000 7f4dc5a00640 1 --2- 172.21.15.90:0/3520675559 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dc00ce110 0x7f4dc0072170 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f4db400c820 tx=0x7f4db400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:08.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.889+0000 7f4dbd600640 1 -- 172.21.15.90:0/3520675559 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db4015070 con 0x7f4dc00ce110 2024-04-25T01:18:08.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.889+0000 7f4dbd600640 1 -- 172.21.15.90:0/3520675559 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4db4007a30 con 0x7f4dc00ce110 2024-04-25T01:18:08.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.889+0000 7f4dbd600640 1 -- 172.21.15.90:0/3520675559 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db4007e00 con 0x7f4dc00ce110 2024-04-25T01:18:08.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.889+0000 7f4dc8400640 1 -- 172.21.15.90:0/3520675559 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dc015b9c0 con 0x7f4dc00ce110 2024-04-25T01:18:08.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.889+0000 7f4dc8400640 1 -- 172.21.15.90:0/3520675559 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dc015bf80 con 0x7f4dc00ce110 2024-04-25T01:18:08.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.891+0000 7f4dc8400640 1 -- 172.21.15.90:0/3520675559 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4dc0068fb0 con 0x7f4dc00ce110 2024-04-25T01:18:08.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.892+0000 7f4dbd600640 1 -- 172.21.15.90:0/3520675559 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4db4018070 con 0x7f4dc00ce110 2024-04-25T01:18:08.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.892+0000 7f4dbd600640 1 --2- 172.21.15.90:0/3520675559 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4da006be80 0x7f4da006e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:08.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.893+0000 7f4dbd600640 1 -- 172.21.15.90:0/3520675559 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6101+0+0 (secure 0 0 0) 0x7f4db408b090 con 0x7f4dc00ce110 2024-04-25T01:18:08.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.893+0000 7f4dc5000640 1 --2- 172.21.15.90:0/3520675559 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4da006be80 0x7f4da006e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:08.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.893+0000 7f4dc5000640 1 --2- 172.21.15.90:0/3520675559 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4da006be80 0x7f4da006e340 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4dac034000 tx=0x7f4dac02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:08.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:08.897+0000 7f4dbd600640 1 -- 172.21.15.90:0/3520675559 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4db4059800 con 0x7f4dc00ce110 2024-04-25T01:18:09.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.133+0000 7f4dc8400640 1 -- 172.21.15.90:0/3520675559 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4dc006ea50 con 0x7f4dc00ce110 2024-04-25T01:18:09.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.134+0000 7f4dbd600640 1 -- 172.21.15.90:0/3520675559 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5184 (secure 0 0 0) 0x7f4db4059100 con 0x7f4dc00ce110 2024-04-25T01:18:09.118 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:09.118 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4598,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1475874994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1475874994},{"type":"v1","addr":"172.21.15.90:6835","nonce":1475874994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4610,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6834/1993144745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1993144745},{"type":"v1","addr":"172.21.15.103:6834","nonce":1993144745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4840,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2376794333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6833","nonce":2376794333},{"type":"v1","addr":"172.21.15.103:6835","nonce":2376794333}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:18:02.701652+0000","modified":"2024-04-25T01:18:05.730947+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5334},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5334":{"gid":5334,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.90:6837/431390375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":431390375},{"type":"v1","addr":"172.21.15.90:6837","nonce":431390375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5334,"qdb_cluster":[5334]},"id":2}]} 2024-04-25T01:18:09.118 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 18 2024-04-25T01:18:09.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.137+0000 7f4dc8400640 1 -- 172.21.15.90:0/3520675559 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4da006be80 msgr2=0x7f4da006e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:09.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.137+0000 7f4dc8400640 1 --2- 172.21.15.90:0/3520675559 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4da006be80 0x7f4da006e340 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4dac034000 tx=0x7f4dac02e000 comp rx=0 tx=0).stop 2024-04-25T01:18:09.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.137+0000 7f4dc8400640 1 -- 172.21.15.90:0/3520675559 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dc00ce110 msgr2=0x7f4dc0072170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:09.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.137+0000 7f4dc8400640 1 --2- 172.21.15.90:0/3520675559 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dc00ce110 0x7f4dc0072170 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f4db400c820 tx=0x7f4db400cc20 comp rx=0 tx=0).stop 2024-04-25T01:18:09.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.137+0000 7f4dc8400640 1 -- 172.21.15.90:0/3520675559 shutdown_connections 2024-04-25T01:18:09.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.137+0000 7f4dc8400640 1 --2- 172.21.15.90:0/3520675559 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4da006be80 0x7f4da006e340 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.137+0000 7f4dc8400640 1 --2- 172.21.15.90:0/3520675559 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4dc015f9b0 0x7f4dc0156db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.137+0000 7f4dc8400640 1 --2- 172.21.15.90:0/3520675559 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4dc0145590 0x7f4dc0156870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.137+0000 7f4dc8400640 1 --2- 172.21.15.90:0/3520675559 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dc00ce110 0x7f4dc0072170 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.137+0000 7f4dc8400640 1 -- 172.21.15.90:0/3520675559 >> 172.21.15.90:0/3520675559 conn(0x7f4dc00b1820 msgr2=0x7f4dc006de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:09.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.137+0000 7f4dc8400640 1 -- 172.21.15.90:0/3520675559 shutdown_connections 2024-04-25T01:18:09.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.137+0000 7f4dc8400640 1 -- 172.21.15.90:0/3520675559 wait complete. 2024-04-25T01:18:09.542 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T01:18:02.701652+0000', 'modified': '2024-04-25T01:18:05.730947+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5334}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5334': {'gid': 5334, 'name': 'c', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.90:6837/431390375', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6836', 'nonce': 431390375}, {'type': 'v1', 'addr': '172.21.15.90:6837', 'nonce': 431390375}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5334, 'qdb_cluster': [5334]} 2024-04-25T01:18:09.542 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T01:18:09.542 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-25T01:18:09.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.767+0000 7f3111600640 1 Processor -- start 2024-04-25T01:18:09.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.767+0000 7f3111600640 1 -- start start 2024-04-25T01:18:09.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.769+0000 7f3111600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f310c0b4db0 0x7f310c162360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:09.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.769+0000 7f3111600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f310c15e1f0 0x7f310c0719c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:09.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.769+0000 7f3111600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f310c0b3a10 con 0x7f310c15de10 2024-04-25T01:18:09.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.769+0000 7f3111600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f310c0b3cf0 con 0x7f310c15e1f0 2024-04-25T01:18:09.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.769+0000 7f3111600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f310c0b3fd0 con 0x7f310c0b4db0 2024-04-25T01:18:09.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.769+0000 7f310aa00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f310c15de10 0x7f310c161c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:49414/0 (socket says 172.21.15.90:49414) 2024-04-25T01:18:09.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.769+0000 7f310aa00640 1 -- 172.21.15.90:0/3578430875 learned_addr learned my addr 172.21.15.90:0/3578430875 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:09.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.769+0000 7f310b400640 1 --2- 172.21.15.90:0/3578430875 >> v2:172.21.15.90:3300/0 conn(0x7f310c15e1f0 0x7f310c0719c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:09.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.770+0000 7f310a000640 1 --2- 172.21.15.90:0/3578430875 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f310c0b4db0 0x7f310c162360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:09.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.770+0000 7f3109600640 1 -- 172.21.15.90:0/3578430875 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3346017781 0 0) 0x7f310c0b3a10 con 0x7f310c15de10 2024-04-25T01:18:09.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.770+0000 7f310b400640 1 -- 172.21.15.90:0/3578430875 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f310c0b4db0 msgr2=0x7f310c162360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:09.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.770+0000 7f310b400640 1 --2- 172.21.15.90:0/3578430875 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f310c0b4db0 0x7f310c162360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.770+0000 7f310b400640 1 -- 172.21.15.90:0/3578430875 >> v1:172.21.15.90:6789/0 conn(0x7f310c15de10 legacy=0x7f310c161c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:09.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.770+0000 7f310b400640 1 -- 172.21.15.90:0/3578430875 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f310c0b4660 con 0x7f310c15e1f0 2024-04-25T01:18:09.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.771+0000 7f310b400640 1 --2- 172.21.15.90:0/3578430875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f310c15e1f0 0x7f310c0719c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f31040097a0 tx=0x7f310402a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd55d0792480459e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:09.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.771+0000 7f3109600640 1 -- 172.21.15.90:0/3578430875 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3104037070 con 0x7f310c15e1f0 2024-04-25T01:18:09.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.771+0000 7f3109600640 1 -- 172.21.15.90:0/3578430875 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f310402adb0 con 0x7f310c15e1f0 2024-04-25T01:18:09.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.771+0000 7f3109600640 1 -- 172.21.15.90:0/3578430875 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3104032660 con 0x7f310c15e1f0 2024-04-25T01:18:09.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.772+0000 7f3111600640 1 -- 172.21.15.90:0/3578430875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f310c15e1f0 msgr2=0x7f310c0719c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:09.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.772+0000 7f3111600640 1 --2- 172.21.15.90:0/3578430875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f310c15e1f0 0x7f310c0719c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f31040097a0 tx=0x7f310402a060 comp rx=0 tx=0).stop 2024-04-25T01:18:09.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.772+0000 7f3111600640 1 -- 172.21.15.90:0/3578430875 shutdown_connections 2024-04-25T01:18:09.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.772+0000 7f3111600640 1 --2- 172.21.15.90:0/3578430875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f310c15e1f0 0x7f310c0719c0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.772+0000 7f3111600640 1 --2- 172.21.15.90:0/3578430875 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f310c0b4db0 0x7f310c162360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.772+0000 7f3111600640 1 -- 172.21.15.90:0/3578430875 >> 172.21.15.90:0/3578430875 conn(0x7f310c0b1820 msgr2=0x7f310c0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:09.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.772+0000 7f3111600640 1 -- 172.21.15.90:0/3578430875 shutdown_connections 2024-04-25T01:18:09.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.772+0000 7f3111600640 1 -- 172.21.15.90:0/3578430875 wait complete. 2024-04-25T01:18:09.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.773+0000 7f3111600640 1 Processor -- start 2024-04-25T01:18:09.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.774+0000 7f3111600640 1 -- start start 2024-04-25T01:18:09.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.774+0000 7f3111600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f310c0b4db0 0x7f310c157e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:09.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.774+0000 7f3111600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f310c15de10 0x7f310c157740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:09.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.775+0000 7f3111600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f310c15e1f0 0x7f310c159e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:09.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.775+0000 7f3111600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f310c071f00 con 0x7f310c0b4db0 2024-04-25T01:18:09.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.775+0000 7f3111600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f310c162ab0 con 0x7f310c15e1f0 2024-04-25T01:18:09.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.775+0000 7f3111600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f310c072080 con 0x7f310c15de10 2024-04-25T01:18:09.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.775+0000 7f310aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f310c0b4db0 0x7f310c157e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:09.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.775+0000 7f310aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f310c0b4db0 0x7f310c157e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33238/0 (socket says 172.21.15.90:33238) 2024-04-25T01:18:09.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.775+0000 7f310aa00640 1 -- 172.21.15.90:0/2126467741 learned_addr learned my addr 172.21.15.90:0/2126467741 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:09.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.775+0000 7f310aa00640 1 -- 172.21.15.90:0/2126467741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f310c15de10 msgr2=0x7f310c157740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:09.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.775+0000 7f310aa00640 1 --2- 172.21.15.90:0/2126467741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f310c15de10 0x7f310c157740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.775+0000 7f310aa00640 1 -- 172.21.15.90:0/2126467741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f310c15e1f0 msgr2=0x7f310c159e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:09.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.775+0000 7f310aa00640 1 --2- 172.21.15.90:0/2126467741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f310c15e1f0 0x7f310c159e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.775+0000 7f310aa00640 1 -- 172.21.15.90:0/2126467741 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f310c15a340 con 0x7f310c0b4db0 2024-04-25T01:18:09.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.775+0000 7f310aa00640 1 --2- 172.21.15.90:0/2126467741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f310c0b4db0 0x7f310c157e80 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f30fc00c8c0 tx=0x7f30fc00cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:09.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.776+0000 7f3102a00640 1 -- 172.21.15.90:0/2126467741 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30fc018070 con 0x7f310c0b4db0 2024-04-25T01:18:09.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.776+0000 7f3102a00640 1 -- 172.21.15.90:0/2126467741 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30fc00a640 con 0x7f310c0b4db0 2024-04-25T01:18:09.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.776+0000 7f3102a00640 1 -- 172.21.15.90:0/2126467741 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30fc00aa30 con 0x7f310c0b4db0 2024-04-25T01:18:09.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.776+0000 7f3111600640 1 -- 172.21.15.90:0/2126467741 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f310c158580 con 0x7f310c0b4db0 2024-04-25T01:18:09.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.776+0000 7f3111600640 1 -- 172.21.15.90:0/2126467741 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f310c158b40 con 0x7f310c0b4db0 2024-04-25T01:18:09.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.778+0000 7f3102a00640 1 -- 172.21.15.90:0/2126467741 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f30fc01b070 con 0x7f310c0b4db0 2024-04-25T01:18:09.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.779+0000 7f3102a00640 1 --2- 172.21.15.90:0/2126467741 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f30e806be30 0x7f30e806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:09.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.779+0000 7f3102a00640 1 -- 172.21.15.90:0/2126467741 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6159+0+0 (secure 0 0 0) 0x7f30fc08a770 con 0x7f310c0b4db0 2024-04-25T01:18:09.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.779+0000 7f310a000640 1 --2- 172.21.15.90:0/2126467741 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f30e806be30 0x7f30e806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:09.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.780+0000 7f31145be740 1 Processor -- start 2024-04-25T01:18:09.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.780+0000 7f31145be740 1 -- start start 2024-04-25T01:18:09.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.780+0000 7f310a000640 1 --2- 172.21.15.90:0/2126467741 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f30e806be30 0x7f30e806e2f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f310c15cae0 tx=0x7f30f4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:09.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.781+0000 7f31145be740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56354b330510 0x56354b3308f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:09.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.781+0000 7f31145be740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56354b34ddb0 0x56354b34e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:09.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.781+0000 7f31145be740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56354b3061f0 0x56354b34c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:09.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.781+0000 7f31145be740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56354b205ed0 con 0x56354b330510 2024-04-25T01:18:09.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.781+0000 7f31145be740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56354b268df0 con 0x56354b3061f0 2024-04-25T01:18:09.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.781+0000 7f31145be740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x56354b2118d0 con 0x56354b34ddb0 2024-04-25T01:18:09.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.781+0000 7f310aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56354b330510 0x56354b3308f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:09.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.781+0000 7f310aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56354b330510 0x56354b3308f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33246/0 (socket says 172.21.15.90:33246) 2024-04-25T01:18:09.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.781+0000 7f310aa00640 1 -- 172.21.15.90:0/355533473 learned_addr learned my addr 172.21.15.90:0/355533473 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:09.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.781+0000 7f310aa00640 1 -- 172.21.15.90:0/355533473 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56354b34ddb0 msgr2=0x56354b34e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:09.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.781+0000 7f310aa00640 1 --2- 172.21.15.90:0/355533473 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56354b34ddb0 0x56354b34e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.781+0000 7f310aa00640 1 -- 172.21.15.90:0/355533473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56354b3061f0 msgr2=0x56354b34c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:09.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.781+0000 7f310aa00640 1 --2- 172.21.15.90:0/355533473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56354b3061f0 0x56354b34c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.781+0000 7f310aa00640 1 -- 172.21.15.90:0/355533473 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56354af7d2d0 con 0x56354b330510 2024-04-25T01:18:09.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.782+0000 7f310aa00640 1 --2- 172.21.15.90:0/355533473 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56354b330510 0x56354b3308f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f30fc058750 tx=0x7f30fc058880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=857a049c05a3bf4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:09.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.782+0000 7f30fb000640 1 -- 172.21.15.90:0/355533473 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30fc056860 con 0x56354b330510 2024-04-25T01:18:09.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.782+0000 7f30fb000640 1 -- 172.21.15.90:0/355533473 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30fc056ac0 con 0x56354b330510 2024-04-25T01:18:09.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.782+0000 7f30fb000640 1 -- 172.21.15.90:0/355533473 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30fc00abd0 con 0x56354b330510 2024-04-25T01:18:09.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.782+0000 7f31145be740 1 -- 172.21.15.90:0/355533473 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56354b330510 msgr2=0x56354b3308f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:09.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.782+0000 7f31145be740 1 --2- 172.21.15.90:0/355533473 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56354b330510 0x56354b3308f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f30fc058750 tx=0x7f30fc058880 comp rx=0 tx=0).stop 2024-04-25T01:18:09.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.783+0000 7f31145be740 1 -- 172.21.15.90:0/355533473 shutdown_connections 2024-04-25T01:18:09.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.783+0000 7f31145be740 1 --2- 172.21.15.90:0/355533473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56354b3061f0 0x56354b34c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.783+0000 7f31145be740 1 --2- 172.21.15.90:0/355533473 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56354b34ddb0 0x56354b34e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.783+0000 7f31145be740 1 --2- 172.21.15.90:0/355533473 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56354b330510 0x56354b3308f0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.783+0000 7f31145be740 1 -- 172.21.15.90:0/355533473 >> 172.21.15.90:0/355533473 conn(0x56354b302b20 msgr2=0x56354b34b320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:09.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.783+0000 7f31145be740 1 -- 172.21.15.90:0/355533473 shutdown_connections 2024-04-25T01:18:09.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.783+0000 7f31145be740 1 -- 172.21.15.90:0/355533473 wait complete. 2024-04-25T01:18:09.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.783+0000 7f31145be740 1 -- start start 2024-04-25T01:18:09.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.784+0000 7f31145be740 1 Processor -- start 2024-04-25T01:18:09.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.785+0000 7f31145be740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56354b330510 0x56354b3341c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:09.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.785+0000 7f31145be740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56354b34ddb0 0x56354b32b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:09.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.785+0000 7f31145be740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56354b3329b0 0x56354b3439c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:09.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.785+0000 7f31145be740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56354b224d20 con 0x56354b3329b0 2024-04-25T01:18:09.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.785+0000 7f31145be740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56354b264db0 con 0x56354b34ddb0 2024-04-25T01:18:09.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.785+0000 7f31145be740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x56354b306f90 con 0x56354b330510 2024-04-25T01:18:09.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.785+0000 7f31145be740 10 client.0 _ll_register_callbacks cb 0x56354b25ff10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:18:09.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.785+0000 7f310b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56354b3329b0 0x56354b3439c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:09.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.785+0000 7f310b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56354b3329b0 0x56354b3439c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33256/0 (socket says 172.21.15.90:33256) 2024-04-25T01:18:09.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.785+0000 7f310b400640 1 -- 172.21.15.90:0/1801329982 learned_addr learned my addr 172.21.15.90:0/1801329982 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:09.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.785+0000 7f310a000640 1 --2- 172.21.15.90:0/1801329982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56354b34ddb0 0x56354b32b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:09.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.785+0000 7f310aa00640 1 --2- 172.21.15.90:0/1801329982 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56354b330510 0x56354b3341c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:09.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.785+0000 7f310b400640 1 -- 172.21.15.90:0/1801329982 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56354b330510 msgr2=0x56354b3341c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:09.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.785+0000 7f310b400640 1 --2- 172.21.15.90:0/1801329982 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56354b330510 0x56354b3341c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.785+0000 7f310b400640 1 -- 172.21.15.90:0/1801329982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56354b34ddb0 msgr2=0x56354b32b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:09.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.785+0000 7f310b400640 1 --2- 172.21.15.90:0/1801329982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56354b34ddb0 0x56354b32b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.785+0000 7f310b400640 1 -- 172.21.15.90:0/1801329982 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56354b0476b0 con 0x56354b3329b0 2024-04-25T01:18:09.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.786+0000 7f310b400640 1 --2- 172.21.15.90:0/1801329982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56354b3329b0 0x56354b3439c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f31040098c0 tx=0x7f31040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd095fd470c3469d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:09.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.786+0000 7f30ef400640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T01:18:09.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.786+0000 7f30ef400640 1 -- 172.21.15.90:0/1801329982 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3104040020 con 0x56354b3329b0 2024-04-25T01:18:09.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.786+0000 7f30ef400640 1 -- 172.21.15.90:0/1801329982 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(41..41 src has 1..41) ==== 6159+0+0 (secure 0 0 0) 0x7f3104003810 con 0x56354b3329b0 2024-04-25T01:18:09.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.786+0000 7f31145be740 1 -- 172.21.15.90:0/1801329982 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30fc056ac0 con 0x56354b3329b0 2024-04-25T01:18:09.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.786+0000 7f30ef400640 1 -- 172.21.15.90:0/1801329982 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7f310404a050 con 0x56354b3329b0 2024-04-25T01:18:09.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.786+0000 7f31145be740 10 client.5508 fetch_fsmap learned FSMap version 18 2024-04-25T01:18:09.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.786+0000 7f31145be740 1 -- 172.21.15.90:0/1801329982 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) -- 0x56354b333400 con 0x56354b3329b0 2024-04-25T01:18:09.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.786+0000 7f30ef400640 1 -- 172.21.15.90:0/1801329982 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30fc056ac0 con 0x56354b3329b0 2024-04-25T01:18:09.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.787+0000 7f30ef400640 1 -- 172.21.15.90:0/1801329982 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 18) ==== 3494+0+0 (secure 0 0 0) 0x7f310404d020 con 0x56354b3329b0 2024-04-25T01:18:09.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.787+0000 7f31145be740 10 client.5508 fetch_fsmap finished waiting for FSMap version 18 2024-04-25T01:18:09.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.787+0000 7f31145be740 10 client.5508 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T01:18:09.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.787+0000 7f31145be740 20 client.5508 populate_metadata read hostname 'smithi090' 2024-04-25T01:18:09.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.787+0000 7f31145be740 1 --2- 172.21.15.90:0/1801329982 >> [v2:172.21.15.90:6836/431390375,v1:172.21.15.90:6837/431390375] conn(0x56354b331b00 0x56354b369f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:09.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.787+0000 7f31145be740 4 client.5508 mds_command: new command op to 5334 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:18:09.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.787+0000 7f31145be740 1 -- 172.21.15.90:0/1801329982 --> [v2:172.21.15.90:6836/431390375,v1:172.21.15.90:6837/431390375] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56354b276330 con 0x56354b331b00 2024-04-25T01:18:09.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.787+0000 7f310aa00640 1 --2- 172.21.15.90:0/1801329982 >> [v2:172.21.15.90:6836/431390375,v1:172.21.15.90:6837/431390375] conn(0x56354b331b00 0x56354b369f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:09.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.787+0000 7f310aa00640 1 --2- 172.21.15.90:0/1801329982 >> [v2:172.21.15.90:6836/431390375,v1:172.21.15.90:6837/431390375] conn(0x56354b331b00 0x56354b369f80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d2c4779f7616d06 server_cookie=f2e6c173cbb612d3 in_seq=0 out_seq=0 2024-04-25T01:18:09.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.787+0000 7f30ef400640 10 client.5508 ms_handle_connect on v2:172.21.15.90:6836/431390375 2024-04-25T01:18:09.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.788+0000 7f30ef400640 1 -- 172.21.15.90:0/1801329982 <== mds.0 v2:172.21.15.90:6836/431390375 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56354b276330 con 0x56354b331b00 2024-04-25T01:18:09.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.788+0000 7f30ef400640 10 client.5508 handle_command_reply: tid=0 2024-04-25T01:18:09.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.788+0000 7f31145be740 1 client.5508 shutdown 2024-04-25T01:18:09.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.789+0000 7f31145be740 1 -- 172.21.15.90:0/1801329982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56354b3329b0 msgr2=0x56354b3439c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:09.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.789+0000 7f31145be740 1 --2- 172.21.15.90:0/1801329982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56354b3329b0 0x56354b3439c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f31040098c0 tx=0x7f31040046c0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.789+0000 7f31145be740 1 -- 172.21.15.90:0/1801329982 shutdown_connections 2024-04-25T01:18:09.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.789+0000 7f31145be740 1 --2- 172.21.15.90:0/1801329982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56354b3329b0 0x56354b3439c0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.789+0000 7f31145be740 1 --2- 172.21.15.90:0/1801329982 >> [v2:172.21.15.90:6836/431390375,v1:172.21.15.90:6837/431390375] conn(0x56354b331b00 0x56354b369f80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.789+0000 7f31145be740 1 --2- 172.21.15.90:0/1801329982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56354b34ddb0 0x56354b32b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.789+0000 7f31145be740 1 --2- 172.21.15.90:0/1801329982 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56354b330510 0x56354b3341c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.789+0000 7f31145be740 1 -- 172.21.15.90:0/1801329982 >> 172.21.15.90:0/1801329982 conn(0x56354b3061f0 msgr2=0x56354b34f130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:09.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.789+0000 7f30ef400640 0 client.5508 ms_handle_reset on v2:172.21.15.90:6836/431390375 2024-04-25T01:18:09.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.789+0000 7f31145be740 1 -- 172.21.15.90:0/1801329982 shutdown_connections 2024-04-25T01:18:09.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.789+0000 7f31145be740 1 -- 172.21.15.90:0/1801329982 wait complete. 2024-04-25T01:18:09.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.789+0000 7f31145be740 20 client.5508 trim_cache size 0 max 16384 2024-04-25T01:18:09.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.808+0000 7f31145be740 1 Processor -- start 2024-04-25T01:18:09.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.808+0000 7f31145be740 1 -- start start 2024-04-25T01:18:09.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.809+0000 7f31145be740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56354b330510 0x56354b34a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:09.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.809+0000 7f31145be740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56354b3329b0 0x56354b2fbb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:09.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.809+0000 7f31145be740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56354b34ddb0 0x56354b3067c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:09.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.809+0000 7f31145be740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f310c1ffae0 con 0x56354b3329b0 2024-04-25T01:18:09.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.809+0000 7f31145be740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f310c0cd0e0 con 0x56354b330510 2024-04-25T01:18:09.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.809+0000 7f31145be740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f310c1ff490 con 0x56354b34ddb0 2024-04-25T01:18:09.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.809+0000 7f310a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56354b3329b0 0x56354b2fbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:09.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.809+0000 7f310a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56354b3329b0 0x56354b2fbb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33258/0 (socket says 172.21.15.90:33258) 2024-04-25T01:18:09.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.809+0000 7f310a000640 1 -- 172.21.15.90:0/1922108411 learned_addr learned my addr 172.21.15.90:0/1922108411 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:09.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.810+0000 7f310a000640 1 -- 172.21.15.90:0/1922108411 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56354b34ddb0 msgr2=0x56354b3067c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:09.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.810+0000 7f310aa00640 1 --2- 172.21.15.90:0/1922108411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56354b330510 0x56354b34a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:09.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.810+0000 7f310a000640 1 --2- 172.21.15.90:0/1922108411 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56354b34ddb0 0x56354b3067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.810+0000 7f310a000640 1 -- 172.21.15.90:0/1922108411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56354b330510 msgr2=0x56354b34a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:09.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.810+0000 7f310a000640 1 --2- 172.21.15.90:0/1922108411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56354b330510 0x56354b34a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.810+0000 7f310a000640 1 -- 172.21.15.90:0/1922108411 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56354b301fe0 con 0x56354b3329b0 2024-04-25T01:18:09.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.810+0000 7f310a000640 1 --2- 172.21.15.90:0/1922108411 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56354b3329b0 0x56354b2fbb30 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f30f400d910 tx=0x7f30f4010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f6b78a613d082b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:09.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.810+0000 7f30fa600640 1 -- 172.21.15.90:0/1922108411 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f4014930 con 0x56354b3329b0 2024-04-25T01:18:09.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.810+0000 7f30fa600640 1 -- 172.21.15.90:0/1922108411 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30f4025070 con 0x56354b3329b0 2024-04-25T01:18:09.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.810+0000 7f30fa600640 1 -- 172.21.15.90:0/1922108411 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f4014ad0 con 0x56354b3329b0 2024-04-25T01:18:09.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.811+0000 7f31145be740 1 -- 172.21.15.90:0/1922108411 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56354b3329b0 msgr2=0x56354b2fbb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:09.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.811+0000 7f31145be740 1 --2- 172.21.15.90:0/1922108411 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56354b3329b0 0x56354b2fbb30 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f30f400d910 tx=0x7f30f4010000 comp rx=0 tx=0).stop 2024-04-25T01:18:09.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.811+0000 7f31145be740 1 -- 172.21.15.90:0/1922108411 shutdown_connections 2024-04-25T01:18:09.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.811+0000 7f31145be740 1 --2- 172.21.15.90:0/1922108411 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56354b34ddb0 0x56354b3067c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.811+0000 7f31145be740 1 --2- 172.21.15.90:0/1922108411 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56354b3329b0 0x56354b2fbb30 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.811+0000 7f31145be740 1 --2- 172.21.15.90:0/1922108411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56354b330510 0x56354b34a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.811+0000 7f31145be740 1 -- 172.21.15.90:0/1922108411 >> 172.21.15.90:0/1922108411 conn(0x56354b3061f0 msgr2=0x56354b3341a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:09.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.811+0000 7f31145be740 1 -- 172.21.15.90:0/1922108411 shutdown_connections 2024-04-25T01:18:09.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.811+0000 7f31145be740 1 -- 172.21.15.90:0/1922108411 wait complete. 2024-04-25T01:18:09.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.812+0000 7f31145be740 1 -- start start 2024-04-25T01:18:09.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.812+0000 7f31145be740 1 Processor -- start 2024-04-25T01:18:09.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.813+0000 7f31145be740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56354b330510 0x56354b3ab8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:09.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.813+0000 7f31145be740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56354b34ddb0 0x56354b32a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:09.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.813+0000 7f31145be740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56354b333e90 0x56354b32a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:09.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.813+0000 7f31145be740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56354b35c2c0 con 0x56354b34ddb0 2024-04-25T01:18:09.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.813+0000 7f31145be740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56354b3336e0 con 0x56354b333e90 2024-04-25T01:18:09.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.813+0000 7f31145be740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x56354b35bf00 con 0x56354b330510 2024-04-25T01:18:09.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.813+0000 7f31145be740 10 client.0 _ll_register_callbacks cb 0x7f3104035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:18:09.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.813+0000 7f310a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56354b34ddb0 0x56354b32a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:09.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.814+0000 7f310a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56354b34ddb0 0x56354b32a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33270/0 (socket says 172.21.15.90:33270) 2024-04-25T01:18:09.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.814+0000 7f310a000640 1 -- 172.21.15.90:0/1832573471 learned_addr learned my addr 172.21.15.90:0/1832573471 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:09.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.814+0000 7f310a000640 1 -- 172.21.15.90:0/1832573471 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56354b330510 msgr2=0x56354b3ab8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:09.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.814+0000 7f310a000640 1 --2- 172.21.15.90:0/1832573471 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56354b330510 0x56354b3ab8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.814+0000 7f310a000640 1 -- 172.21.15.90:0/1832573471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56354b333e90 msgr2=0x56354b32a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:09.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.814+0000 7f310a000640 1 --2- 172.21.15.90:0/1832573471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56354b333e90 0x56354b32a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.814+0000 7f310a000640 1 -- 172.21.15.90:0/1832573471 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56354b3af020 con 0x56354b34ddb0 2024-04-25T01:18:09.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.814+0000 7f310a000640 1 --2- 172.21.15.90:0/1832573471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56354b34ddb0 0x56354b32a200 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f30f400d8b0 tx=0x7f30f4006e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f9b14b67e6fcf83 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:09.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.814+0000 7f30ef400640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T01:18:09.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.814+0000 7f30ef400640 1 -- 172.21.15.90:0/1832573471 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f4005d60 con 0x56354b34ddb0 2024-04-25T01:18:09.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.814+0000 7f30ef400640 1 -- 172.21.15.90:0/1832573471 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(41..41 src has 1..41) ==== 6159+0+0 (secure 0 0 0) 0x7f30f400f040 con 0x56354b34ddb0 2024-04-25T01:18:09.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.814+0000 7f31145be740 1 -- 172.21.15.90:0/1832573471 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30f4025070 con 0x56354b34ddb0 2024-04-25T01:18:09.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.815+0000 7f30ef400640 1 -- 172.21.15.90:0/1832573471 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7f30f4037050 con 0x56354b34ddb0 2024-04-25T01:18:09.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.815+0000 7f31145be740 10 client.5514 fetch_fsmap learned FSMap version 18 2024-04-25T01:18:09.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.815+0000 7f31145be740 1 -- 172.21.15.90:0/1832573471 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) -- 0x56354b306e80 con 0x56354b34ddb0 2024-04-25T01:18:09.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.815+0000 7f30ef400640 1 -- 172.21.15.90:0/1832573471 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30f4025070 con 0x56354b34ddb0 2024-04-25T01:18:09.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.815+0000 7f30ef400640 1 -- 172.21.15.90:0/1832573471 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 18) ==== 3494+0+0 (secure 0 0 0) 0x7f30f403a020 con 0x56354b34ddb0 2024-04-25T01:18:09.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.815+0000 7f31145be740 10 client.5514 fetch_fsmap finished waiting for FSMap version 18 2024-04-25T01:18:09.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.815+0000 7f31145be740 10 client.5514 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T01:18:09.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.815+0000 7f31145be740 20 client.5514 populate_metadata read hostname 'smithi090' 2024-04-25T01:18:09.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.815+0000 7f31145be740 1 --2- 172.21.15.90:0/1832573471 >> [v2:172.21.15.90:6836/431390375,v1:172.21.15.90:6837/431390375] conn(0x56354b32e4f0 0x56354b32e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:09.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.815+0000 7f31145be740 4 client.5514 mds_command: new command op to 5334 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:18:09.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.815+0000 7f31145be740 1 -- 172.21.15.90:0/1832573471 --> [v2:172.21.15.90:6836/431390375,v1:172.21.15.90:6837/431390375] -- command(tid 0: {"prefix": "status"}) -- 0x56354b270680 con 0x56354b32e4f0 2024-04-25T01:18:09.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.815+0000 7f310aa00640 1 --2- 172.21.15.90:0/1832573471 >> [v2:172.21.15.90:6836/431390375,v1:172.21.15.90:6837/431390375] conn(0x56354b32e4f0 0x56354b32e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:09.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.816+0000 7f310aa00640 1 --2- 172.21.15.90:0/1832573471 >> [v2:172.21.15.90:6836/431390375,v1:172.21.15.90:6837/431390375] conn(0x56354b32e4f0 0x56354b32e8d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f6f70881cccc2db server_cookie=c4fbc0fe448b1df7 in_seq=0 out_seq=0 2024-04-25T01:18:09.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.816+0000 7f30ef400640 10 client.5514 ms_handle_connect on v2:172.21.15.90:6836/431390375 2024-04-25T01:18:09.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.816+0000 7f30ef400640 1 -- 172.21.15.90:0/1832573471 <== mds.0 v2:172.21.15.90:6836/431390375 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x56354b270680 con 0x56354b32e4f0 2024-04-25T01:18:09.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.816+0000 7f30ef400640 10 client.5514 handle_command_reply: tid=0 2024-04-25T01:18:09.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.816+0000 7f31145be740 1 client.5514 shutdown 2024-04-25T01:18:09.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.816+0000 7f31145be740 1 -- 172.21.15.90:0/1832573471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56354b34ddb0 msgr2=0x56354b32a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:09.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.816+0000 7f31145be740 1 --2- 172.21.15.90:0/1832573471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56354b34ddb0 0x56354b32a200 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f30f400d8b0 tx=0x7f30f4006e10 comp rx=0 tx=0).stop 2024-04-25T01:18:09.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.816+0000 7f31145be740 1 -- 172.21.15.90:0/1832573471 shutdown_connections 2024-04-25T01:18:09.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.816+0000 7f31145be740 1 --2- 172.21.15.90:0/1832573471 >> [v2:172.21.15.90:6836/431390375,v1:172.21.15.90:6837/431390375] conn(0x56354b32e4f0 0x56354b32e8d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.816+0000 7f31145be740 1 --2- 172.21.15.90:0/1832573471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56354b333e90 0x56354b32a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.816+0000 7f31145be740 1 --2- 172.21.15.90:0/1832573471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56354b34ddb0 0x56354b32a200 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.816+0000 7f31145be740 1 --2- 172.21.15.90:0/1832573471 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56354b330510 0x56354b3ab8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.816+0000 7f30ef400640 0 client.5514 ms_handle_reset on v2:172.21.15.90:6836/431390375 2024-04-25T01:18:09.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.816+0000 7f31145be740 1 -- 172.21.15.90:0/1832573471 >> 172.21.15.90:0/1832573471 conn(0x56354b3061f0 msgr2=0x56354b35a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:09.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.816+0000 7f31145be740 1 -- 172.21.15.90:0/1832573471 shutdown_connections 2024-04-25T01:18:09.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.816+0000 7f31145be740 1 -- 172.21.15.90:0/1832573471 wait complete. 2024-04-25T01:18:09.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.817+0000 7f31145be740 20 client.5514 trim_cache size 0 max 16384 2024-04-25T01:18:09.804 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T01:18:09.805 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21", 2024-04-25T01:18:09.805 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 0, 2024-04-25T01:18:09.805 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 5334, 2024-04-25T01:18:09.805 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-04-25T01:18:09.805 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-04-25T01:18:09.805 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-04-25T01:18:09.805 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 6.5711055050000002, 2024-04-25T01:18:09.805 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 18, 2024-04-25T01:18:09.805 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 41, 2024-04-25T01:18:09.805 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 40, 2024-04-25T01:18:09.805 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 15.40094422 2024-04-25T01:18:09.805 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T01:18:09.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.817+0000 7f3111600640 1 -- 172.21.15.90:0/2126467741 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f30e806be30 msgr2=0x7f30e806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:09.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.817+0000 7f3111600640 1 --2- 172.21.15.90:0/2126467741 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f30e806be30 0x7f30e806e2f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f310c15cae0 tx=0x7f30f4008040 comp rx=0 tx=0).stop 2024-04-25T01:18:09.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.817+0000 7f3111600640 1 -- 172.21.15.90:0/2126467741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f310c0b4db0 msgr2=0x7f310c157e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:09.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.817+0000 7f3111600640 1 --2- 172.21.15.90:0/2126467741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f310c0b4db0 0x7f310c157e80 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f30fc00c8c0 tx=0x7f30fc00cd80 comp rx=0 tx=0).stop 2024-04-25T01:18:09.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.817+0000 7f3111600640 1 -- 172.21.15.90:0/2126467741 shutdown_connections 2024-04-25T01:18:09.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.817+0000 7f3111600640 1 --2- 172.21.15.90:0/2126467741 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f30e806be30 0x7f30e806e2f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.817+0000 7f3111600640 1 --2- 172.21.15.90:0/2126467741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f310c15e1f0 0x7f310c159e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.817+0000 7f3111600640 1 --2- 172.21.15.90:0/2126467741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f310c15de10 0x7f310c157740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.817+0000 7f3111600640 1 --2- 172.21.15.90:0/2126467741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f310c0b4db0 0x7f310c157e80 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:09.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.817+0000 7f3111600640 1 -- 172.21.15.90:0/2126467741 >> 172.21.15.90:0/2126467741 conn(0x7f310c0b1820 msgr2=0x7f310c15b400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:09.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.817+0000 7f3111600640 1 -- 172.21.15.90:0/2126467741 shutdown_connections 2024-04-25T01:18:09.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:09.817+0000 7f3111600640 1 -- 172.21.15.90:0/2126467741 wait complete. 2024-04-25T01:18:09.816 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T01:18:10.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.021+0000 7f7fbf400640 1 Processor -- start 2024-04-25T01:18:10.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.021+0000 7f7fbf400640 1 -- start start 2024-04-25T01:18:10.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.023+0000 7f7fbf400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fb815f9b0 0x7f7fb815fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:10.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.023+0000 7f7fbf400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fb80b4240 con 0x7f7fb806d270 2024-04-25T01:18:10.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.023+0000 7f7fbf400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fb80b39a0 con 0x7f7fb8145590 2024-04-25T01:18:10.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.023+0000 7f7fbf400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7fb8160830 con 0x7f7fb815f9b0 2024-04-25T01:18:10.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.023+0000 7f7fbd400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f7fb806d270 0x7f7fb806d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:49426/0 (socket says 172.21.15.90:49426) 2024-04-25T01:18:10.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.023+0000 7f7fbd400640 1 -- 172.21.15.90:0/2641289346 learned_addr learned my addr 172.21.15.90:0/2641289346 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:10.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.024+0000 7f7fb7e00640 1 --2- 172.21.15.90:0/2641289346 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fb815f9b0 0x7f7fb815fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:10.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.024+0000 7f7fb7400640 1 -- 172.21.15.90:0/2641289346 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1017827591 0 0) 0x7f7fb80b4240 con 0x7f7fb806d270 2024-04-25T01:18:10.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.024+0000 7f7fb7400640 1 -- 172.21.15.90:0/2641289346 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7fa0003520 con 0x7f7fb806d270 2024-04-25T01:18:10.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.024+0000 7f7fb7400640 1 -- 172.21.15.90:0/2641289346 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3749715076 0 0) 0x7f7fa0003520 con 0x7f7fb806d270 2024-04-25T01:18:10.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.024+0000 7f7fb7400640 1 -- 172.21.15.90:0/2641289346 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fb815f9b0 msgr2=0x7f7fb815fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:10.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.024+0000 7f7fb7400640 1 --2- 172.21.15.90:0/2641289346 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fb815f9b0 0x7f7fb815fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:10.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.024+0000 7f7fb7400640 1 -- 172.21.15.90:0/2641289346 >> v1:172.21.15.103:6789/0 conn(0x7f7fb8145590 legacy=0x7f7fb80b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:10.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.024+0000 7f7fb7400640 1 -- 172.21.15.90:0/2641289346 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb80b3c80 con 0x7f7fb806d270 2024-04-25T01:18:10.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.024+0000 7f7fb7400640 1 -- 172.21.15.90:0/2641289346 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f7fb00019f0 con 0x7f7fb806d270 2024-04-25T01:18:10.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.024+0000 7f7fb7400640 1 -- 172.21.15.90:0/2641289346 >> v1:172.21.15.90:6789/0 conn(0x7f7fb806d270 legacy=0x7f7fb806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:10.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.024+0000 7f7fb7400640 1 --2- 172.21.15.90:0/2641289346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fa00039e0 0x7f7fa0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:10.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.024+0000 7f7fb7400640 1 --2- 172.21.15.90:0/2641289346 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fa0004800 0x7f7fa0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:10.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.024+0000 7f7fb7400640 1 --2- 172.21.15.90:0/2641289346 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fb815f9b0 0x7f7fa00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:10.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.025+0000 7f7fb7400640 1 -- 172.21.15.90:0/2641289346 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa000ba70 con 0x7f7fa00039e0 2024-04-25T01:18:10.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.025+0000 7f7fb7400640 1 -- 172.21.15.90:0/2641289346 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa000bbf0 con 0x7f7fa0004800 2024-04-25T01:18:10.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.025+0000 7f7fb7400640 1 -- 172.21.15.90:0/2641289346 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7fa000bdf0 con 0x7f7fb815f9b0 2024-04-25T01:18:10.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.025+0000 7f7fbca00640 1 --2- 172.21.15.90:0/2641289346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fa00039e0 0x7f7fa0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:10.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.025+0000 7f7fb7e00640 1 --2- 172.21.15.90:0/2641289346 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fa0004800 0x7f7fa0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:10.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.025+0000 7f7fbd400640 1 --2- 172.21.15.90:0/2641289346 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fb815f9b0 0x7f7fa00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:10.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.025+0000 7f7fbca00640 1 -- 172.21.15.90:0/2641289346 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fb815f9b0 msgr2=0x7f7fa00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:10.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.025+0000 7f7fbca00640 1 --2- 172.21.15.90:0/2641289346 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fb815f9b0 0x7f7fa00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:10.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.025+0000 7f7fbca00640 1 -- 172.21.15.90:0/2641289346 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fa0004800 msgr2=0x7f7fa0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:10.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.025+0000 7f7fbca00640 1 --2- 172.21.15.90:0/2641289346 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fa0004800 0x7f7fa0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:10.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.025+0000 7f7fbca00640 1 -- 172.21.15.90:0/2641289346 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fa0003520 con 0x7f7fa00039e0 2024-04-25T01:18:10.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.025+0000 7f7fb7e00640 1 --2- 172.21.15.90:0/2641289346 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fa0004800 0x7f7fa0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:10.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.025+0000 7f7fbd400640 1 --2- 172.21.15.90:0/2641289346 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fb815f9b0 0x7f7fa00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:10.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.026+0000 7f7fbca00640 1 --2- 172.21.15.90:0/2641289346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fa00039e0 0x7f7fa0003e00 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f7fac001720 tx=0x7f7fa0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8838d982f7bf4024 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:10.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.027+0000 7f7fb7400640 1 -- 172.21.15.90:0/2641289346 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac001370 con 0x7f7fa00039e0 2024-04-25T01:18:10.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.027+0000 7f7fb7400640 1 -- 172.21.15.90:0/2641289346 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fac0052d0 con 0x7f7fa00039e0 2024-04-25T01:18:10.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.027+0000 7f7fb7400640 1 -- 172.21.15.90:0/2641289346 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac02ab90 con 0x7f7fa00039e0 2024-04-25T01:18:10.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.027+0000 7f7fbf400640 1 -- 172.21.15.90:0/2641289346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fa00039e0 msgr2=0x7f7fa0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:10.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.027+0000 7f7fbf400640 1 --2- 172.21.15.90:0/2641289346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fa00039e0 0x7f7fa0003e00 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f7fac001720 tx=0x7f7fa0005830 comp rx=0 tx=0).stop 2024-04-25T01:18:10.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.027+0000 7f7fbd400640 1 -- 172.21.15.90:0/2641289346 reap_dead start 2024-04-25T01:18:10.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.027+0000 7f7fbf400640 1 -- 172.21.15.90:0/2641289346 shutdown_connections 2024-04-25T01:18:10.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.027+0000 7f7fbf400640 1 -- 172.21.15.90:0/2641289346 >> 172.21.15.90:0/2641289346 conn(0x7f7fb80b1820 msgr2=0x7f7fb80b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:10.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.027+0000 7f7fbf400640 1 -- 172.21.15.90:0/2641289346 shutdown_connections 2024-04-25T01:18:10.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.027+0000 7f7fbf400640 1 -- 172.21.15.90:0/2641289346 wait complete. 2024-04-25T01:18:10.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.029+0000 7f7fbf400640 1 Processor -- start 2024-04-25T01:18:10.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.029+0000 7f7fbf400640 1 -- start start 2024-04-25T01:18:10.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.030+0000 7f7fbf400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fa00039e0 0x7f7fb815ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:10.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.030+0000 7f7fbf400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fb813d4a0 0x7f7fb81432f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:10.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.030+0000 7f7fbf400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fb8143830 0x7f7fb815c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:10.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.030+0000 7f7fbf400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb8145970 con 0x7f7fb813d4a0 2024-04-25T01:18:10.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.030+0000 7f7fbf400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb806cde0 con 0x7f7fb8143830 2024-04-25T01:18:10.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.030+0000 7f7fbf400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7fb80b3530 con 0x7f7fa00039e0 2024-04-25T01:18:10.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.030+0000 7f7fb7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fb813d4a0 0x7f7fb81432f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:10.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.030+0000 7f7fb7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fb813d4a0 0x7f7fb81432f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33288/0 (socket says 172.21.15.90:33288) 2024-04-25T01:18:10.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.030+0000 7f7fb7e00640 1 -- 172.21.15.90:0/2207232305 learned_addr learned my addr 172.21.15.90:0/2207232305 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:10.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.031+0000 7f7fb7e00640 1 -- 172.21.15.90:0/2207232305 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fa00039e0 msgr2=0x7f7fb815ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:10.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.031+0000 7f7fb7e00640 1 --2- 172.21.15.90:0/2207232305 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fa00039e0 0x7f7fb815ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:10.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.031+0000 7f7fb7e00640 1 -- 172.21.15.90:0/2207232305 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fb8143830 msgr2=0x7f7fb815c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:10.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.031+0000 7f7fb7e00640 1 --2- 172.21.15.90:0/2207232305 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fb8143830 0x7f7fb815c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:10.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.031+0000 7f7fb7e00640 1 -- 172.21.15.90:0/2207232305 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb815f380 con 0x7f7fb813d4a0 2024-04-25T01:18:10.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.031+0000 7f7fb7e00640 1 --2- 172.21.15.90:0/2207232305 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fb813d4a0 0x7f7fb81432f0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f7fa40108e0 tx=0x7f7fa4010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:10.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.031+0000 7f7fb4c00640 1 -- 172.21.15.90:0/2207232305 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa400f040 con 0x7f7fb813d4a0 2024-04-25T01:18:10.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.031+0000 7f7fb4c00640 1 -- 172.21.15.90:0/2207232305 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fa400ec20 con 0x7f7fb813d4a0 2024-04-25T01:18:10.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.031+0000 7f7fb4c00640 1 -- 172.21.15.90:0/2207232305 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa401c400 con 0x7f7fb813d4a0 2024-04-25T01:18:10.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.032+0000 7f7fbf400640 1 -- 172.21.15.90:0/2207232305 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fb81e5490 con 0x7f7fb813d4a0 2024-04-25T01:18:10.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.032+0000 7f7fbf400640 1 -- 172.21.15.90:0/2207232305 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fb81e5a10 con 0x7f7fb813d4a0 2024-04-25T01:18:10.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.033+0000 7f7fbf400640 1 -- 172.21.15.90:0/2207232305 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fb8068fb0 con 0x7f7fb813d4a0 2024-04-25T01:18:10.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.034+0000 7f7fb4c00640 1 -- 172.21.15.90:0/2207232305 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7fa401c5a0 con 0x7f7fb813d4a0 2024-04-25T01:18:10.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.035+0000 7f7fb4c00640 1 --2- 172.21.15.90:0/2207232305 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7f9806be30 0x7f7f9806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:10.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.035+0000 7f7fb4c00640 1 -- 172.21.15.90:0/2207232305 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6159+0+0 (secure 0 0 0) 0x7f7fa408b7b0 con 0x7f7fb813d4a0 2024-04-25T01:18:10.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.035+0000 7f7fbca00640 1 --2- 172.21.15.90:0/2207232305 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7f9806be30 0x7f7f9806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:10.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.035+0000 7f7fbca00640 1 --2- 172.21.15.90:0/2207232305 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7f9806be30 0x7f7f9806e2f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7fac032000 tx=0x7f7fac02c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:10.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.039+0000 7f7fb4c00640 1 -- 172.21.15.90:0/2207232305 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7fb815f380 con 0x7f7fb813d4a0 2024-04-25T01:18:10.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.259+0000 7f7fbf400640 1 -- 172.21.15.90:0/2207232305 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f7fb806ea50 con 0x7f7fb813d4a0 2024-04-25T01:18:10.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.260+0000 7f7fb4c00640 1 -- 172.21.15.90:0/2207232305 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) ==== 53+0+73 (secure 0 0 0) 0x7f7fa4018030 con 0x7f7fb813d4a0 2024-04-25T01:18:10.243 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T01:18:10.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.262+0000 7f7fbf400640 1 -- 172.21.15.90:0/2207232305 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7f9806be30 msgr2=0x7f7f9806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:10.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.262+0000 7f7fbf400640 1 --2- 172.21.15.90:0/2207232305 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7f9806be30 0x7f7f9806e2f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7fac032000 tx=0x7f7fac02c000 comp rx=0 tx=0).stop 2024-04-25T01:18:10.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.262+0000 7f7fbf400640 1 -- 172.21.15.90:0/2207232305 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fb813d4a0 msgr2=0x7f7fb81432f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:10.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.262+0000 7f7fbf400640 1 --2- 172.21.15.90:0/2207232305 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fb813d4a0 0x7f7fb81432f0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f7fa40108e0 tx=0x7f7fa4010da0 comp rx=0 tx=0).stop 2024-04-25T01:18:10.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.263+0000 7f7fbf400640 1 -- 172.21.15.90:0/2207232305 shutdown_connections 2024-04-25T01:18:10.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.263+0000 7f7fbf400640 1 --2- 172.21.15.90:0/2207232305 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7f9806be30 0x7f7f9806e2f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:10.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.263+0000 7f7fbf400640 1 --2- 172.21.15.90:0/2207232305 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7fb8143830 0x7f7fb815c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:10.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.263+0000 7f7fbf400640 1 --2- 172.21.15.90:0/2207232305 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fb813d4a0 0x7f7fb81432f0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:10.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.263+0000 7f7fbf400640 1 --2- 172.21.15.90:0/2207232305 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7fa00039e0 0x7f7fb815ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:10.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.263+0000 7f7fbf400640 1 -- 172.21.15.90:0/2207232305 >> 172.21.15.90:0/2207232305 conn(0x7f7fb80b1820 msgr2=0x7f7fb815fe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:10.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.263+0000 7f7fbf400640 1 -- 172.21.15.90:0/2207232305 shutdown_connections 2024-04-25T01:18:10.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10.263+0000 7f7fbf400640 1 -- 172.21.15.90:0/2207232305 wait complete. 2024-04-25T01:18:10.264 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-25T01:18:10.264 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T01:18:10.264 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2024-04-25T01:18:10.264 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T01:18:10.264 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T01:18:10.264 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-25T01:18:10.264 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-25T01:18:10.264 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T01:18:10.264 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T01:18:10.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:10.264 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2024-04-25T01:18:10.312 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T01:18:10.312 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T01:18:10.312 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T01:18:10.312 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:18:10.312 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-04-25T01:18:10.312 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:18:10.312 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:18:10.312 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:18:10.312 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f0 2024-04-25T01:18:10.312 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:18:10.312 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:18:10.312 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f1 2024-04-25T01:18:10.312 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:18:10.313 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:18:10.313 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T01:18:10.313 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:18:10.313 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-25T01:18:10.313 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 42549sec preferred_lft 42549sec 2024-04-25T01:18:10.313 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::ec4:7aff:fe8f:d767/64 scope link 2024-04-25T01:18:10.313 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:18:10.313 INFO:teuthology.orchestra.run.smithi090.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T01:18:10.313 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 1e:6a:0d:c3:bc:1e brd ff:ff:ff:ff:ff:ff 2024-04-25T01:18:10.313 INFO:teuthology.orchestra.run.smithi090.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T01:18:10.313 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:18:10.313 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::405b:4dff:feb7:7d54/64 scope link 2024-04-25T01:18:10.313 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:18:10.313 INFO:teuthology.orchestra.run.smithi090.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T01:18:10.313 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 1e:6a:0d:c3:bc:1e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T01:18:10.314 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::1c6a:dff:fec3:bc1e/64 scope link 2024-04-25T01:18:10.314 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:18:10.314 INFO:teuthology.orchestra.run.smithi090.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T01:18:10.314 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 9a:38:36:61:4f:7c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T01:18:10.314 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::9838:36ff:fe61:4f7c/64 scope link 2024-04-25T01:18:10.314 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:18:10.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:10.314 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T01:18:10.434 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-04-25T01:18:10.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:10.437 DEBUG:teuthology.orchestra.run.smithi090:> route 2024-04-25T01:18:10.491 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2024-04-25T01:18:10.491 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T01:18:10.491 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T01:18:10.491 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T01:18:10.492 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T01:18:10.492 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:18:10.492 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-04-25T01:18:10.492 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T01:18:10.492 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T01:18:10.492 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T01:18:10.492 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-04-25T01:18:10.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:18:10.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:18:10.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:10.693 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-04-25T01:18:10.742 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T01:18:10.742 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-25T01:18:10.743 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T01:18:10.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:10.743 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.0 up 2024-04-25T01:18:10.824 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:18:10.824 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T01:18:10.875 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T01:18:10.875 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:18:10.875 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T01:18:10.919 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-25T01:18:10.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:10.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-25T01:18:11.089 INFO:teuthology.orchestra.run.smithi090.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-25T01:18:11.089 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-25T01:18:11.089 INFO:teuthology.orchestra.run.smithi090.stdout:invalid new device string format 2024-04-25T01:18:11.110 INFO:teuthology.orchestra.run.smithi090.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-25T01:18:11.110 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-25T01:18:11.110 INFO:teuthology.orchestra.run.smithi090.stdout:invalid new device string format 2024-04-25T01:18:11.110 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: resolved to: "172.21.15.90:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-25T01:18:11.110 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-25T01:18:11.135 INFO:teuthology.orchestra.run.smithi090.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-25T01:18:11.136 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-25T01:18:11.136 INFO:teuthology.orchestra.run.smithi090.stdout:invalid new device string format 2024-04-25T01:18:11.136 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: resolved to: "172.21.15.90:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-25T01:18:11.136 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-25T01:18:11.136 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: resolved to: "172.21.15.90:6789,172.21.15.103:6789,172.21.15.103:6790" 2024-04-25T01:18:11.136 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.90:6789,172.21.15.103:6789,172.21.15.103:6790:/ 2024-04-25T01:18:11.136 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21" will pass to kernel 2024-04-25T01:18:11.139 INFO:teuthology.orchestra.run.smithi090.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-25T01:18:11.139 INFO:teuthology.orchestra.run.smithi090.stdout: You just mounted a file system that supports labels which does not 2024-04-25T01:18:11.139 INFO:teuthology.orchestra.run.smithi090.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T01:18:11.139 INFO:teuthology.orchestra.run.smithi090.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T01:18:11.139 INFO:teuthology.orchestra.run.smithi090.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T01:18:11.144 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T01:18:11.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:11.144 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T01:18:11.216 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:18:11.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:11 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:18:11.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:18:11.355 INFO:teuthology.orchestra.run.smithi090.stdout:client5523 2024-04-25T01:18:11.355 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:18:11.355 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:18:11.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:11.401 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir) 2024-04-25T01:18:11.504 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:18:11.504 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:18:11.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:11.550 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=subdir/sixmegs bs=1M conv=fdatasync count=6 seek=0) 2024-04-25T01:18:11.797 INFO:teuthology.orchestra.run.smithi090.stderr:6+0 records in 2024-04-25T01:18:11.797 INFO:teuthology.orchestra.run.smithi090.stderr:6+0 records out 2024-04-25T01:18:11.797 INFO:teuthology.orchestra.run.smithi090.stderr:6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.154541 s, 40.7 MB/s 2024-04-25T01:18:11.799 DEBUG:teuthology.orchestra.run.smithi090:> stdin-killer -- python3 -c ' 2024-04-25T01:18:11.799 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-04-25T01:18:11.799 DEBUG:teuthology.orchestra.run.smithi090:> import stat 2024-04-25T01:18:11.799 DEBUG:teuthology.orchestra.run.smithi090:> import json 2024-04-25T01:18:11.799 DEBUG:teuthology.orchestra.run.smithi090:> import sys 2024-04-25T01:18:11.799 DEBUG:teuthology.orchestra.run.smithi090:> 2024-04-25T01:18:11.799 DEBUG:teuthology.orchestra.run.smithi090:> try: 2024-04-25T01:18:11.799 DEBUG:teuthology.orchestra.run.smithi090:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/subdir/sixmegs'"'"') 2024-04-25T01:18:11.800 DEBUG:teuthology.orchestra.run.smithi090:> except OSError as e: 2024-04-25T01:18:11.800 DEBUG:teuthology.orchestra.run.smithi090:> sys.exit(e.errno) 2024-04-25T01:18:11.800 DEBUG:teuthology.orchestra.run.smithi090:> 2024-04-25T01:18:11.800 DEBUG:teuthology.orchestra.run.smithi090:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-04-25T01:18:11.800 DEBUG:teuthology.orchestra.run.smithi090:> print(json.dumps( 2024-04-25T01:18:11.800 DEBUG:teuthology.orchestra.run.smithi090:> dict([(a, getattr(s, a)) for a in attrs]), 2024-04-25T01:18:11.800 DEBUG:teuthology.orchestra.run.smithi090:> indent=2)) 2024-04-25T01:18:11.800 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-04-25T01:18:11.930 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T01:18:11.930 INFO:teuthology.orchestra.run.smithi090.stdout: "st_mode": 33188, 2024-04-25T01:18:11.930 INFO:teuthology.orchestra.run.smithi090.stdout: "st_ino": 1099511627777, 2024-04-25T01:18:11.930 INFO:teuthology.orchestra.run.smithi090.stdout: "st_dev": 42, 2024-04-25T01:18:11.931 INFO:teuthology.orchestra.run.smithi090.stdout: "st_nlink": 1, 2024-04-25T01:18:11.931 INFO:teuthology.orchestra.run.smithi090.stdout: "st_uid": 1000, 2024-04-25T01:18:11.931 INFO:teuthology.orchestra.run.smithi090.stdout: "st_gid": 1316, 2024-04-25T01:18:11.931 INFO:teuthology.orchestra.run.smithi090.stdout: "st_size": 6291456, 2024-04-25T01:18:11.931 INFO:teuthology.orchestra.run.smithi090.stdout: "st_atime": 1714007891.657211, 2024-04-25T01:18:11.931 INFO:teuthology.orchestra.run.smithi090.stdout: "st_mtime": 1714007891.7082105, 2024-04-25T01:18:11.931 INFO:teuthology.orchestra.run.smithi090.stdout: "st_ctime": 1714007891.7082105 2024-04-25T01:18:11.931 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T01:18:11.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:18:11.943 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:18:11.943 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:18:12.030 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:18:12.030 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:18:12.112 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:18:12.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.355+0000 7f465de00640 1 Processor -- start 2024-04-25T01:18:12.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.356+0000 7f465de00640 1 -- start start 2024-04-25T01:18:12.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.357+0000 7f465de00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f465815f9b0 0x7f465815fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:12.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.357+0000 7f465de00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46580b4240 con 0x7f4658145590 2024-04-25T01:18:12.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.357+0000 7f465de00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46580b39a0 con 0x7f465806d270 2024-04-25T01:18:12.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.357+0000 7f465de00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4658160830 con 0x7f465815f9b0 2024-04-25T01:18:12.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.358+0000 7f4656a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4658145590 0x7f46580b4a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:49428/0 (socket says 172.21.15.90:49428) 2024-04-25T01:18:12.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.358+0000 7f4656a00640 1 -- 172.21.15.90:0/1476669333 learned_addr learned my addr 172.21.15.90:0/1476669333 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:12.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.358+0000 7f4656000640 1 -- 172.21.15.90:0/1476669333 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2613802089 0 0) 0x7f46580b4240 con 0x7f4658145590 2024-04-25T01:18:12.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.358+0000 7f4656000640 1 -- 172.21.15.90:0/1476669333 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f463c003520 con 0x7f4658145590 2024-04-25T01:18:12.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.358+0000 7f464fe00640 1 --2- 172.21.15.90:0/1476669333 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f465815f9b0 0x7f465815fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:12.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.358+0000 7f4656000640 1 -- 172.21.15.90:0/1476669333 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2464929152 0 0) 0x7f463c003520 con 0x7f4658145590 2024-04-25T01:18:12.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.358+0000 7f4656000640 1 -- 172.21.15.90:0/1476669333 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f465815f9b0 msgr2=0x7f465815fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:12.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.358+0000 7f4656000640 1 --2- 172.21.15.90:0/1476669333 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f465815f9b0 0x7f465815fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:12.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.358+0000 7f4656000640 1 -- 172.21.15.90:0/1476669333 >> v1:172.21.15.103:6789/0 conn(0x7f465806d270 legacy=0x7f465806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:12.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.358+0000 7f4656000640 1 -- 172.21.15.90:0/1476669333 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46580b3c80 con 0x7f4658145590 2024-04-25T01:18:12.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.358+0000 7f4656000640 1 -- 172.21.15.90:0/1476669333 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f4648001670 con 0x7f4658145590 2024-04-25T01:18:12.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.359+0000 7f4656000640 1 -- 172.21.15.90:0/1476669333 >> v1:172.21.15.90:6789/0 conn(0x7f4658145590 legacy=0x7f46580b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:12.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.359+0000 7f4656000640 1 --2- 172.21.15.90:0/1476669333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f463c0039e0 0x7f463c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:12.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.359+0000 7f4656000640 1 --2- 172.21.15.90:0/1476669333 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f465815f9b0 0x7f463c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:12.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.359+0000 7f4656000640 1 --2- 172.21.15.90:0/1476669333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f463c004d40 0x7f463c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:12.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.359+0000 7f4656000640 1 -- 172.21.15.90:0/1476669333 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f463c00bb50 con 0x7f463c004d40 2024-04-25T01:18:12.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.359+0000 7f4656000640 1 -- 172.21.15.90:0/1476669333 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f463c00bcd0 con 0x7f463c0039e0 2024-04-25T01:18:12.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.359+0000 7f4656000640 1 -- 172.21.15.90:0/1476669333 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f463c00bea0 con 0x7f465815f9b0 2024-04-25T01:18:12.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.359+0000 7f4657400640 1 --2- 172.21.15.90:0/1476669333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f463c004d40 0x7f463c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:12.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.359+0000 7f4656a00640 1 --2- 172.21.15.90:0/1476669333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f463c0039e0 0x7f463c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:12.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.359+0000 7f464fe00640 1 --2- 172.21.15.90:0/1476669333 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f465815f9b0 0x7f463c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:12.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.359+0000 7f4657400640 1 -- 172.21.15.90:0/1476669333 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f465815f9b0 msgr2=0x7f463c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:12.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.359+0000 7f4657400640 1 --2- 172.21.15.90:0/1476669333 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f465815f9b0 0x7f463c004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:12.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.359+0000 7f4657400640 1 -- 172.21.15.90:0/1476669333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f463c0039e0 msgr2=0x7f463c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:12.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.359+0000 7f4657400640 1 --2- 172.21.15.90:0/1476669333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f463c0039e0 0x7f463c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:12.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.359+0000 7f4657400640 1 -- 172.21.15.90:0/1476669333 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f463c003520 con 0x7f463c004d40 2024-04-25T01:18:12.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.359+0000 7f4656a00640 1 --2- 172.21.15.90:0/1476669333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f463c0039e0 0x7f463c003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:12.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.360+0000 7f4657400640 1 --2- 172.21.15.90:0/1476669333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f463c004d40 0x7f463c0051c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f4650001e80 tx=0x7f463c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=369fead21f641208 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:12.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.361+0000 7f4656000640 1 -- 172.21.15.90:0/1476669333 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4650001a70 con 0x7f463c004d40 2024-04-25T01:18:12.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.361+0000 7f4656000640 1 -- 172.21.15.90:0/1476669333 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f465002ae50 con 0x7f463c004d40 2024-04-25T01:18:12.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.361+0000 7f4656000640 1 -- 172.21.15.90:0/1476669333 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4650003bb0 con 0x7f463c004d40 2024-04-25T01:18:12.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.361+0000 7f465de00640 1 -- 172.21.15.90:0/1476669333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f463c004d40 msgr2=0x7f463c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:12.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.361+0000 7f465de00640 1 --2- 172.21.15.90:0/1476669333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f463c004d40 0x7f463c0051c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f4650001e80 tx=0x7f463c005830 comp rx=0 tx=0).stop 2024-04-25T01:18:12.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.361+0000 7f4657400640 1 -- 172.21.15.90:0/1476669333 reap_dead start 2024-04-25T01:18:12.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.361+0000 7f465de00640 1 -- 172.21.15.90:0/1476669333 shutdown_connections 2024-04-25T01:18:12.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.361+0000 7f465de00640 1 -- 172.21.15.90:0/1476669333 >> 172.21.15.90:0/1476669333 conn(0x7f46580b1820 msgr2=0x7f46580b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:12.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.361+0000 7f465de00640 1 -- 172.21.15.90:0/1476669333 shutdown_connections 2024-04-25T01:18:12.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.361+0000 7f465de00640 1 -- 172.21.15.90:0/1476669333 wait complete. 2024-04-25T01:18:12.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.363+0000 7f465de00640 1 Processor -- start 2024-04-25T01:18:12.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.363+0000 7f465de00640 1 -- start start 2024-04-25T01:18:12.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.364+0000 7f465de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f463c004d40 0x7f465815b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:12.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.364+0000 7f465de00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f465815d420 0x7f465815d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:12.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.364+0000 7f465de00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46581f1c20 0x7f46581f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:12.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.364+0000 7f465de00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4658145970 con 0x7f463c004d40 2024-04-25T01:18:12.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.364+0000 7f465de00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f465806cde0 con 0x7f46581f1c20 2024-04-25T01:18:12.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.364+0000 7f465de00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f46580b3530 con 0x7f465815d420 2024-04-25T01:18:12.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.364+0000 7f4656a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f463c004d40 0x7f465815b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:12.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.364+0000 7f4656a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f463c004d40 0x7f465815b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33302/0 (socket says 172.21.15.90:33302) 2024-04-25T01:18:12.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.364+0000 7f4656a00640 1 -- 172.21.15.90:0/3919869058 learned_addr learned my addr 172.21.15.90:0/3919869058 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:12.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.365+0000 7f4657400640 1 --2- 172.21.15.90:0/3919869058 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46581f1c20 0x7f46581f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:12.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.365+0000 7f464fe00640 1 --2- 172.21.15.90:0/3919869058 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f465815d420 0x7f465815d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:12.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.365+0000 7f4656a00640 1 -- 172.21.15.90:0/3919869058 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f465815d420 msgr2=0x7f465815d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:12.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.365+0000 7f4656a00640 1 --2- 172.21.15.90:0/3919869058 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f465815d420 0x7f465815d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:12.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.365+0000 7f4656a00640 1 -- 172.21.15.90:0/3919869058 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46581f1c20 msgr2=0x7f46581f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:12.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.365+0000 7f4656a00640 1 --2- 172.21.15.90:0/3919869058 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46581f1c20 0x7f46581f20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:12.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.365+0000 7f4656a00640 1 -- 172.21.15.90:0/3919869058 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f465815bb70 con 0x7f463c004d40 2024-04-25T01:18:12.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.365+0000 7f4656a00640 1 --2- 172.21.15.90:0/3919869058 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f463c004d40 0x7f465815b630 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f464800d8c0 tx=0x7f464800dd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:12.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.365+0000 7f464f400640 1 -- 172.21.15.90:0/3919869058 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4648019070 con 0x7f463c004d40 2024-04-25T01:18:12.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.365+0000 7f464f400640 1 -- 172.21.15.90:0/3919869058 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46480074e0 con 0x7f463c004d40 2024-04-25T01:18:12.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.365+0000 7f464f400640 1 -- 172.21.15.90:0/3919869058 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4648004a20 con 0x7f463c004d40 2024-04-25T01:18:12.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.366+0000 7f465de00640 1 -- 172.21.15.90:0/3919869058 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f465815e150 con 0x7f463c004d40 2024-04-25T01:18:12.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.366+0000 7f465de00640 1 -- 172.21.15.90:0/3919869058 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f465815e710 con 0x7f463c004d40 2024-04-25T01:18:12.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.367+0000 7f465de00640 1 -- 172.21.15.90:0/3919869058 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4658068fb0 con 0x7f463c004d40 2024-04-25T01:18:12.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.368+0000 7f464f400640 1 -- 172.21.15.90:0/3919869058 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4648004bc0 con 0x7f463c004d40 2024-04-25T01:18:12.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.368+0000 7f464f400640 1 --2- 172.21.15.90:0/3919869058 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f463406be30 0x7f463406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:12.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.369+0000 7f464f400640 1 -- 172.21.15.90:0/3919869058 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6159+0+0 (secure 0 0 0) 0x7f464808ab90 con 0x7f463c004d40 2024-04-25T01:18:12.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.369+0000 7f464fe00640 1 --2- 172.21.15.90:0/3919869058 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f463406be30 0x7f463406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:12.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.370+0000 7f464fe00640 1 --2- 172.21.15.90:0/3919869058 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f463406be30 0x7f463406e2f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f46580ab9f0 tx=0x7f4640008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:12.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.373+0000 7f464f400640 1 -- 172.21.15.90:0/3919869058 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f465815bb70 con 0x7f463c004d40 2024-04-25T01:18:12.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.563+0000 7f465de00640 1 -- 172.21.15.90:0/3919869058 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f465806ea50 con 0x7f463c004d40 2024-04-25T01:18:12.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.566+0000 7f464f400640 1 -- 172.21.15.90:0/3919869058 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+16942 (secure 0 0 0) 0x7f464805f020 con 0x7f463c004d40 2024-04-25T01:18:12.550 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:12.550 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":42,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:18:10.289978+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:18:00.504992+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T01:18:01.714313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.16","mappings":[{"from":4,"to":2}]},{"pgid":"6.20","mappings":[{"from":4,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:18:12.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.569+0000 7f465de00640 1 -- 172.21.15.90:0/3919869058 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f463406be30 msgr2=0x7f463406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:12.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.569+0000 7f465de00640 1 --2- 172.21.15.90:0/3919869058 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f463406be30 0x7f463406e2f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f46580ab9f0 tx=0x7f4640008040 comp rx=0 tx=0).stop 2024-04-25T01:18:12.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.569+0000 7f465de00640 1 -- 172.21.15.90:0/3919869058 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f463c004d40 msgr2=0x7f465815b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:12.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.569+0000 7f465de00640 1 --2- 172.21.15.90:0/3919869058 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f463c004d40 0x7f465815b630 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f464800d8c0 tx=0x7f464800dd80 comp rx=0 tx=0).stop 2024-04-25T01:18:12.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.569+0000 7f465de00640 1 -- 172.21.15.90:0/3919869058 shutdown_connections 2024-04-25T01:18:12.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.569+0000 7f465de00640 1 --2- 172.21.15.90:0/3919869058 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f463406be30 0x7f463406e2f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:12.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.569+0000 7f465de00640 1 --2- 172.21.15.90:0/3919869058 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46581f1c20 0x7f46581f20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:12.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.569+0000 7f465de00640 1 --2- 172.21.15.90:0/3919869058 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f465815d420 0x7f465815d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:12.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.569+0000 7f465de00640 1 --2- 172.21.15.90:0/3919869058 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f463c004d40 0x7f465815b630 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:12.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.569+0000 7f465de00640 1 -- 172.21.15.90:0/3919869058 >> 172.21.15.90:0/3919869058 conn(0x7f46580b1820 msgr2=0x7f4658140130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:12.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.569+0000 7f465de00640 1 -- 172.21.15.90:0/3919869058 shutdown_connections 2024-04-25T01:18:12.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.569+0000 7f465de00640 1 -- 172.21.15.90:0/3919869058 wait complete. 2024-04-25T01:18:12.570 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:16:15.894126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:18:00.504992+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:18:01.714313+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:18:12.571 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:18:12.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.783+0000 7f030c800640 1 Processor -- start 2024-04-25T01:18:12.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.783+0000 7f030c800640 1 -- start start 2024-04-25T01:18:12.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.785+0000 7f030c800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f03041478a0 0x7f0304072b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:12.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.785+0000 7f030c800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f030406b120 0x7f030406b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:12.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.785+0000 7f030c800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f030406bbc0 con 0x7f030406e530 2024-04-25T01:18:12.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.785+0000 7f030c800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f0304071da0 con 0x7f030406b120 2024-04-25T01:18:12.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.785+0000 7f030c800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0304072080 con 0x7f03041478a0 2024-04-25T01:18:12.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.785+0000 7f0309400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f030406b120 0x7f030406b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:12.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.785+0000 7f0309400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f030406b120 0x7f030406b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33318/0 (socket says 172.21.15.90:33318) 2024-04-25T01:18:12.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.785+0000 7f0309400640 1 -- 172.21.15.90:0/717031486 learned_addr learned my addr 172.21.15.90:0/717031486 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:12.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.785+0000 7f0309e00640 1 --2- 172.21.15.90:0/717031486 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f03041478a0 0x7f0304072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:12.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.786+0000 7f0309400640 1 -- 172.21.15.90:0/717031486 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f03041478a0 msgr2=0x7f0304072b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:12.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.786+0000 7f0309400640 1 --2- 172.21.15.90:0/717031486 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f03041478a0 0x7f0304072b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:12.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.786+0000 7f0309400640 1 -- 172.21.15.90:0/717031486 >> v1:172.21.15.103:6789/0 conn(0x7f030406e530 legacy=0x7f03040cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:12.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.786+0000 7f0309400640 1 -- 172.21.15.90:0/717031486 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0304072360 con 0x7f030406b120 2024-04-25T01:18:12.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.787+0000 7f0309400640 1 --2- 172.21.15.90:0/717031486 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f030406b120 0x7f030406b580 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f02f4009870 tx=0x7f02f402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebd8947e809b6c1f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:12.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.787+0000 7f0308a00640 1 -- 172.21.15.90:0/717031486 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f4037070 con 0x7f030406b120 2024-04-25T01:18:12.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.787+0000 7f0308a00640 1 -- 172.21.15.90:0/717031486 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02f4032440 con 0x7f030406b120 2024-04-25T01:18:12.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.787+0000 7f0308a00640 1 -- 172.21.15.90:0/717031486 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f4032810 con 0x7f030406b120 2024-04-25T01:18:12.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.788+0000 7f030c800640 1 -- 172.21.15.90:0/717031486 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f030406b120 msgr2=0x7f030406b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:12.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.788+0000 7f030c800640 1 --2- 172.21.15.90:0/717031486 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f030406b120 0x7f030406b580 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f02f4009870 tx=0x7f02f402a210 comp rx=0 tx=0).stop 2024-04-25T01:18:12.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.788+0000 7f030c800640 1 -- 172.21.15.90:0/717031486 shutdown_connections 2024-04-25T01:18:12.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.788+0000 7f030c800640 1 --2- 172.21.15.90:0/717031486 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f030406b120 0x7f030406b580 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:12.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.788+0000 7f030c800640 1 --2- 172.21.15.90:0/717031486 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f03041478a0 0x7f0304072b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:12.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.788+0000 7f030c800640 1 -- 172.21.15.90:0/717031486 >> 172.21.15.90:0/717031486 conn(0x7f03040d0bf0 msgr2=0x7f03040d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:12.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.788+0000 7f030c800640 1 -- 172.21.15.90:0/717031486 shutdown_connections 2024-04-25T01:18:12.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.788+0000 7f030c800640 1 -- 172.21.15.90:0/717031486 wait complete. 2024-04-25T01:18:12.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.790+0000 7f030c800640 1 Processor -- start 2024-04-25T01:18:12.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.790+0000 7f030c800640 1 -- start start 2024-04-25T01:18:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.791+0000 7f030c800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f030406b120 0x7f03041f4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.791+0000 7f030c800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f030406e530 0x7f03041e82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.792+0000 7f030c800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f03041478a0 0x7f03040cc730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.792+0000 7f030c800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0304162900 con 0x7f030406e530 2024-04-25T01:18:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.792+0000 7f030c800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f030406e990 con 0x7f030406b120 2024-04-25T01:18:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.792+0000 7f030c800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f03041e7f70 con 0x7f03041478a0 2024-04-25T01:18:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.792+0000 7f0309400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f030406e530 0x7f03041e82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.792+0000 7f0309400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f030406e530 0x7f03041e82a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33330/0 (socket says 172.21.15.90:33330) 2024-04-25T01:18:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.792+0000 7f0309400640 1 -- 172.21.15.90:0/1056138507 learned_addr learned my addr 172.21.15.90:0/1056138507 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.792+0000 7f0309e00640 1 --2- 172.21.15.90:0/1056138507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f030406b120 0x7f03041f4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.792+0000 7f0309400640 1 -- 172.21.15.90:0/1056138507 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f03041478a0 msgr2=0x7f03040cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.792+0000 7f030a800640 1 --2- 172.21.15.90:0/1056138507 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f03041478a0 0x7f03040cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.792+0000 7f0309400640 1 --2- 172.21.15.90:0/1056138507 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f03041478a0 0x7f03040cc730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.792+0000 7f0309400640 1 -- 172.21.15.90:0/1056138507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f030406b120 msgr2=0x7f03041f4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.792+0000 7f0309400640 1 --2- 172.21.15.90:0/1056138507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f030406b120 0x7f03041f4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.792+0000 7f0309400640 1 -- 172.21.15.90:0/1056138507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03041e87e0 con 0x7f030406e530 2024-04-25T01:18:12.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.793+0000 7f0309400640 1 --2- 172.21.15.90:0/1056138507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f030406e530 0x7f03041e82a0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f02f4009870 tx=0x7f02f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:12.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.793+0000 7f02fe000640 1 -- 172.21.15.90:0/1056138507 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f4041020 con 0x7f030406e530 2024-04-25T01:18:12.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.793+0000 7f02fe000640 1 -- 172.21.15.90:0/1056138507 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02f40306c0 con 0x7f030406e530 2024-04-25T01:18:12.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.793+0000 7f02fe000640 1 -- 172.21.15.90:0/1056138507 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f4037040 con 0x7f030406e530 2024-04-25T01:18:12.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.793+0000 7f030c800640 1 -- 172.21.15.90:0/1056138507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03041e7c90 con 0x7f030406e530 2024-04-25T01:18:12.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.793+0000 7f030c800640 1 -- 172.21.15.90:0/1056138507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03040b5260 con 0x7f030406e530 2024-04-25T01:18:12.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.794+0000 7f030c800640 1 -- 172.21.15.90:0/1056138507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0304068180 con 0x7f030406e530 2024-04-25T01:18:12.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.795+0000 7f02fe000640 1 -- 172.21.15.90:0/1056138507 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f02f4044020 con 0x7f030406e530 2024-04-25T01:18:12.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.796+0000 7f02fe000640 1 --2- 172.21.15.90:0/1056138507 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f02e806be60 0x7f02e806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:12.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.796+0000 7f02fe000640 1 -- 172.21.15.90:0/1056138507 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6159+0+0 (secure 0 0 0) 0x7f02f40ab7f0 con 0x7f030406e530 2024-04-25T01:18:12.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.797+0000 7f0309e00640 1 --2- 172.21.15.90:0/1056138507 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f02e806be60 0x7f02e806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:12.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.797+0000 7f0309e00640 1 --2- 172.21.15.90:0/1056138507 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f02e806be60 0x7f02e806e320 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f02f8006fd0 tx=0x7f02f8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:12.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:12.805+0000 7f02fe000640 1 -- 172.21.15.90:0/1056138507 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f02f404b030 con 0x7f030406e530 2024-04-25T01:18:12.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.003+0000 7f030c800640 1 -- 172.21.15.90:0/1056138507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f030406be00 con 0x7f030406e530 2024-04-25T01:18:12.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.003+0000 7f02fe000640 1 -- 172.21.15.90:0/1056138507 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) ==== 93+0+10 (secure 0 0 0) 0x7f02f4079fd0 con 0x7f030406e530 2024-04-25T01:18:12.987 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T01:18:12.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.005+0000 7f030c800640 1 -- 172.21.15.90:0/1056138507 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f02e806be60 msgr2=0x7f02e806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:12.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.005+0000 7f030c800640 1 --2- 172.21.15.90:0/1056138507 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f02e806be60 0x7f02e806e320 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f02f8006fd0 tx=0x7f02f8008040 comp rx=0 tx=0).stop 2024-04-25T01:18:12.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.005+0000 7f030c800640 1 -- 172.21.15.90:0/1056138507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f030406e530 msgr2=0x7f03041e82a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:12.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.005+0000 7f030c800640 1 --2- 172.21.15.90:0/1056138507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f030406e530 0x7f03041e82a0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f02f4009870 tx=0x7f02f40046c0 comp rx=0 tx=0).stop 2024-04-25T01:18:12.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.006+0000 7f030c800640 1 -- 172.21.15.90:0/1056138507 shutdown_connections 2024-04-25T01:18:12.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.006+0000 7f030c800640 1 --2- 172.21.15.90:0/1056138507 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f02e806be60 0x7f02e806e320 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:12.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.006+0000 7f030c800640 1 --2- 172.21.15.90:0/1056138507 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f03041478a0 0x7f03040cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:12.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.006+0000 7f030c800640 1 --2- 172.21.15.90:0/1056138507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f030406e530 0x7f03041e82a0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:12.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.006+0000 7f030c800640 1 --2- 172.21.15.90:0/1056138507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f030406b120 0x7f03041f4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:12.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.006+0000 7f030c800640 1 -- 172.21.15.90:0/1056138507 >> 172.21.15.90:0/1056138507 conn(0x7f03040d0bf0 msgr2=0x7f03040b1d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:12.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.006+0000 7f030c800640 1 -- 172.21.15.90:0/1056138507 shutdown_connections 2024-04-25T01:18:12.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.006+0000 7f030c800640 1 -- 172.21.15.90:0/1056138507 wait complete. 2024-04-25T01:18:13.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:18:13.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.221+0000 7f9043200640 1 Processor -- start 2024-04-25T01:18:13.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.221+0000 7f9043200640 1 -- start start 2024-04-25T01:18:13.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.222+0000 7f9043200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9044155130 0x7f904406db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:13.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.222+0000 7f9043200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f904415b950 0x7f90440ae620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:13.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.222+0000 7f9043200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9044155510 con 0x7f904406e0b0 2024-04-25T01:18:13.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.222+0000 7f9043200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f904415bdc0 con 0x7f904415b950 2024-04-25T01:18:13.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.222+0000 7f9043200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f904415bfe0 con 0x7f9044155130 2024-04-25T01:18:13.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.223+0000 7f9042800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f904415b950 0x7f90440ae620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:13.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.223+0000 7f9042800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f904415b950 0x7f90440ae620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33336/0 (socket says 172.21.15.90:33336) 2024-04-25T01:18:13.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.223+0000 7f9042800640 1 -- 172.21.15.90:0/3415345953 learned_addr learned my addr 172.21.15.90:0/3415345953 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:13.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.223+0000 7f9041e00640 1 --2- 172.21.15.90:0/3415345953 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9044155130 0x7f904406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:13.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.223+0000 7f9042800640 1 -- 172.21.15.90:0/3415345953 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9044155130 msgr2=0x7f904406db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:13.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.223+0000 7f9042800640 1 --2- 172.21.15.90:0/3415345953 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9044155130 0x7f904406db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:13.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.223+0000 7f9042800640 1 -- 172.21.15.90:0/3415345953 >> v1:172.21.15.103:6789/0 conn(0x7f904406e0b0 legacy=0x7f904415b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:13.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.223+0000 7f9042800640 1 -- 172.21.15.90:0/3415345953 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f904415d1d0 con 0x7f904415b950 2024-04-25T01:18:13.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.223+0000 7f9041e00640 1 --2- 172.21.15.90:0/3415345953 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9044155130 0x7f904406db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:13.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.224+0000 7f9042800640 1 --2- 172.21.15.90:0/3415345953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f904415b950 0x7f90440ae620 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f903c009e50 tx=0x7f903c02a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ac89c04c5d5c0b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:13.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.225+0000 7f9041400640 1 -- 172.21.15.90:0/3415345953 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f903c036070 con 0x7f904415b950 2024-04-25T01:18:13.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.225+0000 7f9041400640 1 -- 172.21.15.90:0/3415345953 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f903c031600 con 0x7f904415b950 2024-04-25T01:18:13.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.225+0000 7f9041400640 1 -- 172.21.15.90:0/3415345953 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f903c0319d0 con 0x7f904415b950 2024-04-25T01:18:13.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.225+0000 7f9043200640 1 -- 172.21.15.90:0/3415345953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f904415b950 msgr2=0x7f90440ae620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:13.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.225+0000 7f9043200640 1 --2- 172.21.15.90:0/3415345953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f904415b950 0x7f90440ae620 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f903c009e50 tx=0x7f903c02a640 comp rx=0 tx=0).stop 2024-04-25T01:18:13.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.225+0000 7f9043200640 1 -- 172.21.15.90:0/3415345953 shutdown_connections 2024-04-25T01:18:13.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.225+0000 7f9043200640 1 --2- 172.21.15.90:0/3415345953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f904415b950 0x7f90440ae620 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:13.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.225+0000 7f9043200640 1 --2- 172.21.15.90:0/3415345953 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9044155130 0x7f904406db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:13.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.225+0000 7f9043200640 1 -- 172.21.15.90:0/3415345953 >> 172.21.15.90:0/3415345953 conn(0x7f904406b120 msgr2=0x7f904406b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:13.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.226+0000 7f9043200640 1 -- 172.21.15.90:0/3415345953 shutdown_connections 2024-04-25T01:18:13.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.226+0000 7f9043200640 1 -- 172.21.15.90:0/3415345953 wait complete. 2024-04-25T01:18:13.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.227+0000 7f9043200640 1 Processor -- start 2024-04-25T01:18:13.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.227+0000 7f9043200640 1 -- start start 2024-04-25T01:18:13.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.228+0000 7f9043200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f904406e0b0 0x7f90441f5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:13.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.228+0000 7f9043200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9044155130 0x7f90441534b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:13.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.228+0000 7f9043200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f904415b950 0x7f90440acd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:13.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.228+0000 7f9043200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f90440720b0 con 0x7f904415b950 2024-04-25T01:18:13.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.228+0000 7f9043200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9044071dd0 con 0x7f9044155130 2024-04-25T01:18:13.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.228+0000 7f9043200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f90440728b0 con 0x7f904406e0b0 2024-04-25T01:18:13.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.228+0000 7f9042800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f904415b950 0x7f90440acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.228+0000 7f9042800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f904415b950 0x7f90440acd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33352/0 (socket says 172.21.15.90:33352) 2024-04-25T01:18:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.228+0000 7f9042800640 1 -- 172.21.15.90:0/2580516263 learned_addr learned my addr 172.21.15.90:0/2580516263 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.229+0000 7f9042800640 1 -- 172.21.15.90:0/2580516263 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f904406e0b0 msgr2=0x7f90441f5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.229+0000 7f9042800640 1 --2- 172.21.15.90:0/2580516263 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f904406e0b0 0x7f90441f5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.229+0000 7f903be00640 1 --2- 172.21.15.90:0/2580516263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9044155130 0x7f90441534b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.229+0000 7f9042800640 1 -- 172.21.15.90:0/2580516263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9044155130 msgr2=0x7f90441534b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.229+0000 7f9042800640 1 --2- 172.21.15.90:0/2580516263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9044155130 0x7f90441534b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.229+0000 7f9042800640 1 -- 172.21.15.90:0/2580516263 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90441539f0 con 0x7f904415b950 2024-04-25T01:18:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.229+0000 7f9042800640 1 --2- 172.21.15.90:0/2580516263 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f904415b950 0x7f90440acd50 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f903c034040 tx=0x7f903c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:13.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.229+0000 7f903a000640 1 -- 172.21.15.90:0/2580516263 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f903c036050 con 0x7f904415b950 2024-04-25T01:18:13.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.229+0000 7f903a000640 1 -- 172.21.15.90:0/2580516263 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f903c042070 con 0x7f904415b950 2024-04-25T01:18:13.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.229+0000 7f903a000640 1 -- 172.21.15.90:0/2580516263 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f903c009510 con 0x7f904415b950 2024-04-25T01:18:13.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.230+0000 7f9043200640 1 -- 172.21.15.90:0/2580516263 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90440ad290 con 0x7f904415b950 2024-04-25T01:18:13.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.230+0000 7f9043200640 1 -- 172.21.15.90:0/2580516263 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90440aefa0 con 0x7f904415b950 2024-04-25T01:18:13.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.231+0000 7f9043200640 1 -- 172.21.15.90:0/2580516263 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90440aedd0 con 0x7f904415b950 2024-04-25T01:18:13.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.232+0000 7f903a000640 1 -- 172.21.15.90:0/2580516263 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f903c007bf0 con 0x7f904415b950 2024-04-25T01:18:13.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.233+0000 7f903a000640 1 --2- 172.21.15.90:0/2580516263 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f902006be30 0x7f902006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:13.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.233+0000 7f903a000640 1 -- 172.21.15.90:0/2580516263 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6159+0+0 (secure 0 0 0) 0x7f903c0abff0 con 0x7f904415b950 2024-04-25T01:18:13.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.233+0000 7f9041e00640 1 --2- 172.21.15.90:0/2580516263 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f902006be30 0x7f902006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:13.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.234+0000 7f9041e00640 1 --2- 172.21.15.90:0/2580516263 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f902006be30 0x7f902006e2f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9034006fd0 tx=0x7f9034008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:13.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.238+0000 7f903a000640 1 -- 172.21.15.90:0/2580516263 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f90441539f0 con 0x7f904415b950 2024-04-25T01:18:13.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.419+0000 7f9043200640 1 -- 172.21.15.90:0/2580516263 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9044073cd0 con 0x7f904415b950 2024-04-25T01:18:13.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.420+0000 7f903a000640 1 -- 172.21.15.90:0/2580516263 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) ==== 104+0+11 (secure 0 0 0) 0x7f903c02ae20 con 0x7f904415b950 2024-04-25T01:18:13.404 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:18:13.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.422+0000 7f9043200640 1 -- 172.21.15.90:0/2580516263 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f902006be30 msgr2=0x7f902006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:13.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.422+0000 7f9043200640 1 --2- 172.21.15.90:0/2580516263 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f902006be30 0x7f902006e2f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9034006fd0 tx=0x7f9034008040 comp rx=0 tx=0).stop 2024-04-25T01:18:13.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.422+0000 7f9043200640 1 -- 172.21.15.90:0/2580516263 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f904415b950 msgr2=0x7f90440acd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:13.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.422+0000 7f9043200640 1 --2- 172.21.15.90:0/2580516263 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f904415b950 0x7f90440acd50 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f903c034040 tx=0x7f903c004a40 comp rx=0 tx=0).stop 2024-04-25T01:18:13.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.423+0000 7f9043200640 1 -- 172.21.15.90:0/2580516263 shutdown_connections 2024-04-25T01:18:13.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.423+0000 7f9043200640 1 --2- 172.21.15.90:0/2580516263 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f902006be30 0x7f902006e2f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:13.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.423+0000 7f9043200640 1 --2- 172.21.15.90:0/2580516263 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f904415b950 0x7f90440acd50 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:13.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.423+0000 7f9043200640 1 --2- 172.21.15.90:0/2580516263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9044155130 0x7f90441534b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:13.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.423+0000 7f9043200640 1 --2- 172.21.15.90:0/2580516263 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f904406e0b0 0x7f90441f5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:13.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.423+0000 7f9043200640 1 -- 172.21.15.90:0/2580516263 >> 172.21.15.90:0/2580516263 conn(0x7f904406b120 msgr2=0x7f904415e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:13.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.423+0000 7f9043200640 1 -- 172.21.15.90:0/2580516263 shutdown_connections 2024-04-25T01:18:13.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.423+0000 7f9043200640 1 -- 172.21.15.90:0/2580516263 wait complete. 2024-04-25T01:18:13.424 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:18:13.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.641+0000 7f4fdce00640 1 Processor -- start 2024-04-25T01:18:13.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.641+0000 7f4fdce00640 1 -- start start 2024-04-25T01:18:13.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.642+0000 7f4fdce00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f4fd80cf4d0 0x7f4fd81600a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:13.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.642+0000 7f4fdce00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fd806d110 con 0x7f4fd80722e0 2024-04-25T01:18:13.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.642+0000 7f4fdce00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fd806d3f0 con 0x7f4fd815fc40 2024-04-25T01:18:13.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.642+0000 7f4fdce00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f4fd8160aa0 con 0x7f4fd80cf4d0 2024-04-25T01:18:13.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.643+0000 7f4fd5e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4fd80722e0 0x7f4fd806ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:49438/0 (socket says 172.21.15.90:49438) 2024-04-25T01:18:13.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.643+0000 7f4fd5e00640 1 -- 172.21.15.90:0/2674609574 learned_addr learned my addr 172.21.15.90:0/2674609574 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:13.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.643+0000 7f4fd5400640 1 -- 172.21.15.90:0/2674609574 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1467850362 0 0) 0x7f4fd806d110 con 0x7f4fd80722e0 2024-04-25T01:18:13.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.643+0000 7f4fd5400640 1 -- 172.21.15.90:0/2674609574 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4fbc003520 con 0x7f4fd80722e0 2024-04-25T01:18:13.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.643+0000 7f4fd6800640 1 --2- 172.21.15.90:0/2674609574 >> v2:172.21.15.103:3300/0 conn(0x7f4fd80cf4d0 0x7f4fd81600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:13.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.643+0000 7f4fd5400640 1 -- 172.21.15.90:0/2674609574 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 519635540 0 0) 0x7f4fbc003520 con 0x7f4fd80722e0 2024-04-25T01:18:13.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.643+0000 7f4fd5400640 1 -- 172.21.15.90:0/2674609574 >> v2:172.21.15.103:3300/0 conn(0x7f4fd80cf4d0 msgr2=0x7f4fd81600a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:13.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.643+0000 7f4fd5400640 1 --2- 172.21.15.90:0/2674609574 >> v2:172.21.15.103:3300/0 conn(0x7f4fd80cf4d0 0x7f4fd81600a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:13.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.643+0000 7f4fd5400640 1 -- 172.21.15.90:0/2674609574 >> v1:172.21.15.103:6789/0 conn(0x7f4fd815fc40 legacy=0x7f4fd80cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:13.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.643+0000 7f4fd5400640 1 -- 172.21.15.90:0/2674609574 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fd806d6d0 con 0x7f4fd80722e0 2024-04-25T01:18:13.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.643+0000 7f4fd5400640 1 -- 172.21.15.90:0/2674609574 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f4fc8001670 con 0x7f4fd80722e0 2024-04-25T01:18:13.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.643+0000 7f4fd5400640 1 -- 172.21.15.90:0/2674609574 >> v1:172.21.15.90:6789/0 conn(0x7f4fd80722e0 legacy=0x7f4fd806ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:13.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.644+0000 7f4fd5400640 1 --2- 172.21.15.90:0/2674609574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fbc003a00 0x7f4fbc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:13.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.644+0000 7f4fd5400640 1 --2- 172.21.15.90:0/2674609574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fbc004800 0x7f4fbc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:13.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.644+0000 7f4fd5400640 1 --2- 172.21.15.90:0/2674609574 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fbc005290 0x7f4fbc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:13.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.644+0000 7f4fd5400640 1 -- 172.21.15.90:0/2674609574 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd8160aa0 con 0x7f4fbc004800 2024-04-25T01:18:13.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.644+0000 7f4fd5400640 1 -- 172.21.15.90:0/2674609574 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4fbc00c1e0 con 0x7f4fbc003a00 2024-04-25T01:18:13.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.644+0000 7f4fd5400640 1 -- 172.21.15.90:0/2674609574 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4fbc00c3e0 con 0x7f4fbc005290 2024-04-25T01:18:13.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.644+0000 7f4fcfe00640 1 --2- 172.21.15.90:0/2674609574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fbc004800 0x7f4fbc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:13.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.644+0000 7f4fd5e00640 1 --2- 172.21.15.90:0/2674609574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fbc003a00 0x7f4fbc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:13.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.644+0000 7f4fcfe00640 1 -- 172.21.15.90:0/2674609574 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fbc005290 msgr2=0x7f4fbc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:13.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.644+0000 7f4fcfe00640 1 --2- 172.21.15.90:0/2674609574 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fbc005290 0x7f4fbc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:13.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.644+0000 7f4fcfe00640 1 -- 172.21.15.90:0/2674609574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fbc003a00 msgr2=0x7f4fbc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:13.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.644+0000 7f4fcfe00640 1 --2- 172.21.15.90:0/2674609574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fbc003a00 0x7f4fbc003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:13.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.644+0000 7f4fcfe00640 1 -- 172.21.15.90:0/2674609574 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fbc003520 con 0x7f4fbc004800 2024-04-25T01:18:13.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.644+0000 7f4fd6800640 1 -- 172.21.15.90:0/2674609574 reap_dead start 2024-04-25T01:18:13.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.644+0000 7f4fd5e00640 1 --2- 172.21.15.90:0/2674609574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fbc003a00 0x7f4fbc003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:13.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.645+0000 7f4fcfe00640 1 --2- 172.21.15.90:0/2674609574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fbc004800 0x7f4fbc004c80 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0003db0 tx=0x7f4fbc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eefbd5aff329e47 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:13.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.645+0000 7f4fd5400640 1 -- 172.21.15.90:0/2674609574 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc0037070 con 0x7f4fbc004800 2024-04-25T01:18:13.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.645+0000 7f4fd5400640 1 -- 172.21.15.90:0/2674609574 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fc002aa00 con 0x7f4fbc004800 2024-04-25T01:18:13.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.645+0000 7f4fd5400640 1 -- 172.21.15.90:0/2674609574 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc0003980 con 0x7f4fbc004800 2024-04-25T01:18:13.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.646+0000 7f4fdce00640 1 -- 172.21.15.90:0/2674609574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fbc004800 msgr2=0x7f4fbc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:13.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.646+0000 7f4fdce00640 1 --2- 172.21.15.90:0/2674609574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fbc004800 0x7f4fbc004c80 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0003db0 tx=0x7f4fbc005830 comp rx=0 tx=0).stop 2024-04-25T01:18:13.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.646+0000 7f4fdce00640 1 -- 172.21.15.90:0/2674609574 shutdown_connections 2024-04-25T01:18:13.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.646+0000 7f4fdce00640 1 --2- 172.21.15.90:0/2674609574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fbc004800 0x7f4fbc004c80 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:13.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.646+0000 7f4fdce00640 1 -- 172.21.15.90:0/2674609574 >> 172.21.15.90:0/2674609574 conn(0x7f4fd806b720 msgr2=0x7f4fd80b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:13.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.646+0000 7f4fdce00640 1 -- 172.21.15.90:0/2674609574 shutdown_connections 2024-04-25T01:18:13.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.646+0000 7f4fdce00640 1 -- 172.21.15.90:0/2674609574 wait complete. 2024-04-25T01:18:13.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.648+0000 7f4fdce00640 1 Processor -- start 2024-04-25T01:18:13.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.648+0000 7f4fdce00640 1 -- start start 2024-04-25T01:18:13.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.649+0000 7f4fdce00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fbc004800 0x7f4fd8150880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:13.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.649+0000 7f4fdce00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fd813aff0 0x7f4fd813b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:13.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.649+0000 7f4fdce00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fd8149e60 0x7f4fd814a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:13.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.649+0000 7f4fdce00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd8160430 con 0x7f4fbc004800 2024-04-25T01:18:13.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.649+0000 7f4fdce00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd81602b0 con 0x7f4fd8149e60 2024-04-25T01:18:13.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.649+0000 7f4fdce00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f4fd81605b0 con 0x7f4fd813aff0 2024-04-25T01:18:13.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.649+0000 7f4fd5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fbc004800 0x7f4fd8150880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:13.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.649+0000 7f4fd5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fbc004800 0x7f4fd8150880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33364/0 (socket says 172.21.15.90:33364) 2024-04-25T01:18:13.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.649+0000 7f4fd5e00640 1 -- 172.21.15.90:0/76620435 learned_addr learned my addr 172.21.15.90:0/76620435 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:13.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.649+0000 7f4fd5e00640 1 -- 172.21.15.90:0/76620435 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fd813aff0 msgr2=0x7f4fd813b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:13.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.649+0000 7f4fd5e00640 1 --2- 172.21.15.90:0/76620435 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fd813aff0 0x7f4fd813b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:13.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.649+0000 7f4fd5e00640 1 -- 172.21.15.90:0/76620435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fd8149e60 msgr2=0x7f4fd814a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:13.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.649+0000 7f4fd6800640 1 --2- 172.21.15.90:0/76620435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fd8149e60 0x7f4fd814a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:13.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.649+0000 7f4fd5e00640 1 --2- 172.21.15.90:0/76620435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fd8149e60 0x7f4fd814a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:13.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.650+0000 7f4fd5e00640 1 -- 172.21.15.90:0/76620435 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fd81487b0 con 0x7f4fbc004800 2024-04-25T01:18:13.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.650+0000 7f4fd5e00640 1 --2- 172.21.15.90:0/76620435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fbc004800 0x7f4fd8150880 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f4fc800d780 tx=0x7f4fc800dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:13.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.650+0000 7f4fce000640 1 -- 172.21.15.90:0/76620435 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc8019070 con 0x7f4fbc004800 2024-04-25T01:18:13.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.650+0000 7f4fce000640 1 -- 172.21.15.90:0/76620435 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fc80074e0 con 0x7f4fbc004800 2024-04-25T01:18:13.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.650+0000 7f4fce000640 1 -- 172.21.15.90:0/76620435 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc8004a00 con 0x7f4fbc004800 2024-04-25T01:18:13.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.650+0000 7f4fdce00640 1 -- 172.21.15.90:0/76620435 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fd8148a90 con 0x7f4fbc004800 2024-04-25T01:18:13.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.650+0000 7f4fdce00640 1 -- 172.21.15.90:0/76620435 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fd8149050 con 0x7f4fbc004800 2024-04-25T01:18:13.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.651+0000 7f4fdce00640 1 -- 172.21.15.90:0/76620435 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fd8068fb0 con 0x7f4fbc004800 2024-04-25T01:18:13.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.652+0000 7f4fce000640 1 -- 172.21.15.90:0/76620435 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4fc8004ba0 con 0x7f4fbc004800 2024-04-25T01:18:13.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.653+0000 7f4fce000640 1 --2- 172.21.15.90:0/76620435 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4fb406be60 0x7f4fb406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:13.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.653+0000 7f4fce000640 1 -- 172.21.15.90:0/76620435 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6159+0+0 (secure 0 0 0) 0x7f4fc808aa80 con 0x7f4fbc004800 2024-04-25T01:18:13.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.654+0000 7f4fcfe00640 1 --2- 172.21.15.90:0/76620435 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4fb406be60 0x7f4fb406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:13.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.654+0000 7f4fcfe00640 1 --2- 172.21.15.90:0/76620435 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4fb406be60 0x7f4fb406e320 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0034000 tx=0x7f4fc002c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:13.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.657+0000 7f4fce000640 1 -- 172.21.15.90:0/76620435 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4fc808ce20 con 0x7f4fbc004800 2024-04-25T01:18:13.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.857+0000 7f4fdce00640 1 -- 172.21.15.90:0/76620435 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4fd8072c90 con 0x7f4fbc004800 2024-04-25T01:18:13.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.858+0000 7f4fce000640 1 -- 172.21.15.90:0/76620435 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) ==== 100+0+11 (secure 0 0 0) 0x7f4fc80591d0 con 0x7f4fbc004800 2024-04-25T01:18:13.841 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-25T01:18:13.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.861+0000 7f4fdce00640 1 -- 172.21.15.90:0/76620435 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4fb406be60 msgr2=0x7f4fb406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:13.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.861+0000 7f4fdce00640 1 --2- 172.21.15.90:0/76620435 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4fb406be60 0x7f4fb406e320 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0034000 tx=0x7f4fc002c000 comp rx=0 tx=0).stop 2024-04-25T01:18:13.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.861+0000 7f4fdce00640 1 -- 172.21.15.90:0/76620435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fbc004800 msgr2=0x7f4fd8150880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:13.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.861+0000 7f4fdce00640 1 --2- 172.21.15.90:0/76620435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fbc004800 0x7f4fd8150880 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f4fc800d780 tx=0x7f4fc800dc40 comp rx=0 tx=0).stop 2024-04-25T01:18:13.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.861+0000 7f4fdce00640 1 -- 172.21.15.90:0/76620435 shutdown_connections 2024-04-25T01:18:13.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.861+0000 7f4fdce00640 1 --2- 172.21.15.90:0/76620435 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f4fb406be60 0x7f4fb406e320 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:13.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.861+0000 7f4fdce00640 1 --2- 172.21.15.90:0/76620435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fd8149e60 0x7f4fd814a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:13.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.861+0000 7f4fdce00640 1 --2- 172.21.15.90:0/76620435 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f4fd813aff0 0x7f4fd813b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:13.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.861+0000 7f4fdce00640 1 --2- 172.21.15.90:0/76620435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fbc004800 0x7f4fd8150880 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:13.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.861+0000 7f4fdce00640 1 -- 172.21.15.90:0/76620435 >> 172.21.15.90:0/76620435 conn(0x7f4fd806b720 msgr2=0x7f4fd8070b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:13.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.861+0000 7f4fdce00640 1 -- 172.21.15.90:0/76620435 shutdown_connections 2024-04-25T01:18:13.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:13.861+0000 7f4fdce00640 1 -- 172.21.15.90:0/76620435 wait complete. 2024-04-25T01:18:13.863 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T01:18:14.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.082+0000 7f15a2a00640 1 Processor -- start 2024-04-25T01:18:14.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.083+0000 7f15a2a00640 1 -- start start 2024-04-25T01:18:14.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.084+0000 7f15a2a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f159c0b2bf0 0x7f159c0b1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:14.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.084+0000 7f15a2a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f159c0d0e90 con 0x7f159c161680 2024-04-25T01:18:14.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.084+0000 7f15a2a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f159c0d4940 con 0x7f159c0d26c0 2024-04-25T01:18:14.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.084+0000 7f15a2a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f159c0d4c20 con 0x7f159c0b2bf0 2024-04-25T01:18:14.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.084+0000 7f15a1600640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f159c161680 0x7f159c0d1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:49452/0 (socket says 172.21.15.90:49452) 2024-04-25T01:18:14.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.085+0000 7f15a1600640 1 -- 172.21.15.90:0/3425611661 learned_addr learned my addr 172.21.15.90:0/3425611661 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:14.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.085+0000 7f15a2000640 1 --2- 172.21.15.90:0/3425611661 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f159c0b2bf0 0x7f159c0b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:14.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.085+0000 7f159be00640 1 -- 172.21.15.90:0/3425611661 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2757880648 0 0) 0x7f159c0d0e90 con 0x7f159c161680 2024-04-25T01:18:14.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.085+0000 7f159be00640 1 -- 172.21.15.90:0/3425611661 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1584003520 con 0x7f159c161680 2024-04-25T01:18:14.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.085+0000 7f159be00640 1 -- 172.21.15.90:0/3425611661 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 217264377 0 0) 0x7f1584003520 con 0x7f159c161680 2024-04-25T01:18:14.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.085+0000 7f159be00640 1 -- 172.21.15.90:0/3425611661 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f159c0b2bf0 msgr2=0x7f159c0b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:14.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.085+0000 7f159be00640 1 --2- 172.21.15.90:0/3425611661 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f159c0b2bf0 0x7f159c0b1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:14.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.085+0000 7f159be00640 1 -- 172.21.15.90:0/3425611661 >> v1:172.21.15.103:6789/0 conn(0x7f159c0d26c0 legacy=0x7f159c0b1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:14.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.085+0000 7f159be00640 1 -- 172.21.15.90:0/3425611661 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f159c0b0ae0 con 0x7f159c161680 2024-04-25T01:18:14.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.086+0000 7f159be00640 1 -- 172.21.15.90:0/3425611661 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f1590001690 con 0x7f159c161680 2024-04-25T01:18:14.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.086+0000 7f159be00640 1 -- 172.21.15.90:0/3425611661 >> v1:172.21.15.90:6789/0 conn(0x7f159c161680 legacy=0x7f159c0d1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:14.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.086+0000 7f159be00640 1 --2- 172.21.15.90:0/3425611661 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f159c0b2bf0 0x7f15840039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:14.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.086+0000 7f159be00640 1 --2- 172.21.15.90:0/3425611661 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1584003f60 0x7f1584004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:14.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.086+0000 7f159be00640 1 --2- 172.21.15.90:0/3425611661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1584004d40 0x7f15840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:14.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.086+0000 7f159be00640 1 -- 172.21.15.90:0/3425611661 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f159c0d4c20 con 0x7f1584003f60 2024-04-25T01:18:14.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.086+0000 7f159be00640 1 -- 172.21.15.90:0/3425611661 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f158400bb50 con 0x7f1584004d40 2024-04-25T01:18:14.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.086+0000 7f159be00640 1 -- 172.21.15.90:0/3425611661 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f158400bd50 con 0x7f159c0b2bf0 2024-04-25T01:18:14.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.086+0000 7f15a0c00640 1 --2- 172.21.15.90:0/3425611661 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1584003f60 0x7f1584004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:14.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.086+0000 7f15a2000640 1 --2- 172.21.15.90:0/3425611661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1584004d40 0x7f15840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:14.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.086+0000 7f15a1600640 1 --2- 172.21.15.90:0/3425611661 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f159c0b2bf0 0x7f15840039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:14.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.086+0000 7f15a0c00640 1 -- 172.21.15.90:0/3425611661 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f159c0b2bf0 msgr2=0x7f15840039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:14.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.086+0000 7f15a0c00640 1 --2- 172.21.15.90:0/3425611661 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f159c0b2bf0 0x7f15840039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:14.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.086+0000 7f15a0c00640 1 -- 172.21.15.90:0/3425611661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1584004d40 msgr2=0x7f15840051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:14.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.086+0000 7f15a0c00640 1 --2- 172.21.15.90:0/3425611661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1584004d40 0x7f15840051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:14.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.086+0000 7f15a0c00640 1 -- 172.21.15.90:0/3425611661 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1584003520 con 0x7f1584003f60 2024-04-25T01:18:14.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.086+0000 7f15a2000640 1 --2- 172.21.15.90:0/3425611661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1584004d40 0x7f15840051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:14.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.086+0000 7f15a1600640 1 --2- 172.21.15.90:0/3425611661 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f159c0b2bf0 0x7f15840039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:14.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.088+0000 7f15a0c00640 1 --2- 172.21.15.90:0/3425611661 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1584003f60 0x7f1584004800 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f158c001750 tx=0x7f1584005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0491eb5dfdcc4a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:14.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.088+0000 7f159be00640 1 -- 172.21.15.90:0/3425611661 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f158c001370 con 0x7f1584003f60 2024-04-25T01:18:14.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.088+0000 7f159be00640 1 -- 172.21.15.90:0/3425611661 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f158c0052d0 con 0x7f1584003f60 2024-04-25T01:18:14.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.088+0000 7f159be00640 1 -- 172.21.15.90:0/3425611661 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f158c02abe0 con 0x7f1584003f60 2024-04-25T01:18:14.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.088+0000 7f15a2a00640 1 -- 172.21.15.90:0/3425611661 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1584003f60 msgr2=0x7f1584004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:14.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.088+0000 7f15a2a00640 1 --2- 172.21.15.90:0/3425611661 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1584003f60 0x7f1584004800 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f158c001750 tx=0x7f1584005830 comp rx=0 tx=0).stop 2024-04-25T01:18:14.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.088+0000 7f15a2000640 1 -- 172.21.15.90:0/3425611661 reap_dead start 2024-04-25T01:18:14.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.089+0000 7f15a2a00640 1 -- 172.21.15.90:0/3425611661 shutdown_connections 2024-04-25T01:18:14.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.089+0000 7f15a2a00640 1 -- 172.21.15.90:0/3425611661 >> 172.21.15.90:0/3425611661 conn(0x7f159c0d3cf0 msgr2=0x7f159c0d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:14.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.089+0000 7f15a2a00640 1 -- 172.21.15.90:0/3425611661 shutdown_connections 2024-04-25T01:18:14.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.089+0000 7f15a2a00640 1 -- 172.21.15.90:0/3425611661 wait complete. 2024-04-25T01:18:14.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.090+0000 7f15a2a00640 1 Processor -- start 2024-04-25T01:18:14.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.091+0000 7f15a2a00640 1 -- start start 2024-04-25T01:18:14.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.091+0000 7f15a2a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1584003f60 0x7f159c15be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:14.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.092+0000 7f15a2a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f159c15c5c0 0x7f159c0cd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:14.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.092+0000 7f15a2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f159c0cbc50 0x7f159c06c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:14.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.092+0000 7f15a2a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f159c0d1ad0 con 0x7f159c0cbc50 2024-04-25T01:18:14.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.092+0000 7f15a2a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f159c0d1c50 con 0x7f1584003f60 2024-04-25T01:18:14.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.092+0000 7f15a2a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f159c0d1dd0 con 0x7f159c15c5c0 2024-04-25T01:18:14.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.092+0000 7f15a2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f159c0cbc50 0x7f159c06c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:14.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.092+0000 7f15a2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f159c0cbc50 0x7f159c06c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33384/0 (socket says 172.21.15.90:33384) 2024-04-25T01:18:14.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.092+0000 7f15a2000640 1 -- 172.21.15.90:0/1102323542 learned_addr learned my addr 172.21.15.90:0/1102323542 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:14.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.092+0000 7f15a0c00640 1 --2- 172.21.15.90:0/1102323542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f159c15c5c0 0x7f159c0cd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:14.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.092+0000 7f15a1600640 1 --2- 172.21.15.90:0/1102323542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1584003f60 0x7f159c15be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:14.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.092+0000 7f15a2000640 1 -- 172.21.15.90:0/1102323542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f159c15c5c0 msgr2=0x7f159c0cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:14.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.092+0000 7f15a2000640 1 --2- 172.21.15.90:0/1102323542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f159c15c5c0 0x7f159c0cd7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:14.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.092+0000 7f15a2000640 1 -- 172.21.15.90:0/1102323542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1584003f60 msgr2=0x7f159c15be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:14.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.092+0000 7f15a2000640 1 --2- 172.21.15.90:0/1102323542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1584003f60 0x7f159c15be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:14.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.092+0000 7f15a2000640 1 -- 172.21.15.90:0/1102323542 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f159c15c390 con 0x7f159c0cbc50 2024-04-25T01:18:14.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.093+0000 7f15a2000640 1 --2- 172.21.15.90:0/1102323542 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f159c0cbc50 0x7f159c06c300 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f159c08c690 tx=0x7f159400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:14.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.093+0000 7f1599600640 1 -- 172.21.15.90:0/1102323542 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159401c070 con 0x7f159c0cbc50 2024-04-25T01:18:14.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.093+0000 7f1599600640 1 -- 172.21.15.90:0/1102323542 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1594017400 con 0x7f159c0cbc50 2024-04-25T01:18:14.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.093+0000 7f15a2a00640 1 -- 172.21.15.90:0/1102323542 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f159c0cca20 con 0x7f159c0cbc50 2024-04-25T01:18:14.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.093+0000 7f1599600640 1 -- 172.21.15.90:0/1102323542 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15940177d0 con 0x7f159c0cbc50 2024-04-25T01:18:14.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.093+0000 7f15a2a00640 1 -- 172.21.15.90:0/1102323542 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f159c0ccfe0 con 0x7f159c0cbc50 2024-04-25T01:18:14.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.094+0000 7f15a2a00640 1 -- 172.21.15.90:0/1102323542 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f159c075350 con 0x7f159c0cbc50 2024-04-25T01:18:14.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.095+0000 7f1599600640 1 -- 172.21.15.90:0/1102323542 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1594005d10 con 0x7f159c0cbc50 2024-04-25T01:18:14.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.096+0000 7f1599600640 1 --2- 172.21.15.90:0/1102323542 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f158006be60 0x7f158006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:14.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.096+0000 7f1599600640 1 -- 172.21.15.90:0/1102323542 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6159+0+0 (secure 0 0 0) 0x7f159408b2e0 con 0x7f159c0cbc50 2024-04-25T01:18:14.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.096+0000 7f15a1600640 1 --2- 172.21.15.90:0/1102323542 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f158006be60 0x7f158006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:14.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.097+0000 7f15a1600640 1 --2- 172.21.15.90:0/1102323542 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f158006be60 0x7f158006e320 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1590000fd0 tx=0x7f1590009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:14.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.100+0000 7f1599600640 1 -- 172.21.15.90:0/1102323542 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1594012070 con 0x7f159c0cbc50 2024-04-25T01:18:14.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.311+0000 7f15a2a00640 1 -- 172.21.15.90:0/1102323542 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f159c0d0860 con 0x7f159c0cbc50 2024-04-25T01:18:14.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.311+0000 7f1599600640 1 -- 172.21.15.90:0/1102323542 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v42) ==== 80+0+0 (secure 0 0 0) 0x7f1594059ac0 con 0x7f159c0cbc50 2024-04-25T01:18:14.295 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-04-25T01:18:14.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.315+0000 7f15a2a00640 1 -- 172.21.15.90:0/1102323542 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f158006be60 msgr2=0x7f158006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:14.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.315+0000 7f15a2a00640 1 --2- 172.21.15.90:0/1102323542 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f158006be60 0x7f158006e320 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1590000fd0 tx=0x7f1590009040 comp rx=0 tx=0).stop 2024-04-25T01:18:14.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.315+0000 7f15a2a00640 1 -- 172.21.15.90:0/1102323542 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f159c0cbc50 msgr2=0x7f159c06c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:14.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.315+0000 7f15a2a00640 1 --2- 172.21.15.90:0/1102323542 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f159c0cbc50 0x7f159c06c300 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f159c08c690 tx=0x7f159400f4a0 comp rx=0 tx=0).stop 2024-04-25T01:18:14.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.315+0000 7f15a2a00640 1 -- 172.21.15.90:0/1102323542 shutdown_connections 2024-04-25T01:18:14.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.315+0000 7f15a2a00640 1 --2- 172.21.15.90:0/1102323542 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f158006be60 0x7f158006e320 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:14.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.315+0000 7f15a2a00640 1 --2- 172.21.15.90:0/1102323542 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f159c0cbc50 0x7f159c06c300 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:14.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.315+0000 7f15a2a00640 1 --2- 172.21.15.90:0/1102323542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f159c15c5c0 0x7f159c0cd7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:14.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.315+0000 7f15a2a00640 1 --2- 172.21.15.90:0/1102323542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1584003f60 0x7f159c15be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:14.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.315+0000 7f15a2a00640 1 -- 172.21.15.90:0/1102323542 >> 172.21.15.90:0/1102323542 conn(0x7f159c0d3cf0 msgr2=0x7f159c0d1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:14.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.315+0000 7f15a2a00640 1 -- 172.21.15.90:0/1102323542 shutdown_connections 2024-04-25T01:18:14.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.315+0000 7f15a2a00640 1 -- 172.21.15.90:0/1102323542 wait complete. 2024-04-25T01:18:14.317 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-25T01:18:14.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:14.317 DEBUG:teuthology.orchestra.run.smithi090:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-25T01:18:14.452 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-04-25T01:18:14.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:14.453 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T01:18:14.509 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:18:14.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.686+0000 7fdc90000640 1 Processor -- start 2024-04-25T01:18:14.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.686+0000 7fdc90000640 1 -- start start 2024-04-25T01:18:14.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.688+0000 7fdc90000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc880b4db0 0x7fdc88162360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:14.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.688+0000 7fdc90000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc880b3a10 con 0x7fdc8815de10 2024-04-25T01:18:14.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.688+0000 7fdc90000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc880b3cf0 con 0x7fdc8815e1f0 2024-04-25T01:18:14.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.688+0000 7fdc90000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdc880b3fd0 con 0x7fdc880b4db0 2024-04-25T01:18:14.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.688+0000 7fdc8d600640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fdc8815de10 0x7fdc88161c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:49466/0 (socket says 172.21.15.90:49466) 2024-04-25T01:18:14.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.688+0000 7fdc8d600640 1 -- 172.21.15.90:0/2217844458 learned_addr learned my addr 172.21.15.90:0/2217844458 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:14.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.688+0000 7fdc87e00640 1 -- 172.21.15.90:0/2217844458 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1900215484 0 0) 0x7fdc880b3a10 con 0x7fdc8815de10 2024-04-25T01:18:14.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.688+0000 7fdc87e00640 1 -- 172.21.15.90:0/2217844458 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc70003520 con 0x7fdc8815de10 2024-04-25T01:18:14.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.689+0000 7fdc8cc00640 1 --2- 172.21.15.90:0/2217844458 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc880b4db0 0x7fdc88162360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:14.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.689+0000 7fdc87e00640 1 -- 172.21.15.90:0/2217844458 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4137290199 0 0) 0x7fdc70003520 con 0x7fdc8815de10 2024-04-25T01:18:14.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.689+0000 7fdc87e00640 1 -- 172.21.15.90:0/2217844458 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc880b4db0 msgr2=0x7fdc88162360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:14.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.689+0000 7fdc87e00640 1 --2- 172.21.15.90:0/2217844458 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc880b4db0 0x7fdc88162360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:14.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.689+0000 7fdc87e00640 1 -- 172.21.15.90:0/2217844458 >> v1:172.21.15.103:6789/0 conn(0x7fdc8815e1f0 legacy=0x7fdc880719c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:14.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.689+0000 7fdc87e00640 1 -- 172.21.15.90:0/2217844458 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc880b4660 con 0x7fdc8815de10 2024-04-25T01:18:14.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.689+0000 7fdc87e00640 1 -- 172.21.15.90:0/2217844458 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7fdc7c001670 con 0x7fdc8815de10 2024-04-25T01:18:14.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.689+0000 7fdc87e00640 1 -- 172.21.15.90:0/2217844458 >> v1:172.21.15.90:6789/0 conn(0x7fdc8815de10 legacy=0x7fdc88161c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:14.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.689+0000 7fdc87e00640 1 --2- 172.21.15.90:0/2217844458 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc700039e0 0x7fdc70003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:14.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.689+0000 7fdc87e00640 1 --2- 172.21.15.90:0/2217844458 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc70004800 0x7fdc70004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:14.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.689+0000 7fdc87e00640 1 --2- 172.21.15.90:0/2217844458 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc880b4db0 0x7fdc7000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:14.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.689+0000 7fdc87e00640 1 -- 172.21.15.90:0/2217844458 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdc880b3fd0 con 0x7fdc700039e0 2024-04-25T01:18:14.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.689+0000 7fdc87e00640 1 -- 172.21.15.90:0/2217844458 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdc70005550 con 0x7fdc70004800 2024-04-25T01:18:14.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.689+0000 7fdc87e00640 1 -- 172.21.15.90:0/2217844458 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdc7000bf00 con 0x7fdc880b4db0 2024-04-25T01:18:14.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.689+0000 7fdc8d600640 1 --2- 172.21.15.90:0/2217844458 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc700039e0 0x7fdc70003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:14.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.690+0000 7fdc8cc00640 1 --2- 172.21.15.90:0/2217844458 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc70004800 0x7fdc70004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:14.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.690+0000 7fdc8d600640 1 -- 172.21.15.90:0/2217844458 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc880b4db0 msgr2=0x7fdc7000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:14.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.690+0000 7fdc8d600640 1 --2- 172.21.15.90:0/2217844458 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc880b4db0 0x7fdc7000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:14.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.690+0000 7fdc8d600640 1 -- 172.21.15.90:0/2217844458 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc70004800 msgr2=0x7fdc70004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:14.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.690+0000 7fdc8d600640 1 --2- 172.21.15.90:0/2217844458 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc70004800 0x7fdc70004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:14.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.690+0000 7fdc8d600640 1 -- 172.21.15.90:0/2217844458 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc70003520 con 0x7fdc700039e0 2024-04-25T01:18:14.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.691+0000 7fdc8d600640 1 --2- 172.21.15.90:0/2217844458 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc700039e0 0x7fdc70003de0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c00c290 tx=0x7fdc70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e82e0bbcad549a48 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:14.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.691+0000 7fdc87e00640 1 -- 172.21.15.90:0/2217844458 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c007d50 con 0x7fdc700039e0 2024-04-25T01:18:14.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.691+0000 7fdc87e00640 1 -- 172.21.15.90:0/2217844458 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc7c005af0 con 0x7fdc700039e0 2024-04-25T01:18:14.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.691+0000 7fdc87e00640 1 -- 172.21.15.90:0/2217844458 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c007070 con 0x7fdc700039e0 2024-04-25T01:18:14.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.691+0000 7fdc90000640 1 -- 172.21.15.90:0/2217844458 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc700039e0 msgr2=0x7fdc70003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:14.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.691+0000 7fdc90000640 1 --2- 172.21.15.90:0/2217844458 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc700039e0 0x7fdc70003de0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c00c290 tx=0x7fdc70005830 comp rx=0 tx=0).stop 2024-04-25T01:18:14.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.692+0000 7fdc8e000640 1 -- 172.21.15.90:0/2217844458 reap_dead start 2024-04-25T01:18:14.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.692+0000 7fdc90000640 1 -- 172.21.15.90:0/2217844458 shutdown_connections 2024-04-25T01:18:14.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.692+0000 7fdc90000640 1 -- 172.21.15.90:0/2217844458 >> 172.21.15.90:0/2217844458 conn(0x7fdc880b1820 msgr2=0x7fdc880b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:14.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.692+0000 7fdc90000640 1 -- 172.21.15.90:0/2217844458 shutdown_connections 2024-04-25T01:18:14.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.692+0000 7fdc90000640 1 -- 172.21.15.90:0/2217844458 wait complete. 2024-04-25T01:18:14.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.693+0000 7fdc90000640 1 Processor -- start 2024-04-25T01:18:14.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.693+0000 7fdc90000640 1 -- start start 2024-04-25T01:18:14.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.694+0000 7fdc90000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc700039e0 0x7fdc88140e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:14.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.694+0000 7fdc90000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc880cc540 0x7fdc880cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:14.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.694+0000 7fdc90000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc8815bdd0 0x7fdc8815c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:14.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.694+0000 7fdc90000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdc881626f0 con 0x7fdc8815bdd0 2024-04-25T01:18:14.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.694+0000 7fdc90000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdc88162570 con 0x7fdc700039e0 2024-04-25T01:18:14.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.694+0000 7fdc90000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fdc88162870 con 0x7fdc880cc540 2024-04-25T01:18:14.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.695+0000 7fdc8e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc8815bdd0 0x7fdc8815c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:14.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.695+0000 7fdc8e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc8815bdd0 0x7fdc8815c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33412/0 (socket says 172.21.15.90:33412) 2024-04-25T01:18:14.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.695+0000 7fdc8e000640 1 -- 172.21.15.90:0/944481452 learned_addr learned my addr 172.21.15.90:0/944481452 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:14.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.695+0000 7fdc8d600640 1 --2- 172.21.15.90:0/944481452 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc700039e0 0x7fdc88140e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:14.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.695+0000 7fdc8cc00640 1 --2- 172.21.15.90:0/944481452 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc880cc540 0x7fdc880cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:14.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.695+0000 7fdc8e000640 1 -- 172.21.15.90:0/944481452 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc880cc540 msgr2=0x7fdc880cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:14.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.695+0000 7fdc8e000640 1 --2- 172.21.15.90:0/944481452 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc880cc540 0x7fdc880cc9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:14.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.695+0000 7fdc8e000640 1 -- 172.21.15.90:0/944481452 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc700039e0 msgr2=0x7fdc88140e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:14.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.695+0000 7fdc8e000640 1 --2- 172.21.15.90:0/944481452 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc700039e0 0x7fdc88140e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:14.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.695+0000 7fdc8e000640 1 -- 172.21.15.90:0/944481452 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc880ce0a0 con 0x7fdc8815bdd0 2024-04-25T01:18:14.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.696+0000 7fdc8e000640 1 --2- 172.21.15.90:0/944481452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc8815bdd0 0x7fdc8815c250 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fdc8815e650 tx=0x7fdc8000c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:14.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.696+0000 7fdc85600640 1 -- 172.21.15.90:0/944481452 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc80017070 con 0x7fdc8815bdd0 2024-04-25T01:18:14.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.696+0000 7fdc85600640 1 -- 172.21.15.90:0/944481452 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc80007070 con 0x7fdc8815bdd0 2024-04-25T01:18:14.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.696+0000 7fdc85600640 1 -- 172.21.15.90:0/944481452 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc80007460 con 0x7fdc8815bdd0 2024-04-25T01:18:14.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.696+0000 7fdc90000640 1 -- 172.21.15.90:0/944481452 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc88141380 con 0x7fdc8815bdd0 2024-04-25T01:18:14.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.696+0000 7fdc90000640 1 -- 172.21.15.90:0/944481452 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc880cd2e0 con 0x7fdc8815bdd0 2024-04-25T01:18:14.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.697+0000 7fdc90000640 1 -- 172.21.15.90:0/944481452 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc88068fb0 con 0x7fdc8815bdd0 2024-04-25T01:18:14.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.698+0000 7fdc85600640 1 -- 172.21.15.90:0/944481452 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fdc8001a070 con 0x7fdc8815bdd0 2024-04-25T01:18:14.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.699+0000 7fdc85600640 1 --2- 172.21.15.90:0/944481452 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdc6806be30 0x7fdc6806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:14.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.699+0000 7fdc85600640 1 -- 172.21.15.90:0/944481452 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6159+0+0 (secure 0 0 0) 0x7fdc8008a2c0 con 0x7fdc8815bdd0 2024-04-25T01:18:14.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.699+0000 7fdc8d600640 1 --2- 172.21.15.90:0/944481452 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdc6806be30 0x7fdc6806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:14.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.700+0000 7fdc8d600640 1 --2- 172.21.15.90:0/944481452 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdc6806be30 0x7fdc6806e2f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c033000 tx=0x7fdc7c02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:14.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.703+0000 7fdc85600640 1 -- 172.21.15.90:0/944481452 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fdc80058aa0 con 0x7fdc8815bdd0 2024-04-25T01:18:14.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.924+0000 7fdc90000640 1 -- 172.21.15.90:0/944481452 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc88070eb0 con 0x7fdc8815bdd0 2024-04-25T01:18:14.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.925+0000 7fdc85600640 1 -- 172.21.15.90:0/944481452 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5184 (secure 0 0 0) 0x7fdc800583a0 con 0x7fdc8815bdd0 2024-04-25T01:18:14.908 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:14.909 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4598,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1475874994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1475874994},{"type":"v1","addr":"172.21.15.90:6835","nonce":1475874994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4610,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6834/1993144745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1993144745},{"type":"v1","addr":"172.21.15.103:6834","nonce":1993144745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4840,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2376794333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6833","nonce":2376794333},{"type":"v1","addr":"172.21.15.103:6835","nonce":2376794333}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:18:02.701652+0000","modified":"2024-04-25T01:18:05.730947+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5334},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5334":{"gid":5334,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.90:6837/431390375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":431390375},{"type":"v1","addr":"172.21.15.90:6837","nonce":431390375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5334,"qdb_cluster":[5334]},"id":2}]} 2024-04-25T01:18:14.909 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 18 2024-04-25T01:18:14.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.928+0000 7fdc90000640 1 -- 172.21.15.90:0/944481452 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdc6806be30 msgr2=0x7fdc6806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:14.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.928+0000 7fdc90000640 1 --2- 172.21.15.90:0/944481452 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdc6806be30 0x7fdc6806e2f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c033000 tx=0x7fdc7c02e000 comp rx=0 tx=0).stop 2024-04-25T01:18:14.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.928+0000 7fdc90000640 1 -- 172.21.15.90:0/944481452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc8815bdd0 msgr2=0x7fdc8815c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:14.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.928+0000 7fdc90000640 1 --2- 172.21.15.90:0/944481452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc8815bdd0 0x7fdc8815c250 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fdc8815e650 tx=0x7fdc8000c980 comp rx=0 tx=0).stop 2024-04-25T01:18:14.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.928+0000 7fdc90000640 1 -- 172.21.15.90:0/944481452 shutdown_connections 2024-04-25T01:18:14.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.928+0000 7fdc90000640 1 --2- 172.21.15.90:0/944481452 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fdc6806be30 0x7fdc6806e2f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:14.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.928+0000 7fdc90000640 1 --2- 172.21.15.90:0/944481452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc8815bdd0 0x7fdc8815c250 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:14.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.928+0000 7fdc90000640 1 --2- 172.21.15.90:0/944481452 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fdc880cc540 0x7fdc880cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:14.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.928+0000 7fdc90000640 1 --2- 172.21.15.90:0/944481452 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc700039e0 0x7fdc88140e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:14.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.928+0000 7fdc90000640 1 -- 172.21.15.90:0/944481452 >> 172.21.15.90:0/944481452 conn(0x7fdc880b1820 msgr2=0x7fdc880af8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:14.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.928+0000 7fdc90000640 1 -- 172.21.15.90:0/944481452 shutdown_connections 2024-04-25T01:18:14.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:14.928+0000 7fdc90000640 1 -- 172.21.15.90:0/944481452 wait complete. 2024-04-25T01:18:14.928 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-04-25T01:18:15.155 INFO:teuthology.orchestra.run.smithi090.stdout:{"message":"","return_code":0} 2024-04-25T01:18:15.156 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-04-25T01:18:15.156 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T01:18:15.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.365+0000 7f1cf6c00640 1 Processor -- start 2024-04-25T01:18:15.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.365+0000 7f1cf6c00640 1 -- start start 2024-04-25T01:18:15.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.367+0000 7f1cf6c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1cf015e250 0x7f1cf00d25f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:15.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.367+0000 7f1cf6c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1cf00b2db0 0x7f1cf006c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:15.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.367+0000 7f1cf6c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cf006d1c0 con 0x7f1cf006c980 2024-04-25T01:18:15.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.367+0000 7f1cf6c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f1cf00d0ef0 con 0x7f1cf015e250 2024-04-25T01:18:15.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.367+0000 7f1cf6c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1cf00d08a0 con 0x7f1cf00b2db0 2024-04-25T01:18:15.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.367+0000 7f1cf5800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1cf015e250 0x7f1cf00d25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:15.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.367+0000 7f1cf5800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1cf015e250 0x7f1cf00d25f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40850/0 (socket says 172.21.15.90:40850) 2024-04-25T01:18:15.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.367+0000 7f1cf5800640 1 -- 172.21.15.90:0/1322122681 learned_addr learned my addr 172.21.15.90:0/1322122681 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:15.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.367+0000 7f1cf5800640 1 -- 172.21.15.90:0/1322122681 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1cf00b2db0 msgr2=0x7f1cf006c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:15.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.367+0000 7f1cf5800640 1 --2- 172.21.15.90:0/1322122681 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1cf00b2db0 0x7f1cf006c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:15.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.367+0000 7f1cf5800640 1 -- 172.21.15.90:0/1322122681 >> v1:172.21.15.103:6789/0 conn(0x7f1cf006c980 legacy=0x7f1cf00ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:15.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.367+0000 7f1cf5800640 1 -- 172.21.15.90:0/1322122681 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cf00d0b80 con 0x7f1cf015e250 2024-04-25T01:18:15.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.368+0000 7f1cf5800640 1 --2- 172.21.15.90:0/1322122681 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cf015e250 0x7f1cf00d25f0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f1ce4009870 tx=0x7f1ce402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=489f59b808fa6847 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:15.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.369+0000 7f1cefe00640 1 -- 172.21.15.90:0/1322122681 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce4037070 con 0x7f1cf015e250 2024-04-25T01:18:15.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.369+0000 7f1cefe00640 1 -- 172.21.15.90:0/1322122681 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ce4032440 con 0x7f1cf015e250 2024-04-25T01:18:15.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.369+0000 7f1cefe00640 1 -- 172.21.15.90:0/1322122681 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce4032810 con 0x7f1cf015e250 2024-04-25T01:18:15.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.369+0000 7f1cf6c00640 1 -- 172.21.15.90:0/1322122681 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cf015e250 msgr2=0x7f1cf00d25f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:15.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.369+0000 7f1cf6c00640 1 --2- 172.21.15.90:0/1322122681 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cf015e250 0x7f1cf00d25f0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f1ce4009870 tx=0x7f1ce402a180 comp rx=0 tx=0).stop 2024-04-25T01:18:15.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.369+0000 7f1cf6c00640 1 -- 172.21.15.90:0/1322122681 shutdown_connections 2024-04-25T01:18:15.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.369+0000 7f1cf6c00640 1 --2- 172.21.15.90:0/1322122681 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1cf00b2db0 0x7f1cf006c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:15.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.369+0000 7f1cf6c00640 1 --2- 172.21.15.90:0/1322122681 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cf015e250 0x7f1cf00d25f0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:15.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.369+0000 7f1cf6c00640 1 -- 172.21.15.90:0/1322122681 >> 172.21.15.90:0/1322122681 conn(0x7f1cf00d1ad0 msgr2=0x7f1cf00d1ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:15.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.370+0000 7f1cf6c00640 1 -- 172.21.15.90:0/1322122681 shutdown_connections 2024-04-25T01:18:15.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.370+0000 7f1cf6c00640 1 -- 172.21.15.90:0/1322122681 wait complete. 2024-04-25T01:18:15.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.372+0000 7f1cf6c00640 1 Processor -- start 2024-04-25T01:18:15.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.372+0000 7f1cf6c00640 1 -- start start 2024-04-25T01:18:15.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.373+0000 7f1cf6c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cf006c980 0x7f1cf00b5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:15.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.373+0000 7f1cf6c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cf00b2db0 0x7f1cf014eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:15.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.373+0000 7f1cf6c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1cf015e250 0x7f1cf00cb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:15.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.373+0000 7f1cf6c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf015e7b0 con 0x7f1cf006c980 2024-04-25T01:18:15.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.373+0000 7f1cf6c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf015e630 con 0x7f1cf00b2db0 2024-04-25T01:18:15.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.373+0000 7f1cf6c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1cf006ce10 con 0x7f1cf015e250 2024-04-25T01:18:15.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.373+0000 7f1cf5800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cf006c980 0x7f1cf00b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:15.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.373+0000 7f1cf5800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cf006c980 0x7f1cf00b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40856/0 (socket says 172.21.15.90:40856) 2024-04-25T01:18:15.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.373+0000 7f1cf5800640 1 -- 172.21.15.90:0/3822288168 learned_addr learned my addr 172.21.15.90:0/3822288168 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:15.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.374+0000 7f1cf5800640 1 -- 172.21.15.90:0/3822288168 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1cf015e250 msgr2=0x7f1cf00cb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:15.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.374+0000 7f1cf5800640 1 --2- 172.21.15.90:0/3822288168 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1cf015e250 0x7f1cf00cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:15.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.374+0000 7f1cf5800640 1 -- 172.21.15.90:0/3822288168 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cf00b2db0 msgr2=0x7f1cf014eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:15.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.374+0000 7f1cf5800640 1 --2- 172.21.15.90:0/3822288168 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cf00b2db0 0x7f1cf014eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:15.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.374+0000 7f1cf5800640 1 -- 172.21.15.90:0/3822288168 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cf00b5c20 con 0x7f1cf006c980 2024-04-25T01:18:15.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.374+0000 7f1cf5800640 1 --2- 172.21.15.90:0/3822288168 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cf006c980 0x7f1cf00b5640 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f1ce4009870 tx=0x7f1ce40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:15.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.374+0000 7f1ced600640 1 -- 172.21.15.90:0/3822288168 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce4041070 con 0x7f1cf006c980 2024-04-25T01:18:15.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.374+0000 7f1ced600640 1 -- 172.21.15.90:0/3822288168 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ce403c870 con 0x7f1cf006c980 2024-04-25T01:18:15.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.374+0000 7f1ced600640 1 -- 172.21.15.90:0/3822288168 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce4037040 con 0x7f1cf006c980 2024-04-25T01:18:15.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.374+0000 7f1cf6c00640 1 -- 172.21.15.90:0/3822288168 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cf00b5f00 con 0x7f1cf006c980 2024-04-25T01:18:15.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.374+0000 7f1cf6c00640 1 -- 172.21.15.90:0/3822288168 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cf00b64c0 con 0x7f1cf006c980 2024-04-25T01:18:15.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.376+0000 7f1cf6c00640 1 -- 172.21.15.90:0/3822288168 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cf0075350 con 0x7f1cf006c980 2024-04-25T01:18:15.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.376+0000 7f1ced600640 1 -- 172.21.15.90:0/3822288168 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1ce4044070 con 0x7f1cf006c980 2024-04-25T01:18:15.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.377+0000 7f1ced600640 1 --2- 172.21.15.90:0/3822288168 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1cd006be60 0x7f1cd006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:15.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.377+0000 7f1ced600640 1 -- 172.21.15.90:0/3822288168 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6159+0+0 (secure 0 0 0) 0x7f1ce40abb70 con 0x7f1cf006c980 2024-04-25T01:18:15.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.378+0000 7f1cf4e00640 1 --2- 172.21.15.90:0/3822288168 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1cd006be60 0x7f1cd006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:15.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.378+0000 7f1cf4e00640 1 --2- 172.21.15.90:0/3822288168 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1cd006be60 0x7f1cd006e320 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1cf006c950 tx=0x7f1cdc006c10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:15.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.384+0000 7f1ced600640 1 -- 172.21.15.90:0/3822288168 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1ce407a350 con 0x7f1cf006c980 2024-04-25T01:18:15.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.600+0000 7f1cf6c00640 1 -- 172.21.15.90:0/3822288168 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f1cf00c6650 con 0x7f1cf006c980 2024-04-25T01:18:15.585 INFO:tasks.ceph.mon.a.smithi090.stderr:2024-04-25T01:18:15.602+0000 7f0aa5400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T01:18:15.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.606+0000 7f1ced600640 1 -- 172.21.15.90:0/3822288168 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v19) ==== 161+0+0 (secure 0 0 0) 0x7f1ce4079c50 con 0x7f1cf006c980 2024-04-25T01:18:15.589 INFO:teuthology.orchestra.run.smithi090.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T01:18:15.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.609+0000 7f1cf6c00640 1 -- 172.21.15.90:0/3822288168 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1cd006be60 msgr2=0x7f1cd006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:15.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.609+0000 7f1cf6c00640 1 --2- 172.21.15.90:0/3822288168 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1cd006be60 0x7f1cd006e320 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1cf006c950 tx=0x7f1cdc006c10 comp rx=0 tx=0).stop 2024-04-25T01:18:15.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.609+0000 7f1cf6c00640 1 -- 172.21.15.90:0/3822288168 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cf006c980 msgr2=0x7f1cf00b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:15.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.609+0000 7f1cf6c00640 1 --2- 172.21.15.90:0/3822288168 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cf006c980 0x7f1cf00b5640 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f1ce4009870 tx=0x7f1ce40046c0 comp rx=0 tx=0).stop 2024-04-25T01:18:15.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.609+0000 7f1cf6c00640 1 -- 172.21.15.90:0/3822288168 shutdown_connections 2024-04-25T01:18:15.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.609+0000 7f1cf6c00640 1 --2- 172.21.15.90:0/3822288168 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1cd006be60 0x7f1cd006e320 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:15.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.609+0000 7f1cf6c00640 1 --2- 172.21.15.90:0/3822288168 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1cf015e250 0x7f1cf00cb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:15.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.609+0000 7f1cf6c00640 1 --2- 172.21.15.90:0/3822288168 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cf00b2db0 0x7f1cf014eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:15.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.609+0000 7f1cf6c00640 1 --2- 172.21.15.90:0/3822288168 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cf006c980 0x7f1cf00b5640 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:15.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.609+0000 7f1cf6c00640 1 -- 172.21.15.90:0/3822288168 >> 172.21.15.90:0/3822288168 conn(0x7f1cf00d1ad0 msgr2=0x7f1cf00c6b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:15.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.609+0000 7f1cf6c00640 1 -- 172.21.15.90:0/3822288168 shutdown_connections 2024-04-25T01:18:15.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:15.609+0000 7f1cf6c00640 1 -- 172.21.15.90:0/3822288168 wait complete. 2024-04-25T01:18:15.611 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T01:18:15.616 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T01:18:15.616 INFO:tasks.ceph.mds.c.smithi090.stderr: -4520> 2024-04-25T01:18:06.191+0000 7f41d2a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T01:18:15.689 INFO:tasks.ceph.mds.c.smithi090.stdout:starting mds.c at 2024-04-25T01:18:15.724 INFO:teuthology.orchestra.run.smithi090.stdout:[global] 2024-04-25T01:18:15.725 INFO:teuthology.orchestra.run.smithi090.stdout: chdir = "" 2024-04-25T01:18:15.725 INFO:teuthology.orchestra.run.smithi090.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T01:18:15.725 INFO:teuthology.orchestra.run.smithi090.stdout: auth supported = cephx 2024-04-25T01:18:15.725 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.725 INFO:teuthology.orchestra.run.smithi090.stdout: filestore xattr use omap = true 2024-04-25T01:18:15.725 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.725 INFO:teuthology.orchestra.run.smithi090.stdout: mon clock drift allowed = 1.000 2024-04-25T01:18:15.725 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.725 INFO:teuthology.orchestra.run.smithi090.stdout: osd crush chooseleaf type = 0 2024-04-25T01:18:15.725 INFO:teuthology.orchestra.run.smithi090.stdout: auth debug = true 2024-04-25T01:18:15.725 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.725 INFO:teuthology.orchestra.run.smithi090.stdout: ms die on old message = true 2024-04-25T01:18:15.725 INFO:teuthology.orchestra.run.smithi090.stdout: ms die on bug = true 2024-04-25T01:18:15.725 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T01:18:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: mon pg warn max object skew = 0 2024-04-25T01:18:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T01:18:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T01:18:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: osd pool default size = 2 2024-04-25T01:18:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd allow primary affinity = true 2024-04-25T01:18:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd allow pg remap = true 2024-04-25T01:18:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on legacy crush tunables = false 2024-04-25T01:18:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on crush straw calc version zero = false 2024-04-25T01:18:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on no sortbitwise = false 2024-04-25T01:18:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on osd down out interval zero = false 2024-04-25T01:18:15.727 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on too few osds = false 2024-04-25T01:18:15.727 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T01:18:15.727 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T01:18:15.727 INFO:teuthology.orchestra.run.smithi090.stdout: mon_allow_pool_size_one = true 2024-04-25T01:18:15.727 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.727 INFO:teuthology.orchestra.run.smithi090.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T01:18:15.727 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.727 INFO:teuthology.orchestra.run.smithi090.stdout: osd default data pool replay window = 5 2024-04-25T01:18:15.727 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.727 INFO:teuthology.orchestra.run.smithi090.stdout: mon allow pool delete = true 2024-04-25T01:18:15.727 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.727 INFO:teuthology.orchestra.run.smithi090.stdout: mon cluster log file level = debug 2024-04-25T01:18:15.727 INFO:teuthology.orchestra.run.smithi090.stdout: debug asserts on shutdown = true 2024-04-25T01:18:15.727 INFO:teuthology.orchestra.run.smithi090.stdout: mon health detail to clog = false 2024-04-25T01:18:15.727 INFO:teuthology.orchestra.run.smithi090.stdout: mon host = "172.21.15.90,172.21.15.103,[v2:172.21.15.103:3301,v1:172.21.15.103:6790]" 2024-04-25T01:18:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: ms die on skipped message = False 2024-04-25T01:18:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: fsid = e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21 2024-04-25T01:18:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.728 INFO:teuthology.orchestra.run.smithi090.stdout:[osd] 2024-04-25T01:18:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: osd journal size = 100 2024-04-25T01:18:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: osd scrub load threshold = 5.0 2024-04-25T01:18:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: osd scrub max interval = 600 2024-04-25T01:18:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: osd mclock profile = high_recovery_ops 2024-04-25T01:18:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: osd recover clone overlap = true 2024-04-25T01:18:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: osd recovery max chunk = 1048576 2024-04-25T01:18:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug shutdown = true 2024-04-25T01:18:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug op order = true 2024-04-25T01:18:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug verify stray on activate = true 2024-04-25T01:18:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug trim objects = true 2024-04-25T01:18:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: osd open classes on start = true 2024-04-25T01:18:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug pg log writeout = true 2024-04-25T01:18:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: osd deep scrub update digest min age = 30 2024-04-25T01:18:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: osd map max advance = 10 2024-04-25T01:18:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: journal zero on create = true 2024-04-25T01:18:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: filestore ondisk finisher threads = 3 2024-04-25T01:18:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: filestore apply finisher threads = 3 2024-04-25T01:18:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: bdev debug aio = true 2024-04-25T01:18:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug misdirected ops = true 2024-04-25T01:18:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: bluestore block size = 96636764160 2024-04-25T01:18:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: bluestore fsck on mount = True 2024-04-25T01:18:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: debug bluefs = 20 2024-04-25T01:18:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: debug bluestore = 20 2024-04-25T01:18:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-04-25T01:18:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: debug osd = 20 2024-04-25T01:18:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: debug rocksdb = 10 2024-04-25T01:18:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T01:18:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd full ratio = 0.9 2024-04-25T01:18:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd nearfull ratio = 0.8 2024-04-25T01:18:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: osd failsafe full ratio = 0.95 2024-04-25T01:18:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: osd objectstore = bluestore 2024-04-25T01:18:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: osd op complaint time = 180 2024-04-25T01:18:15.731 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.731 INFO:teuthology.orchestra.run.smithi090.stdout:[mgr] 2024-04-25T01:18:15.731 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-04-25T01:18:15.731 INFO:teuthology.orchestra.run.smithi090.stdout: debug mgr = 20 2024-04-25T01:18:15.731 INFO:teuthology.orchestra.run.smithi090.stdout: debug mon = 20 2024-04-25T01:18:15.731 INFO:teuthology.orchestra.run.smithi090.stdout: debug auth = 20 2024-04-25T01:18:15.731 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min pgs per osd = 4 2024-04-25T01:18:15.731 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min bytes per osd = 10 2024-04-25T01:18:15.731 INFO:teuthology.orchestra.run.smithi090.stdout: mgr/telemetry/nag = false 2024-04-25T01:18:15.731 INFO:teuthology.orchestra.run.smithi090.stdout: debug client = 20 2024-04-25T01:18:15.731 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.731 INFO:teuthology.orchestra.run.smithi090.stdout:[mon] 2024-04-25T01:18:15.731 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-04-25T01:18:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: debug mon = 20 2024-04-25T01:18:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: debug paxos = 20 2024-04-25T01:18:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: debug auth = 20 2024-04-25T01:18:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: mon data avail warn = 5 2024-04-25T01:18:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: mon mgr mkfs grace = 240 2024-04-25T01:18:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min pgs per osd = 4 2024-04-25T01:18:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd reporter subtree level = osd 2024-04-25T01:18:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd prime pg temp = true 2024-04-25T01:18:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min bytes per osd = 10 2024-04-25T01:18:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T01:18:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T01:18:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T01:18:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T01:18:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T01:18:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T01:18:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: # 1m isn't quite enough 2024-04-25T01:18:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: mon_down_mkfs_grace = 2m 2024-04-25T01:18:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_filestore_osds = false 2024-04-25T01:18:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: mon down mkfs grace = 300 2024-04-25T01:18:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: mon op complaint time = 120 2024-04-25T01:18:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.733 INFO:teuthology.orchestra.run.smithi090.stdout:[client] 2024-04-25T01:18:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: rgw cache enabled = true 2024-04-25T01:18:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: rgw enable ops log = true 2024-04-25T01:18:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: rgw enable usage log = true 2024-04-25T01:18:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T01:18:15.734 INFO:teuthology.orchestra.run.smithi090.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T01:18:15.734 INFO:teuthology.orchestra.run.smithi090.stdout: client mount timeout = 600 2024-04-25T01:18:15.734 INFO:teuthology.orchestra.run.smithi090.stdout: debug client = 20 2024-04-25T01:18:15.734 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-04-25T01:18:15.734 INFO:teuthology.orchestra.run.smithi090.stdout: pid file = "" 2024-04-25T01:18:15.734 INFO:teuthology.orchestra.run.smithi090.stdout: rados mon op timeout = 900 2024-04-25T01:18:15.734 INFO:teuthology.orchestra.run.smithi090.stdout: rados osd op timeout = 900 2024-04-25T01:18:15.734 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.a] 2024-04-25T01:18:15.734 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.b] 2024-04-25T01:18:15.734 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.c] 2024-04-25T01:18:15.734 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.a] 2024-04-25T01:18:15.734 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.c] 2024-04-25T01:18:15.734 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.b] 2024-04-25T01:18:15.734 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.d] 2024-04-25T01:18:15.735 INFO:teuthology.orchestra.run.smithi090.stdout:[mds] 2024-04-25T01:18:15.735 INFO:teuthology.orchestra.run.smithi090.stdout: debug mds = 20 2024-04-25T01:18:15.735 INFO:teuthology.orchestra.run.smithi090.stdout: debug mds balancer = 20 2024-04-25T01:18:15.735 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-04-25T01:18:15.735 INFO:teuthology.orchestra.run.smithi090.stdout: mds debug frag = True 2024-04-25T01:18:15.735 INFO:teuthology.orchestra.run.smithi090.stdout: mds debug scatterstat = True 2024-04-25T01:18:15.735 INFO:teuthology.orchestra.run.smithi090.stdout: mds op complaint time = 180 2024-04-25T01:18:15.735 INFO:teuthology.orchestra.run.smithi090.stdout: mds verify scatter = False 2024-04-25T01:18:15.735 INFO:teuthology.orchestra.run.smithi090.stdout: osd op complaint time = 180 2024-04-25T01:18:15.735 INFO:teuthology.orchestra.run.smithi090.stdout: rados mon op timeout = 900 2024-04-25T01:18:15.735 INFO:teuthology.orchestra.run.smithi090.stdout: rados osd op timeout = 900 2024-04-25T01:18:15.759 INFO:teuthology.orchestra.run.smithi103.stdout:[global] 2024-04-25T01:18:15.759 INFO:teuthology.orchestra.run.smithi103.stdout: chdir = "" 2024-04-25T01:18:15.760 INFO:teuthology.orchestra.run.smithi103.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T01:18:15.760 INFO:teuthology.orchestra.run.smithi103.stdout: auth supported = cephx 2024-04-25T01:18:15.760 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.760 INFO:teuthology.orchestra.run.smithi103.stdout: filestore xattr use omap = true 2024-04-25T01:18:15.760 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.760 INFO:teuthology.orchestra.run.smithi103.stdout: mon clock drift allowed = 1.000 2024-04-25T01:18:15.760 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.760 INFO:teuthology.orchestra.run.smithi103.stdout: osd crush chooseleaf type = 0 2024-04-25T01:18:15.760 INFO:teuthology.orchestra.run.smithi103.stdout: auth debug = true 2024-04-25T01:18:15.760 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.760 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on old message = true 2024-04-25T01:18:15.760 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on bug = true 2024-04-25T01:18:15.760 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.760 INFO:teuthology.orchestra.run.smithi103.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T01:18:15.760 INFO:teuthology.orchestra.run.smithi103.stdout: mon pg warn max object skew = 0 2024-04-25T01:18:15.761 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.761 INFO:teuthology.orchestra.run.smithi103.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T01:18:15.761 INFO:teuthology.orchestra.run.smithi103.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T01:18:15.761 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.761 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default size = 2 2024-04-25T01:18:15.761 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.761 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd allow primary affinity = true 2024-04-25T01:18:15.761 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd allow pg remap = true 2024-04-25T01:18:15.761 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on legacy crush tunables = false 2024-04-25T01:18:15.761 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on crush straw calc version zero = false 2024-04-25T01:18:15.761 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on no sortbitwise = false 2024-04-25T01:18:15.761 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on osd down out interval zero = false 2024-04-25T01:18:15.761 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on too few osds = false 2024-04-25T01:18:15.761 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T01:18:15.761 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T01:18:15.762 INFO:teuthology.orchestra.run.smithi103.stdout: mon_allow_pool_size_one = true 2024-04-25T01:18:15.762 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.762 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T01:18:15.762 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.762 INFO:teuthology.orchestra.run.smithi103.stdout: osd default data pool replay window = 5 2024-04-25T01:18:15.762 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.762 INFO:teuthology.orchestra.run.smithi103.stdout: mon allow pool delete = true 2024-04-25T01:18:15.762 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.762 INFO:teuthology.orchestra.run.smithi103.stdout: mon cluster log file level = debug 2024-04-25T01:18:15.762 INFO:teuthology.orchestra.run.smithi103.stdout: debug asserts on shutdown = true 2024-04-25T01:18:15.762 INFO:teuthology.orchestra.run.smithi103.stdout: mon health detail to clog = false 2024-04-25T01:18:15.762 INFO:teuthology.orchestra.run.smithi103.stdout: mon host = "172.21.15.90,172.21.15.103,[v2:172.21.15.103:3301,v1:172.21.15.103:6790]" 2024-04-25T01:18:15.762 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on skipped message = False 2024-04-25T01:18:15.762 INFO:teuthology.orchestra.run.smithi103.stdout: fsid = e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21 2024-04-25T01:18:15.762 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.763 INFO:teuthology.orchestra.run.smithi103.stdout:[osd] 2024-04-25T01:18:15.763 INFO:teuthology.orchestra.run.smithi103.stdout: osd journal size = 100 2024-04-25T01:18:15.763 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.763 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub load threshold = 5.0 2024-04-25T01:18:15.763 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub max interval = 600 2024-04-25T01:18:15.763 INFO:teuthology.orchestra.run.smithi103.stdout: osd mclock profile = high_recovery_ops 2024-04-25T01:18:15.763 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.763 INFO:teuthology.orchestra.run.smithi103.stdout: osd recover clone overlap = true 2024-04-25T01:18:15.763 INFO:teuthology.orchestra.run.smithi103.stdout: osd recovery max chunk = 1048576 2024-04-25T01:18:15.763 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.763 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug shutdown = true 2024-04-25T01:18:15.763 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug op order = true 2024-04-25T01:18:15.763 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug verify stray on activate = true 2024-04-25T01:18:15.763 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug trim objects = true 2024-04-25T01:18:15.763 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.764 INFO:teuthology.orchestra.run.smithi103.stdout: osd open classes on start = true 2024-04-25T01:18:15.764 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug pg log writeout = true 2024-04-25T01:18:15.764 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.764 INFO:teuthology.orchestra.run.smithi103.stdout: osd deep scrub update digest min age = 30 2024-04-25T01:18:15.764 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.764 INFO:teuthology.orchestra.run.smithi103.stdout: osd map max advance = 10 2024-04-25T01:18:15.764 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.764 INFO:teuthology.orchestra.run.smithi103.stdout: journal zero on create = true 2024-04-25T01:18:15.764 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.764 INFO:teuthology.orchestra.run.smithi103.stdout: filestore ondisk finisher threads = 3 2024-04-25T01:18:15.764 INFO:teuthology.orchestra.run.smithi103.stdout: filestore apply finisher threads = 3 2024-04-25T01:18:15.764 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.764 INFO:teuthology.orchestra.run.smithi103.stdout: bdev debug aio = true 2024-04-25T01:18:15.764 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug misdirected ops = true 2024-04-25T01:18:15.764 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore block size = 96636764160 2024-04-25T01:18:15.764 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore fsck on mount = True 2024-04-25T01:18:15.765 INFO:teuthology.orchestra.run.smithi103.stdout: debug bluefs = 20 2024-04-25T01:18:15.765 INFO:teuthology.orchestra.run.smithi103.stdout: debug bluestore = 20 2024-04-25T01:18:15.765 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-25T01:18:15.765 INFO:teuthology.orchestra.run.smithi103.stdout: debug osd = 20 2024-04-25T01:18:15.765 INFO:teuthology.orchestra.run.smithi103.stdout: debug rocksdb = 10 2024-04-25T01:18:15.765 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T01:18:15.765 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd full ratio = 0.9 2024-04-25T01:18:15.765 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd nearfull ratio = 0.8 2024-04-25T01:18:15.765 INFO:teuthology.orchestra.run.smithi103.stdout: osd failsafe full ratio = 0.95 2024-04-25T01:18:15.765 INFO:teuthology.orchestra.run.smithi103.stdout: osd objectstore = bluestore 2024-04-25T01:18:15.765 INFO:teuthology.orchestra.run.smithi103.stdout: osd op complaint time = 180 2024-04-25T01:18:15.765 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.765 INFO:teuthology.orchestra.run.smithi103.stdout:[mgr] 2024-04-25T01:18:15.765 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-25T01:18:15.765 INFO:teuthology.orchestra.run.smithi103.stdout: debug mgr = 20 2024-04-25T01:18:15.766 INFO:teuthology.orchestra.run.smithi103.stdout: debug mon = 20 2024-04-25T01:18:15.766 INFO:teuthology.orchestra.run.smithi103.stdout: debug auth = 20 2024-04-25T01:18:15.766 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min pgs per osd = 4 2024-04-25T01:18:15.766 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min bytes per osd = 10 2024-04-25T01:18:15.766 INFO:teuthology.orchestra.run.smithi103.stdout: mgr/telemetry/nag = false 2024-04-25T01:18:15.766 INFO:teuthology.orchestra.run.smithi103.stdout: debug client = 20 2024-04-25T01:18:15.766 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.766 INFO:teuthology.orchestra.run.smithi103.stdout:[mon] 2024-04-25T01:18:15.766 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-25T01:18:15.766 INFO:teuthology.orchestra.run.smithi103.stdout: debug mon = 20 2024-04-25T01:18:15.766 INFO:teuthology.orchestra.run.smithi103.stdout: debug paxos = 20 2024-04-25T01:18:15.766 INFO:teuthology.orchestra.run.smithi103.stdout: debug auth = 20 2024-04-25T01:18:15.766 INFO:teuthology.orchestra.run.smithi103.stdout: mon data avail warn = 5 2024-04-25T01:18:15.766 INFO:teuthology.orchestra.run.smithi103.stdout: mon mgr mkfs grace = 240 2024-04-25T01:18:15.766 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min pgs per osd = 4 2024-04-25T01:18:15.766 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd reporter subtree level = osd 2024-04-25T01:18:15.767 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd prime pg temp = true 2024-04-25T01:18:15.767 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min bytes per osd = 10 2024-04-25T01:18:15.767 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.767 INFO:teuthology.orchestra.run.smithi103.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T01:18:15.767 INFO:teuthology.orchestra.run.smithi103.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T01:18:15.767 INFO:teuthology.orchestra.run.smithi103.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T01:18:15.767 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.767 INFO:teuthology.orchestra.run.smithi103.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T01:18:15.767 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T01:18:15.767 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T01:18:15.767 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.767 INFO:teuthology.orchestra.run.smithi103.stdout: # 1m isn't quite enough 2024-04-25T01:18:15.767 INFO:teuthology.orchestra.run.smithi103.stdout: mon_down_mkfs_grace = 2m 2024-04-25T01:18:15.767 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.767 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_filestore_osds = false 2024-04-25T01:18:15.768 INFO:teuthology.orchestra.run.smithi103.stdout: mon down mkfs grace = 300 2024-04-25T01:18:15.768 INFO:teuthology.orchestra.run.smithi103.stdout: mon op complaint time = 120 2024-04-25T01:18:15.768 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.768 INFO:teuthology.orchestra.run.smithi103.stdout:[client] 2024-04-25T01:18:15.768 INFO:teuthology.orchestra.run.smithi103.stdout: rgw cache enabled = true 2024-04-25T01:18:15.768 INFO:teuthology.orchestra.run.smithi103.stdout: rgw enable ops log = true 2024-04-25T01:18:15.768 INFO:teuthology.orchestra.run.smithi103.stdout: rgw enable usage log = true 2024-04-25T01:18:15.768 INFO:teuthology.orchestra.run.smithi103.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T01:18:15.768 INFO:teuthology.orchestra.run.smithi103.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T01:18:15.768 INFO:teuthology.orchestra.run.smithi103.stdout: client mount timeout = 600 2024-04-25T01:18:15.768 INFO:teuthology.orchestra.run.smithi103.stdout: debug client = 20 2024-04-25T01:18:15.768 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-25T01:18:15.768 INFO:teuthology.orchestra.run.smithi103.stdout: pid file = "" 2024-04-25T01:18:15.768 INFO:teuthology.orchestra.run.smithi103.stdout: rados mon op timeout = 900 2024-04-25T01:18:15.768 INFO:teuthology.orchestra.run.smithi103.stdout: rados osd op timeout = 900 2024-04-25T01:18:15.768 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.a] 2024-04-25T01:18:15.769 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.b] 2024-04-25T01:18:15.769 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.c] 2024-04-25T01:18:15.769 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.a] 2024-04-25T01:18:15.769 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.c] 2024-04-25T01:18:15.769 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.b] 2024-04-25T01:18:15.769 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.d] 2024-04-25T01:18:15.769 INFO:teuthology.orchestra.run.smithi103.stdout:[mds] 2024-04-25T01:18:15.769 INFO:teuthology.orchestra.run.smithi103.stdout: debug mds = 20 2024-04-25T01:18:15.769 INFO:teuthology.orchestra.run.smithi103.stdout: debug mds balancer = 20 2024-04-25T01:18:15.769 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-25T01:18:15.769 INFO:teuthology.orchestra.run.smithi103.stdout: mds debug frag = True 2024-04-25T01:18:15.769 INFO:teuthology.orchestra.run.smithi103.stdout: mds debug scatterstat = True 2024-04-25T01:18:15.769 INFO:teuthology.orchestra.run.smithi103.stdout: mds op complaint time = 180 2024-04-25T01:18:15.769 INFO:teuthology.orchestra.run.smithi103.stdout: mds verify scatter = False 2024-04-25T01:18:15.769 INFO:teuthology.orchestra.run.smithi103.stdout: osd op complaint time = 180 2024-04-25T01:18:15.769 INFO:teuthology.orchestra.run.smithi103.stdout: rados mon op timeout = 900 2024-04-25T01:18:15.770 INFO:teuthology.orchestra.run.smithi103.stdout: rados osd op timeout = 900 2024-04-25T01:18:15.793 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T01:18:15.800 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T01:18:15.913 INFO:teuthology.orchestra.run.smithi090.stdout:[global] 2024-04-25T01:18:15.913 INFO:teuthology.orchestra.run.smithi090.stdout: chdir = "" 2024-04-25T01:18:15.913 INFO:teuthology.orchestra.run.smithi090.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T01:18:15.913 INFO:teuthology.orchestra.run.smithi090.stdout: auth supported = cephx 2024-04-25T01:18:15.914 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.914 INFO:teuthology.orchestra.run.smithi090.stdout: filestore xattr use omap = true 2024-04-25T01:18:15.914 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.914 INFO:teuthology.orchestra.run.smithi090.stdout: mon clock drift allowed = 1.000 2024-04-25T01:18:15.914 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.914 INFO:teuthology.orchestra.run.smithi090.stdout: osd crush chooseleaf type = 0 2024-04-25T01:18:15.914 INFO:teuthology.orchestra.run.smithi090.stdout: auth debug = true 2024-04-25T01:18:15.914 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.914 INFO:teuthology.orchestra.run.smithi090.stdout: ms die on old message = true 2024-04-25T01:18:15.914 INFO:teuthology.orchestra.run.smithi090.stdout: ms die on bug = true 2024-04-25T01:18:15.914 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.914 INFO:teuthology.orchestra.run.smithi090.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T01:18:15.914 INFO:teuthology.orchestra.run.smithi090.stdout: mon pg warn max object skew = 0 2024-04-25T01:18:15.914 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.915 INFO:teuthology.orchestra.run.smithi090.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T01:18:15.915 INFO:teuthology.orchestra.run.smithi090.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T01:18:15.915 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.915 INFO:teuthology.orchestra.run.smithi090.stdout: osd pool default size = 2 2024-04-25T01:18:15.915 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.915 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd allow primary affinity = true 2024-04-25T01:18:15.915 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd allow pg remap = true 2024-04-25T01:18:15.915 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on legacy crush tunables = false 2024-04-25T01:18:15.915 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on crush straw calc version zero = false 2024-04-25T01:18:15.915 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on no sortbitwise = false 2024-04-25T01:18:15.915 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on osd down out interval zero = false 2024-04-25T01:18:15.915 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on too few osds = false 2024-04-25T01:18:15.915 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T01:18:15.915 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T01:18:15.915 INFO:teuthology.orchestra.run.smithi090.stdout: mon_allow_pool_size_one = true 2024-04-25T01:18:15.915 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.916 INFO:teuthology.orchestra.run.smithi090.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T01:18:15.916 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.916 INFO:teuthology.orchestra.run.smithi090.stdout: osd default data pool replay window = 5 2024-04-25T01:18:15.916 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.916 INFO:teuthology.orchestra.run.smithi090.stdout: mon allow pool delete = true 2024-04-25T01:18:15.916 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.916 INFO:teuthology.orchestra.run.smithi090.stdout: mon cluster log file level = debug 2024-04-25T01:18:15.916 INFO:teuthology.orchestra.run.smithi090.stdout: debug asserts on shutdown = true 2024-04-25T01:18:15.916 INFO:teuthology.orchestra.run.smithi090.stdout: mon health detail to clog = false 2024-04-25T01:18:15.916 INFO:teuthology.orchestra.run.smithi090.stdout: mon host = "172.21.15.90,172.21.15.103,[v2:172.21.15.103:3301,v1:172.21.15.103:6790]" 2024-04-25T01:18:15.916 INFO:teuthology.orchestra.run.smithi090.stdout: ms die on skipped message = False 2024-04-25T01:18:15.916 INFO:teuthology.orchestra.run.smithi090.stdout: fsid = e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21 2024-04-25T01:18:15.916 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.916 INFO:teuthology.orchestra.run.smithi090.stdout:[osd] 2024-04-25T01:18:15.916 INFO:teuthology.orchestra.run.smithi090.stdout: osd journal size = 100 2024-04-25T01:18:15.916 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.917 INFO:teuthology.orchestra.run.smithi090.stdout: osd scrub load threshold = 5.0 2024-04-25T01:18:15.917 INFO:teuthology.orchestra.run.smithi090.stdout: osd scrub max interval = 600 2024-04-25T01:18:15.917 INFO:teuthology.orchestra.run.smithi090.stdout: osd mclock profile = high_recovery_ops 2024-04-25T01:18:15.917 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.917 INFO:teuthology.orchestra.run.smithi090.stdout: osd recover clone overlap = true 2024-04-25T01:18:15.917 INFO:teuthology.orchestra.run.smithi090.stdout: osd recovery max chunk = 1048576 2024-04-25T01:18:15.917 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.917 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug shutdown = true 2024-04-25T01:18:15.917 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug op order = true 2024-04-25T01:18:15.917 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug verify stray on activate = true 2024-04-25T01:18:15.917 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug trim objects = true 2024-04-25T01:18:15.917 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.917 INFO:teuthology.orchestra.run.smithi090.stdout: osd open classes on start = true 2024-04-25T01:18:15.917 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug pg log writeout = true 2024-04-25T01:18:15.917 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.918 INFO:teuthology.orchestra.run.smithi090.stdout: osd deep scrub update digest min age = 30 2024-04-25T01:18:15.918 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.918 INFO:teuthology.orchestra.run.smithi090.stdout: osd map max advance = 10 2024-04-25T01:18:15.918 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.918 INFO:teuthology.orchestra.run.smithi090.stdout: journal zero on create = true 2024-04-25T01:18:15.918 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.918 INFO:teuthology.orchestra.run.smithi090.stdout: filestore ondisk finisher threads = 3 2024-04-25T01:18:15.918 INFO:teuthology.orchestra.run.smithi090.stdout: filestore apply finisher threads = 3 2024-04-25T01:18:15.918 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.918 INFO:teuthology.orchestra.run.smithi090.stdout: bdev debug aio = true 2024-04-25T01:18:15.918 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug misdirected ops = true 2024-04-25T01:18:15.918 INFO:teuthology.orchestra.run.smithi090.stdout: bluestore block size = 96636764160 2024-04-25T01:18:15.918 INFO:teuthology.orchestra.run.smithi090.stdout: bluestore fsck on mount = True 2024-04-25T01:18:15.918 INFO:teuthology.orchestra.run.smithi090.stdout: debug bluefs = 20 2024-04-25T01:18:15.918 INFO:teuthology.orchestra.run.smithi090.stdout: debug bluestore = 20 2024-04-25T01:18:15.919 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-04-25T01:18:15.919 INFO:teuthology.orchestra.run.smithi090.stdout: debug osd = 20 2024-04-25T01:18:15.919 INFO:teuthology.orchestra.run.smithi090.stdout: debug rocksdb = 10 2024-04-25T01:18:15.919 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T01:18:15.919 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd full ratio = 0.9 2024-04-25T01:18:15.919 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd nearfull ratio = 0.8 2024-04-25T01:18:15.919 INFO:teuthology.orchestra.run.smithi090.stdout: osd failsafe full ratio = 0.95 2024-04-25T01:18:15.919 INFO:teuthology.orchestra.run.smithi090.stdout: osd objectstore = bluestore 2024-04-25T01:18:15.919 INFO:teuthology.orchestra.run.smithi090.stdout: osd op complaint time = 180 2024-04-25T01:18:15.919 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.919 INFO:teuthology.orchestra.run.smithi090.stdout:[mgr] 2024-04-25T01:18:15.919 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-04-25T01:18:15.919 INFO:teuthology.orchestra.run.smithi090.stdout: debug mgr = 20 2024-04-25T01:18:15.919 INFO:teuthology.orchestra.run.smithi090.stdout: debug mon = 20 2024-04-25T01:18:15.919 INFO:teuthology.orchestra.run.smithi090.stdout: debug auth = 20 2024-04-25T01:18:15.919 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min pgs per osd = 4 2024-04-25T01:18:15.920 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min bytes per osd = 10 2024-04-25T01:18:15.920 INFO:teuthology.orchestra.run.smithi090.stdout: mgr/telemetry/nag = false 2024-04-25T01:18:15.920 INFO:teuthology.orchestra.run.smithi090.stdout: debug client = 20 2024-04-25T01:18:15.920 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.920 INFO:teuthology.orchestra.run.smithi090.stdout:[mon] 2024-04-25T01:18:15.920 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-04-25T01:18:15.920 INFO:teuthology.orchestra.run.smithi090.stdout: debug mon = 20 2024-04-25T01:18:15.920 INFO:teuthology.orchestra.run.smithi090.stdout: debug paxos = 20 2024-04-25T01:18:15.920 INFO:teuthology.orchestra.run.smithi090.stdout: debug auth = 20 2024-04-25T01:18:15.920 INFO:teuthology.orchestra.run.smithi090.stdout: mon data avail warn = 5 2024-04-25T01:18:15.920 INFO:teuthology.orchestra.run.smithi090.stdout: mon mgr mkfs grace = 240 2024-04-25T01:18:15.920 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min pgs per osd = 4 2024-04-25T01:18:15.920 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd reporter subtree level = osd 2024-04-25T01:18:15.920 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd prime pg temp = true 2024-04-25T01:18:15.920 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min bytes per osd = 10 2024-04-25T01:18:15.921 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.921 INFO:teuthology.orchestra.run.smithi090.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T01:18:15.921 INFO:teuthology.orchestra.run.smithi090.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T01:18:15.921 INFO:teuthology.orchestra.run.smithi090.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T01:18:15.921 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.921 INFO:teuthology.orchestra.run.smithi090.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T01:18:15.921 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T01:18:15.921 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T01:18:15.921 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.921 INFO:teuthology.orchestra.run.smithi090.stdout: # 1m isn't quite enough 2024-04-25T01:18:15.921 INFO:teuthology.orchestra.run.smithi090.stdout: mon_down_mkfs_grace = 2m 2024-04-25T01:18:15.921 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.921 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_filestore_osds = false 2024-04-25T01:18:15.921 INFO:teuthology.orchestra.run.smithi090.stdout: mon down mkfs grace = 300 2024-04-25T01:18:15.921 INFO:teuthology.orchestra.run.smithi090.stdout: mon op complaint time = 120 2024-04-25T01:18:15.921 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:15.922 INFO:teuthology.orchestra.run.smithi090.stdout:[client] 2024-04-25T01:18:15.922 INFO:teuthology.orchestra.run.smithi090.stdout: rgw cache enabled = true 2024-04-25T01:18:15.922 INFO:teuthology.orchestra.run.smithi090.stdout: rgw enable ops log = true 2024-04-25T01:18:15.922 INFO:teuthology.orchestra.run.smithi090.stdout: rgw enable usage log = true 2024-04-25T01:18:15.922 INFO:teuthology.orchestra.run.smithi090.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T01:18:15.922 INFO:teuthology.orchestra.run.smithi090.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T01:18:15.922 INFO:teuthology.orchestra.run.smithi090.stdout: client mount timeout = 600 2024-04-25T01:18:15.922 INFO:teuthology.orchestra.run.smithi090.stdout: debug client = 20 2024-04-25T01:18:15.922 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-04-25T01:18:15.922 INFO:teuthology.orchestra.run.smithi090.stdout: pid file = "" 2024-04-25T01:18:15.922 INFO:teuthology.orchestra.run.smithi090.stdout: rados mon op timeout = 900 2024-04-25T01:18:15.922 INFO:teuthology.orchestra.run.smithi090.stdout: rados osd op timeout = 900 2024-04-25T01:18:15.922 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.a] 2024-04-25T01:18:15.922 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.b] 2024-04-25T01:18:15.923 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.c] 2024-04-25T01:18:15.923 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.a] 2024-04-25T01:18:15.923 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.c] 2024-04-25T01:18:15.923 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.b] 2024-04-25T01:18:15.923 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.d] 2024-04-25T01:18:15.923 INFO:teuthology.orchestra.run.smithi090.stdout:[mds] 2024-04-25T01:18:15.923 INFO:teuthology.orchestra.run.smithi090.stdout: debug mds = 20 2024-04-25T01:18:15.923 INFO:teuthology.orchestra.run.smithi090.stdout: debug mds balancer = 20 2024-04-25T01:18:15.923 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-04-25T01:18:15.923 INFO:teuthology.orchestra.run.smithi090.stdout: mds debug frag = True 2024-04-25T01:18:15.923 INFO:teuthology.orchestra.run.smithi090.stdout: mds debug scatterstat = False 2024-04-25T01:18:15.923 INFO:teuthology.orchestra.run.smithi090.stdout: mds op complaint time = 180 2024-04-25T01:18:15.923 INFO:teuthology.orchestra.run.smithi090.stdout: mds verify scatter = False 2024-04-25T01:18:15.923 INFO:teuthology.orchestra.run.smithi090.stdout: osd op complaint time = 180 2024-04-25T01:18:15.923 INFO:teuthology.orchestra.run.smithi090.stdout: rados mon op timeout = 900 2024-04-25T01:18:15.923 INFO:teuthology.orchestra.run.smithi090.stdout: rados osd op timeout = 900 2024-04-25T01:18:15.937 INFO:teuthology.orchestra.run.smithi103.stdout:[global] 2024-04-25T01:18:15.937 INFO:teuthology.orchestra.run.smithi103.stdout: chdir = "" 2024-04-25T01:18:15.937 INFO:teuthology.orchestra.run.smithi103.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T01:18:15.937 INFO:teuthology.orchestra.run.smithi103.stdout: auth supported = cephx 2024-04-25T01:18:15.937 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.937 INFO:teuthology.orchestra.run.smithi103.stdout: filestore xattr use omap = true 2024-04-25T01:18:15.937 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.937 INFO:teuthology.orchestra.run.smithi103.stdout: mon clock drift allowed = 1.000 2024-04-25T01:18:15.937 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.937 INFO:teuthology.orchestra.run.smithi103.stdout: osd crush chooseleaf type = 0 2024-04-25T01:18:15.937 INFO:teuthology.orchestra.run.smithi103.stdout: auth debug = true 2024-04-25T01:18:15.937 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.937 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on old message = true 2024-04-25T01:18:15.938 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on bug = true 2024-04-25T01:18:15.938 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.938 INFO:teuthology.orchestra.run.smithi103.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T01:18:15.938 INFO:teuthology.orchestra.run.smithi103.stdout: mon pg warn max object skew = 0 2024-04-25T01:18:15.938 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.938 INFO:teuthology.orchestra.run.smithi103.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T01:18:15.938 INFO:teuthology.orchestra.run.smithi103.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T01:18:15.938 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.938 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default size = 2 2024-04-25T01:18:15.938 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.938 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd allow primary affinity = true 2024-04-25T01:18:15.938 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd allow pg remap = true 2024-04-25T01:18:15.938 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on legacy crush tunables = false 2024-04-25T01:18:15.938 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on crush straw calc version zero = false 2024-04-25T01:18:15.939 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on no sortbitwise = false 2024-04-25T01:18:15.939 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on osd down out interval zero = false 2024-04-25T01:18:15.939 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on too few osds = false 2024-04-25T01:18:15.939 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T01:18:15.939 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T01:18:15.939 INFO:teuthology.orchestra.run.smithi103.stdout: mon_allow_pool_size_one = true 2024-04-25T01:18:15.939 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.939 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T01:18:15.939 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.939 INFO:teuthology.orchestra.run.smithi103.stdout: osd default data pool replay window = 5 2024-04-25T01:18:15.939 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.939 INFO:teuthology.orchestra.run.smithi103.stdout: mon allow pool delete = true 2024-04-25T01:18:15.939 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.939 INFO:teuthology.orchestra.run.smithi103.stdout: mon cluster log file level = debug 2024-04-25T01:18:15.939 INFO:teuthology.orchestra.run.smithi103.stdout: debug asserts on shutdown = true 2024-04-25T01:18:15.939 INFO:teuthology.orchestra.run.smithi103.stdout: mon health detail to clog = false 2024-04-25T01:18:15.940 INFO:teuthology.orchestra.run.smithi103.stdout: mon host = "172.21.15.90,172.21.15.103,[v2:172.21.15.103:3301,v1:172.21.15.103:6790]" 2024-04-25T01:18:15.940 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on skipped message = False 2024-04-25T01:18:15.940 INFO:teuthology.orchestra.run.smithi103.stdout: fsid = e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21 2024-04-25T01:18:15.940 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.940 INFO:teuthology.orchestra.run.smithi103.stdout:[osd] 2024-04-25T01:18:15.940 INFO:teuthology.orchestra.run.smithi103.stdout: osd journal size = 100 2024-04-25T01:18:15.940 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.940 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub load threshold = 5.0 2024-04-25T01:18:15.940 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub max interval = 600 2024-04-25T01:18:15.940 INFO:teuthology.orchestra.run.smithi103.stdout: osd mclock profile = high_recovery_ops 2024-04-25T01:18:15.940 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.940 INFO:teuthology.orchestra.run.smithi103.stdout: osd recover clone overlap = true 2024-04-25T01:18:15.940 INFO:teuthology.orchestra.run.smithi103.stdout: osd recovery max chunk = 1048576 2024-04-25T01:18:15.940 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.940 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug shutdown = true 2024-04-25T01:18:15.941 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug op order = true 2024-04-25T01:18:15.941 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug verify stray on activate = true 2024-04-25T01:18:15.941 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug trim objects = true 2024-04-25T01:18:15.941 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.941 INFO:teuthology.orchestra.run.smithi103.stdout: osd open classes on start = true 2024-04-25T01:18:15.941 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug pg log writeout = true 2024-04-25T01:18:15.941 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.941 INFO:teuthology.orchestra.run.smithi103.stdout: osd deep scrub update digest min age = 30 2024-04-25T01:18:15.941 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.941 INFO:teuthology.orchestra.run.smithi103.stdout: osd map max advance = 10 2024-04-25T01:18:15.941 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.941 INFO:teuthology.orchestra.run.smithi103.stdout: journal zero on create = true 2024-04-25T01:18:15.941 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.941 INFO:teuthology.orchestra.run.smithi103.stdout: filestore ondisk finisher threads = 3 2024-04-25T01:18:15.941 INFO:teuthology.orchestra.run.smithi103.stdout: filestore apply finisher threads = 3 2024-04-25T01:18:15.942 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.942 INFO:teuthology.orchestra.run.smithi103.stdout: bdev debug aio = true 2024-04-25T01:18:15.942 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug misdirected ops = true 2024-04-25T01:18:15.942 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore block size = 96636764160 2024-04-25T01:18:15.942 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore fsck on mount = True 2024-04-25T01:18:15.942 INFO:teuthology.orchestra.run.smithi103.stdout: debug bluefs = 20 2024-04-25T01:18:15.942 INFO:teuthology.orchestra.run.smithi103.stdout: debug bluestore = 20 2024-04-25T01:18:15.942 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-25T01:18:15.942 INFO:teuthology.orchestra.run.smithi103.stdout: debug osd = 20 2024-04-25T01:18:15.942 INFO:teuthology.orchestra.run.smithi103.stdout: debug rocksdb = 10 2024-04-25T01:18:15.942 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T01:18:15.942 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd full ratio = 0.9 2024-04-25T01:18:15.942 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd nearfull ratio = 0.8 2024-04-25T01:18:15.942 INFO:teuthology.orchestra.run.smithi103.stdout: osd failsafe full ratio = 0.95 2024-04-25T01:18:15.942 INFO:teuthology.orchestra.run.smithi103.stdout: osd objectstore = bluestore 2024-04-25T01:18:15.943 INFO:teuthology.orchestra.run.smithi103.stdout: osd op complaint time = 180 2024-04-25T01:18:15.943 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.943 INFO:teuthology.orchestra.run.smithi103.stdout:[mgr] 2024-04-25T01:18:15.943 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-25T01:18:15.943 INFO:teuthology.orchestra.run.smithi103.stdout: debug mgr = 20 2024-04-25T01:18:15.943 INFO:teuthology.orchestra.run.smithi103.stdout: debug mon = 20 2024-04-25T01:18:15.943 INFO:teuthology.orchestra.run.smithi103.stdout: debug auth = 20 2024-04-25T01:18:15.943 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min pgs per osd = 4 2024-04-25T01:18:15.943 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min bytes per osd = 10 2024-04-25T01:18:15.943 INFO:teuthology.orchestra.run.smithi103.stdout: mgr/telemetry/nag = false 2024-04-25T01:18:15.943 INFO:teuthology.orchestra.run.smithi103.stdout: debug client = 20 2024-04-25T01:18:15.943 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.943 INFO:teuthology.orchestra.run.smithi103.stdout:[mon] 2024-04-25T01:18:15.943 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-25T01:18:15.943 INFO:teuthology.orchestra.run.smithi103.stdout: debug mon = 20 2024-04-25T01:18:15.944 INFO:teuthology.orchestra.run.smithi103.stdout: debug paxos = 20 2024-04-25T01:18:15.944 INFO:teuthology.orchestra.run.smithi103.stdout: debug auth = 20 2024-04-25T01:18:15.944 INFO:teuthology.orchestra.run.smithi103.stdout: mon data avail warn = 5 2024-04-25T01:18:15.944 INFO:teuthology.orchestra.run.smithi103.stdout: mon mgr mkfs grace = 240 2024-04-25T01:18:15.944 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min pgs per osd = 4 2024-04-25T01:18:15.944 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd reporter subtree level = osd 2024-04-25T01:18:15.944 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd prime pg temp = true 2024-04-25T01:18:15.944 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min bytes per osd = 10 2024-04-25T01:18:15.944 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.944 INFO:teuthology.orchestra.run.smithi103.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T01:18:15.944 INFO:teuthology.orchestra.run.smithi103.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T01:18:15.944 INFO:teuthology.orchestra.run.smithi103.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T01:18:15.944 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.944 INFO:teuthology.orchestra.run.smithi103.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T01:18:15.944 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T01:18:15.945 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T01:18:15.945 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.945 INFO:teuthology.orchestra.run.smithi103.stdout: # 1m isn't quite enough 2024-04-25T01:18:15.945 INFO:teuthology.orchestra.run.smithi103.stdout: mon_down_mkfs_grace = 2m 2024-04-25T01:18:15.945 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.945 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_filestore_osds = false 2024-04-25T01:18:15.945 INFO:teuthology.orchestra.run.smithi103.stdout: mon down mkfs grace = 300 2024-04-25T01:18:15.945 INFO:teuthology.orchestra.run.smithi103.stdout: mon op complaint time = 120 2024-04-25T01:18:15.945 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-25T01:18:15.945 INFO:teuthology.orchestra.run.smithi103.stdout:[client] 2024-04-25T01:18:15.945 INFO:teuthology.orchestra.run.smithi103.stdout: rgw cache enabled = true 2024-04-25T01:18:15.945 INFO:teuthology.orchestra.run.smithi103.stdout: rgw enable ops log = true 2024-04-25T01:18:15.945 INFO:teuthology.orchestra.run.smithi103.stdout: rgw enable usage log = true 2024-04-25T01:18:15.945 INFO:teuthology.orchestra.run.smithi103.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T01:18:15.945 INFO:teuthology.orchestra.run.smithi103.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T01:18:15.946 INFO:teuthology.orchestra.run.smithi103.stdout: client mount timeout = 600 2024-04-25T01:18:15.946 INFO:teuthology.orchestra.run.smithi103.stdout: debug client = 20 2024-04-25T01:18:15.946 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-25T01:18:15.946 INFO:teuthology.orchestra.run.smithi103.stdout: pid file = "" 2024-04-25T01:18:15.946 INFO:teuthology.orchestra.run.smithi103.stdout: rados mon op timeout = 900 2024-04-25T01:18:15.946 INFO:teuthology.orchestra.run.smithi103.stdout: rados osd op timeout = 900 2024-04-25T01:18:15.946 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.a] 2024-04-25T01:18:15.946 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.b] 2024-04-25T01:18:15.946 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.c] 2024-04-25T01:18:15.946 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.a] 2024-04-25T01:18:15.946 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.c] 2024-04-25T01:18:15.946 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.b] 2024-04-25T01:18:15.946 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.d] 2024-04-25T01:18:15.946 INFO:teuthology.orchestra.run.smithi103.stdout:[mds] 2024-04-25T01:18:15.946 INFO:teuthology.orchestra.run.smithi103.stdout: debug mds = 20 2024-04-25T01:18:15.947 INFO:teuthology.orchestra.run.smithi103.stdout: debug mds balancer = 20 2024-04-25T01:18:15.947 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-25T01:18:15.947 INFO:teuthology.orchestra.run.smithi103.stdout: mds debug frag = True 2024-04-25T01:18:15.947 INFO:teuthology.orchestra.run.smithi103.stdout: mds debug scatterstat = False 2024-04-25T01:18:15.947 INFO:teuthology.orchestra.run.smithi103.stdout: mds op complaint time = 180 2024-04-25T01:18:15.947 INFO:teuthology.orchestra.run.smithi103.stdout: mds verify scatter = False 2024-04-25T01:18:15.947 INFO:teuthology.orchestra.run.smithi103.stdout: osd op complaint time = 180 2024-04-25T01:18:15.947 INFO:teuthology.orchestra.run.smithi103.stdout: rados mon op timeout = 900 2024-04-25T01:18:15.947 INFO:teuthology.orchestra.run.smithi103.stdout: rados osd op timeout = 900 2024-04-25T01:18:15.971 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T01:18:15.972 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-25T01:18:15.972 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T01:18:15.972 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:18:15.972 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T01:18:15.972 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-25T01:18:15.972 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T01:18:15.972 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:18:15.972 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T01:18:15.973 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-25T01:18:15.973 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T01:18:15.973 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:18:15.973 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T01:18:15.973 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-25T01:18:15.973 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T01:18:15.973 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:18:15.991 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T01:18:15.991 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T01:18:15.994 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T01:18:15.994 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T01:18:15.994 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T01:18:15.995 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T01:18:15.996 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T01:18:15.996 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T01:18:16.038 INFO:tasks.ceph.mds.a:Started 2024-04-25T01:18:16.038 DEBUG:teuthology.parallel:result is None 2024-04-25T01:18:16.041 INFO:tasks.ceph.mds.b:Started 2024-04-25T01:18:16.041 DEBUG:teuthology.parallel:result is None 2024-04-25T01:18:16.044 INFO:tasks.ceph.mds.c:Started 2024-04-25T01:18:16.044 DEBUG:teuthology.parallel:result is None 2024-04-25T01:18:16.046 INFO:tasks.ceph.mds.d:Started 2024-04-25T01:18:16.046 DEBUG:teuthology.parallel:result is None 2024-04-25T01:18:16.047 DEBUG:teuthology.orchestra.run.smithi090:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph purge cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T01:18:16.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.145+0000 7f996fa34f40 1 Processor -- start 2024-04-25T01:18:16.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.145+0000 7f996fa34f40 1 -- start start 2024-04-25T01:18:16.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.146+0000 7f996fa34f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f31c45210 0x562f31c3b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.146+0000 7f996fa34f40 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562f31ccf180 0x562f31cb83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.146+0000 7f996fa34f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562f31ccf560 0x562f31bfb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.146+0000 7f996fa34f40 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x562f31c438f0 con 0x562f31ccf180 2024-04-25T01:18:16.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.146+0000 7f996fa34f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562f31ba1640 con 0x562f31c45210 2024-04-25T01:18:16.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.146+0000 7f996fa34f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x562f31c43b90 con 0x562f31ccf560 2024-04-25T01:18:16.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.147+0000 7f996cc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f31c45210 0x562f31c3b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:16.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.147+0000 7f9967e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562f31ccf180 0x562f31cb83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:16.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.147+0000 7f996cc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f31c45210 0x562f31c3b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.90:57804/0 (socket says 172.21.15.90:57804) 2024-04-25T01:18:16.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.147+0000 7f996cc00640 1 -- 172.21.15.90:0/43961431 learned_addr learned my addr 172.21.15.90:0/43961431 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:16.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.147+0000 7f996d600640 1 --2- 172.21.15.90:0/43961431 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562f31ccf560 0x562f31bfb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:16.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.147+0000 7f996cc00640 1 -- 172.21.15.90:0/43961431 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562f31ccf560 msgr2=0x562f31bfb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:16.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.147+0000 7f996cc00640 1 --2- 172.21.15.90:0/43961431 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562f31ccf560 0x562f31bfb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.147+0000 7f996cc00640 1 -- 172.21.15.90:0/43961431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562f31ccf180 msgr2=0x562f31cb83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:16.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.147+0000 7f996cc00640 1 --2- 172.21.15.90:0/43961431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562f31ccf180 0x562f31cb83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.147+0000 7f996cc00640 1 -- 172.21.15.90:0/43961431 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f31b26b30 con 0x562f31c45210 2024-04-25T01:18:16.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.147+0000 7f996cc00640 1 --2- 172.21.15.90:0/43961431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f31c45210 0x562f31c3b040 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9960025d20 tx=0x7f9960051000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f900d4d9672ca4ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:16.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.148+0000 7f9966a00640 1 -- 172.21.15.90:0/43961431 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9960053780 con 0x562f31c45210 2024-04-25T01:18:16.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.148+0000 7f9966a00640 1 -- 172.21.15.90:0/43961431 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9960053940 con 0x562f31c45210 2024-04-25T01:18:16.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.148+0000 7f9966a00640 1 -- 172.21.15.90:0/43961431 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9960053d30 con 0x562f31c45210 2024-04-25T01:18:16.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.149+0000 7f996fa34f40 1 -- 172.21.15.90:0/43961431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f31c45210 msgr2=0x562f31c3b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:16.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.149+0000 7f996fa34f40 1 --2- 172.21.15.90:0/43961431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f31c45210 0x562f31c3b040 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9960025d20 tx=0x7f9960051000 comp rx=0 tx=0).stop 2024-04-25T01:18:16.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.149+0000 7f996fa34f40 1 -- 172.21.15.90:0/43961431 shutdown_connections 2024-04-25T01:18:16.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.149+0000 7f996fa34f40 1 --2- 172.21.15.90:0/43961431 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562f31ccf560 0x562f31bfb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.149+0000 7f996fa34f40 1 --2- 172.21.15.90:0/43961431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562f31ccf180 0x562f31cb83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.149+0000 7f996fa34f40 1 --2- 172.21.15.90:0/43961431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f31c45210 0x562f31c3b040 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.149+0000 7f996fa34f40 1 -- 172.21.15.90:0/43961431 >> 172.21.15.90:0/43961431 conn(0x562f31bf8b50 msgr2=0x562f31cd1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:16.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.149+0000 7f996fa34f40 1 -- 172.21.15.90:0/43961431 shutdown_connections 2024-04-25T01:18:16.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.149+0000 7f996fa34f40 1 -- 172.21.15.90:0/43961431 wait complete. 2024-04-25T01:18:16.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.150+0000 7f996fa34f40 1 Processor -- start 2024-04-25T01:18:16.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.150+0000 7f996fa34f40 1 -- start start 2024-04-25T01:18:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.152+0000 7f996fa34f40 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562f31c45210 0x562f31ccc950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.152+0000 7f996fa34f40 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f31ccf180 0x562f31cc8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.152+0000 7f996fa34f40 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562f31ccf560 0x562f31cc93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.152+0000 7f996fa34f40 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x562f31cc6040 con 0x562f31c45210 2024-04-25T01:18:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.152+0000 7f996fa34f40 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562f31cb8910 con 0x562f31ccf180 2024-04-25T01:18:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.152+0000 7f996fa34f40 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x562f31bf9280 con 0x562f31ccf560 2024-04-25T01:18:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.152+0000 7f996cc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562f31c45210 0x562f31ccc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.152+0000 7f996cc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562f31c45210 0x562f31ccc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40890/0 (socket says 172.21.15.90:40890) 2024-04-25T01:18:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.152+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 learned_addr learned my addr 172.21.15.90:0/1505982586 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.152+0000 7f9967e00640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f31ccf180 0x562f31cc8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.152+0000 7f996d600640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562f31ccf560 0x562f31cc93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.153+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562f31ccf560 msgr2=0x562f31cc93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:16.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.153+0000 7f996cc00640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562f31ccf560 0x562f31cc93e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.153+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f31ccf180 msgr2=0x562f31cc8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:16.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.153+0000 7f996cc00640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f31ccf180 0x562f31cc8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.153+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f31c408b0 con 0x562f31c45210 2024-04-25T01:18:16.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.153+0000 7f996d600640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562f31ccf560 0x562f31cc93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:16.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.153+0000 7f996cc00640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562f31c45210 0x562f31ccc950 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f9960025d20 tx=0x7f9960002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:16.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.153+0000 7f9965600640 1 -- 172.21.15.90:0/1505982586 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9960073020 con 0x562f31c45210 2024-04-25T01:18:16.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.153+0000 7f9965600640 1 -- 172.21.15.90:0/1505982586 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9960053af0 con 0x562f31c45210 2024-04-25T01:18:16.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.153+0000 7f9965600640 1 -- 172.21.15.90:0/1505982586 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9960067430 con 0x562f31c45210 2024-04-25T01:18:16.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.153+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x562f31c2a0f0 con 0x562f31c45210 2024-04-25T01:18:16.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.154+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f31c424a0 con 0x562f31c45210 2024-04-25T01:18:16.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.155+0000 7f9965600640 1 -- 172.21.15.90:0/1505982586 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9960076020 con 0x562f31c45210 2024-04-25T01:18:16.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.156+0000 7f9965600640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f994c06a810 0x7f994c06ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.156+0000 7f9965600640 1 -- 172.21.15.90:0/1505982586 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6245+0+0 (secure 0 0 0) 0x7f99600cfdc0 con 0x562f31c45210 2024-04-25T01:18:16.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.157+0000 7f9967e00640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f994c06a810 0x7f994c06ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:16.140 INFO:teuthology.orchestra.run.smithi090.stdout:Warning: using slow linear search 2024-04-25T01:18:16.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.157+0000 7f996fa34f40 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x562f31cb76c0 0x562f31d749c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.157+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:1 5.0 5:00000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x562f31d74f00 con 0x562f31cb76c0 2024-04-25T01:18:16.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.157+0000 7f996d600640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x562f31cb76c0 0x562f31d749c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:16.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.157+0000 7f9967e00640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f994c06a810 0x7f994c06ccd0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x562f31cce350 tx=0x7f995800c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:16.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.158+0000 7f996d600640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x562f31cb76c0 0x562f31d749c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:16.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.164+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 <== osd.4 v2:172.21.15.103:6821/1971061680 1 ==== osd_op_reply(1 [pgnls start_epoch 43 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9968002600 con 0x562f31cb76c0 2024-04-25T01:18:16.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.164+0000 7f996d600640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] conn(0x7f9968011830 0x7f99680102b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.164+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:2 5.10 5:08000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x562f31d74f00 con 0x7f9968011830 2024-04-25T01:18:16.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.165+0000 7f996cc00640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] conn(0x7f9968011830 0x7f99680102b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:16.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.165+0000 7f996cc00640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] conn(0x7f9968011830 0x7f99680102b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:16.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.168+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 <== osd.2 v2:172.21.15.90:6808/906792737 1 ==== osd_op_reply(2 [pgnls start_epoch 43 out=177b] v42'10 uv10 ondisk = 1) ==== 144+0+177 (crc 0 0 0) 0x7f996007f070 con 0x7f9968011830 2024-04-25T01:18:16.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.169+0000 7f996cc00640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x7f9960078020 0x7f99600a4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.169+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:3 5.8 5:10000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x562f31d74f00 con 0x7f9960078020 2024-04-25T01:18:16.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.169+0000 7f9967e00640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x7f9960078020 0x7f99600a4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:16.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.170+0000 7f9967e00640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x7f9960078020 0x7f99600a4310 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:16.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.171+0000 7f9967e00640 1 -- 172.21.15.90:0/1505982586 <== osd.6 v2:172.21.15.103:6816/2008388906 1 ==== osd_op_reply(3 [pgnls start_epoch 43 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99580087e0 con 0x7f9960078020 2024-04-25T01:18:16.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.171+0000 7f9967e00640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x7f9958016930 0x7f9958015380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.171+0000 7f9967e00640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:4 5.18 5:18000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x562f31d74f00 con 0x7f9958016930 2024-04-25T01:18:16.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.171+0000 7f996d600640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x7f9958016930 0x7f9958015380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:16.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.172+0000 7f996d600640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x7f9958016930 0x7f9958015380 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:16.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.172+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 <== osd.1 v2:172.21.15.90:6800/2685473653 1 ==== osd_op_reply(4 [pgnls start_epoch 43 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9968002600 con 0x7f9958016930 2024-04-25T01:18:16.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.172+0000 7f996d600640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x7f9968016040 0x7f9968017940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.172+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:5 5.4 5:20000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x562f31d74f00 con 0x7f9968016040 2024-04-25T01:18:16.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.173+0000 7f996cc00640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x7f9968016040 0x7f9968017940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:16.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.173+0000 7f996cc00640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x7f9968016040 0x7f9968017940 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:16.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.176+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 <== osd.0 v2:172.21.15.90:6816/2625481430 1 ==== osd_op_reply(5 [pgnls start_epoch 43 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f996007f070 con 0x7f9968016040 2024-04-25T01:18:16.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.176+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:6 5.14 5:28000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x562f31d74f00 con 0x562f31cb76c0 2024-04-25T01:18:16.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.177+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 <== osd.4 v2:172.21.15.103:6821/1971061680 2 ==== osd_op_reply(6 [pgnls start_epoch 43 out=149b] v42'19 uv19 ondisk = 1) ==== 144+0+149 (crc 0 0 0) 0x7f9968002600 con 0x562f31cb76c0 2024-04-25T01:18:16.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.177+0000 7f996d600640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x7f9968015920 0x7f99680143a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.177+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- osd_op(unknown.0.0:7 5.c 5:30000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x562f31d74f00 con 0x7f9968015920 2024-04-25T01:18:16.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.177+0000 7f9967e00640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x7f9968015920 0x7f99680143a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:16.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.178+0000 7f9967e00640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x7f9968015920 0x7f99680143a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:16.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.178+0000 7f9967e00640 1 -- 172.21.15.90:0/1505982586 <== osd.5 v2:172.21.15.103:6801/1348846168 1 ==== osd_op_reply(7 [pgnls start_epoch 43 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99580087e0 con 0x7f9968015920 2024-04-25T01:18:16.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.178+0000 7f9967e00640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:8 5.1c 5:38000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x562f31d74f00 con 0x7f9968011830 2024-04-25T01:18:16.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.179+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 <== osd.2 v2:172.21.15.90:6808/906792737 2 ==== osd_op_reply(8 [pgnls start_epoch 43 out=73b] v39'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f996007f070 con 0x7f9968011830 2024-04-25T01:18:16.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.179+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:9 5.2 5:40000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x562f31d74f00 con 0x7f9968016040 2024-04-25T01:18:16.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.179+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 <== osd.0 v2:172.21.15.90:6816/2625481430 2 ==== osd_op_reply(9 [pgnls start_epoch 43 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f996007f070 con 0x7f9968016040 2024-04-25T01:18:16.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.179+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:10 5.12 5:48000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x562f31d74f00 con 0x7f9960078020 2024-04-25T01:18:16.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.180+0000 7f9967e00640 1 -- 172.21.15.90:0/1505982586 <== osd.6 v2:172.21.15.103:6816/2008388906 2 ==== osd_op_reply(10 [pgnls start_epoch 43 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99580087e0 con 0x7f9960078020 2024-04-25T01:18:16.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.180+0000 7f9967e00640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:11 5.a 5:50000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x562f31d74f00 con 0x7f9960078020 2024-04-25T01:18:16.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.180+0000 7f9967e00640 1 -- 172.21.15.90:0/1505982586 <== osd.6 v2:172.21.15.103:6816/2008388906 3 ==== osd_op_reply(11 [pgnls start_epoch 43 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99580087e0 con 0x7f9960078020 2024-04-25T01:18:16.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.180+0000 7f9967e00640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:12 5.1a 5:58000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x562f31d74f00 con 0x7f9958016930 2024-04-25T01:18:16.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.181+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 <== osd.1 v2:172.21.15.90:6800/2685473653 2 ==== osd_op_reply(12 [pgnls start_epoch 43 out=49b] v39'1 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9968002600 con 0x7f9958016930 2024-04-25T01:18:16.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.181+0000 7f996d600640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x7f9968024370 0x7f99680247f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.181+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:13 5.6 5:60000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x7f9968025040 con 0x7f9968024370 2024-04-25T01:18:16.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.181+0000 7f996d600640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x7f9968024370 0x7f99680247f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:16.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.182+0000 7f996d600640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x7f9968024370 0x7f99680247f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:16.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.184+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 <== osd.7 v2:172.21.15.103:6800/1145679154 1 ==== osd_op_reply(13 [pgnls start_epoch 43 out=77b] v42'3 uv3 ondisk = 1) ==== 144+0+77 (crc 0 0 0) 0x7f9968002600 con 0x7f9968024370 2024-04-25T01:18:16.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.184+0000 7f996d600640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x7f99680273f0 0x7f9968027870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.184+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:14 5.16 5:68000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x7f9968025040 con 0x7f99680273f0 2024-04-25T01:18:16.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.185+0000 7f996cc00640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x7f99680273f0 0x7f9968027870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:16.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.185+0000 7f996cc00640 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x7f99680273f0 0x7f9968027870 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:16.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.193+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 <== osd.3 v2:172.21.15.90:6812/8683104 1 ==== osd_op_reply(14 [pgnls start_epoch 43 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99600c0040 con 0x7f99680273f0 2024-04-25T01:18:16.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.193+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:15 5.e 5:70000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x7f9968025040 con 0x7f99680273f0 2024-04-25T01:18:16.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.193+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 <== osd.3 v2:172.21.15.90:6812/8683104 2 ==== osd_op_reply(15 [pgnls start_epoch 43 out=73b] v39'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f99600c0040 con 0x7f99680273f0 2024-04-25T01:18:16.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.193+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:16 5.1e 5:78000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x7f9968025040 con 0x7f9960078020 2024-04-25T01:18:16.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.194+0000 7f9967e00640 1 -- 172.21.15.90:0/1505982586 <== osd.6 v2:172.21.15.103:6816/2008388906 4 ==== osd_op_reply(16 [pgnls start_epoch 43 out=49b] v39'1 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99580087e0 con 0x7f9960078020 2024-04-25T01:18:16.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.194+0000 7f9967e00640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:17 5.1 5:80000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x7f9968025040 con 0x7f99680273f0 2024-04-25T01:18:16.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.194+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 <== osd.3 v2:172.21.15.90:6812/8683104 3 ==== osd_op_reply(17 [pgnls start_epoch 43 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99600c0040 con 0x7f99680273f0 2024-04-25T01:18:16.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.194+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:18 5.11 5:88000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x7f9968025040 con 0x7f99680273f0 2024-04-25T01:18:16.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.195+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 <== osd.3 v2:172.21.15.90:6812/8683104 4 ==== osd_op_reply(18 [pgnls start_epoch 43 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99600c0040 con 0x7f99680273f0 2024-04-25T01:18:16.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.195+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:19 5.9 5:90000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x7f9968025040 con 0x7f9958016930 2024-04-25T01:18:16.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.195+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 <== osd.1 v2:172.21.15.90:6800/2685473653 3 ==== osd_op_reply(19 [pgnls start_epoch 43 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9968002600 con 0x7f9958016930 2024-04-25T01:18:16.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.195+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:20 5.19 5:98000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x7f9968025040 con 0x562f31cb76c0 2024-04-25T01:18:16.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.196+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 <== osd.4 v2:172.21.15.103:6821/1971061680 3 ==== osd_op_reply(20 [pgnls start_epoch 43 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9968002600 con 0x562f31cb76c0 2024-04-25T01:18:16.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.196+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:21 5.5 5:a0000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x7f9968025040 con 0x7f9968024370 2024-04-25T01:18:16.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.197+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 <== osd.7 v2:172.21.15.103:6800/1145679154 2 ==== osd_op_reply(21 [pgnls start_epoch 43 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9968002600 con 0x7f9968024370 2024-04-25T01:18:16.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.197+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:22 5.15 5:a8000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x7f9968025040 con 0x7f99680273f0 2024-04-25T01:18:16.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.197+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 <== osd.3 v2:172.21.15.90:6812/8683104 5 ==== osd_op_reply(22 [pgnls start_epoch 43 out=97b] v39'3 uv3 ondisk = 1) ==== 144+0+97 (crc 0 0 0) 0x7f99600c0040 con 0x7f99680273f0 2024-04-25T01:18:16.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.197+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:23 5.d 5:b0000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x7f9968025040 con 0x7f9968024370 2024-04-25T01:18:16.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.198+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 <== osd.7 v2:172.21.15.103:6800/1145679154 3 ==== osd_op_reply(23 [pgnls start_epoch 43 out=74b] v39'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f9968002600 con 0x7f9968024370 2024-04-25T01:18:16.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.198+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:24 5.1d 5:b8000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x7f9968025040 con 0x562f31cb76c0 2024-04-25T01:18:16.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.199+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 <== osd.4 v2:172.21.15.103:6821/1971061680 4 ==== osd_op_reply(24 [pgnls start_epoch 43 out=73b] v39'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f9968002600 con 0x562f31cb76c0 2024-04-25T01:18:16.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.199+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:25 5.3 5:c0000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x7f9968025040 con 0x562f31cb76c0 2024-04-25T01:18:16.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.199+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 <== osd.4 v2:172.21.15.103:6821/1971061680 5 ==== osd_op_reply(25 [pgnls start_epoch 43 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9968002600 con 0x562f31cb76c0 2024-04-25T01:18:16.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.199+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:26 5.13 5:c8000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x7f9968025040 con 0x562f31cb76c0 2024-04-25T01:18:16.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.200+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 <== osd.4 v2:172.21.15.103:6821/1971061680 6 ==== osd_op_reply(26 [pgnls start_epoch 43 out=98b] v42'3 uv3 ondisk = 1) ==== 144+0+98 (crc 0 0 0) 0x7f9968002600 con 0x562f31cb76c0 2024-04-25T01:18:16.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.200+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- osd_op(unknown.0.0:27 5.b 5:d0000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x7f9968025040 con 0x7f9968015920 2024-04-25T01:18:16.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.200+0000 7f9967e00640 1 -- 172.21.15.90:0/1505982586 <== osd.5 v2:172.21.15.103:6801/1348846168 2 ==== osd_op_reply(27 [pgnls start_epoch 43 out=76b] v42'3 uv3 ondisk = 1) ==== 144+0+76 (crc 0 0 0) 0x7f99580087e0 con 0x7f9968015920 2024-04-25T01:18:16.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.200+0000 7f9967e00640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:28 5.1b 5:d8000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x7f9968025040 con 0x7f9968011830 2024-04-25T01:18:16.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.201+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 <== osd.2 v2:172.21.15.90:6808/906792737 3 ==== osd_op_reply(28 [pgnls start_epoch 43 out=49b] v39'1 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99600c0040 con 0x7f9968011830 2024-04-25T01:18:16.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.201+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:29 5.7 5:e0000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x7f9968025040 con 0x7f99680273f0 2024-04-25T01:18:16.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.201+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 <== osd.3 v2:172.21.15.90:6812/8683104 6 ==== osd_op_reply(29 [pgnls start_epoch 43 out=73b] v39'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f99600c0040 con 0x7f99680273f0 2024-04-25T01:18:16.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.201+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:30 5.17 5:e8000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x7f9968025040 con 0x562f31cb76c0 2024-04-25T01:18:16.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.202+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 <== osd.4 v2:172.21.15.103:6821/1971061680 7 ==== osd_op_reply(30 [pgnls start_epoch 43 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9968002600 con 0x562f31cb76c0 2024-04-25T01:18:16.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.202+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:31 5.f 5:f0000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x7f9968025040 con 0x7f9958016930 2024-04-25T01:18:16.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.202+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 <== osd.1 v2:172.21.15.90:6800/2685473653 4 ==== osd_op_reply(31 [pgnls start_epoch 43 out=73b] v39'1 uv1 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f9968002600 con 0x7f9958016930 2024-04-25T01:18:16.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.202+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:32 5.1f 5:f8000000:%01:::head [pgnls start_epoch 43 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e43) -- 0x7f9968025040 con 0x7f9968011830 2024-04-25T01:18:16.186 INFO:tasks.ceph.mds.a.smithi090.stdout:starting mds.a at 2024-04-25T01:18:16.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.203+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 <== osd.2 v2:172.21.15.90:6808/906792737 4 ==== osd_op_reply(32 [pgnls start_epoch 43 out=101b] v42'6 uv6 ondisk = 1) ==== 144+0+101 (crc 0 0 0) 0x7f99600c0040 con 0x7f9968011830 2024-04-25T01:18:16.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.203+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:33 5.10 5:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e43) -- 0x562f31cc7cb0 con 0x7f9968011830 2024-04-25T01:18:16.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.203+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:34 5.10 5:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e43) -- 0x562f31d752a0 con 0x7f9968011830 2024-04-25T01:18:16.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.203+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:35 5.10 5:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e43) -- 0x562f31d75af0 con 0x7f9968011830 2024-04-25T01:18:16.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.203+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:36 5.10 5:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e43) -- 0x562f31d76340 con 0x7f9968011830 2024-04-25T01:18:16.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.203+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:37 5.10 5:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e43) -- 0x562f31d76b90 con 0x7f9968011830 2024-04-25T01:18:16.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.203+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:38 5.14 5:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e43) -- 0x562f31d773e0 con 0x562f31cb76c0 2024-04-25T01:18:16.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.203+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:39 5.14 5:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e43) -- 0x562f31d77c30 con 0x562f31cb76c0 2024-04-25T01:18:16.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.203+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:40 5.14 5:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e43) -- 0x562f31d78480 con 0x562f31cb76c0 2024-04-25T01:18:16.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.203+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:41 5.14 5:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e43) -- 0x562f31d78cd0 con 0x562f31cb76c0 2024-04-25T01:18:16.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.203+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:42 5.1c 5:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e43) -- 0x562f31d79520 con 0x7f9968011830 2024-04-25T01:18:16.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.203+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:43 5.6 5:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e43) -- 0x562f31d7a010 con 0x7f9968024370 2024-04-25T01:18:16.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.203+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:44 5.e 5:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e43) -- 0x562f31d7aac0 con 0x7f99680273f0 2024-04-25T01:18:16.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.203+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:45 5.15 5:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e43) -- 0x562f31d7b5b0 con 0x7f99680273f0 2024-04-25T01:18:16.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.203+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:46 5.15 5:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e43) -- 0x562f31d7c0f0 con 0x7f99680273f0 2024-04-25T01:18:16.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.203+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:47 5.d 5:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e43) -- 0x562f31d7cbb0 con 0x7f9968024370 2024-04-25T01:18:16.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.203+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:48 5.1d 5:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e43) -- 0x562f31d7d6a0 con 0x562f31cb76c0 2024-04-25T01:18:16.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.207+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 <== osd.3 v2:172.21.15.90:6812/8683104 7 ==== osd_op_reply(44 608.00000000 [delete] v43'3 uv3 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99600c0040 con 0x7f99680273f0 2024-04-25T01:18:16.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.207+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:49 5.13 5:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e43) -- 0x562f31d7a010 con 0x562f31cb76c0 2024-04-25T01:18:16.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.208+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 <== osd.2 v2:172.21.15.90:6808/906792737 5 ==== osd_op_reply(33 601.00000000 [delete] v43'11 uv11 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99600c0040 con 0x7f9968011830 2024-04-25T01:18:16.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.208+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 <== osd.7 v2:172.21.15.103:6800/1145679154 4 ==== osd_op_reply(47 mds_snaptable [delete] v43'2 uv2 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9968002600 con 0x7f9968024370 2024-04-25T01:18:16.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.208+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:50 5.13 5:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e43) -- 0x562f31d7a010 con 0x562f31cb76c0 2024-04-25T01:18:16.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.208+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- osd_op(unknown.0.0:51 5.b 5:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e43) -- 0x562f31d7b5b0 con 0x7f9968015920 2024-04-25T01:18:16.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.208+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 <== osd.2 v2:172.21.15.90:6808/906792737 6 ==== osd_op_reply(34 602.00000000 [delete] v43'12 uv12 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99600c0040 con 0x7f9968011830 2024-04-25T01:18:16.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.208+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 <== osd.7 v2:172.21.15.103:6800/1145679154 5 ==== osd_op_reply(43 mds0_openfiles.0 [delete] v43'4 uv4 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9968002600 con 0x7f9968024370 2024-04-25T01:18:16.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.208+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 <== osd.4 v2:172.21.15.103:6821/1971061680 8 ==== osd_op_reply(38 1.00000000.inode [delete] v43'20 uv20 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9968002600 con 0x562f31cb76c0 2024-04-25T01:18:16.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.208+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 <== osd.4 v2:172.21.15.103:6821/1971061680 9 ==== osd_op_reply(48 605.00000000 [delete] v43'3 uv3 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9968002600 con 0x562f31cb76c0 2024-04-25T01:18:16.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.208+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:52 5.7 5:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e43) -- 0x562f31d79ff0 con 0x7f99680273f0 2024-04-25T01:18:16.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.208+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:53 5.f 5:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e43) -- 0x562f31d7cbc0 con 0x7f9958016930 2024-04-25T01:18:16.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.208+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:54 5.1f 5:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e43) -- 0x562f31d799d0 con 0x7f9968011830 2024-04-25T01:18:16.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.208+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:55 5.1f 5:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e43) -- 0x562f31d7def0 con 0x7f9968011830 2024-04-25T01:18:16.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.209+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 <== osd.2 v2:172.21.15.90:6808/906792737 7 ==== osd_op_reply(35 10000000000.00000000 [delete] v43'13 uv13 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f99600c0040 con 0x7f9968011830 2024-04-25T01:18:16.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.209+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 <== osd.2 v2:172.21.15.90:6808/906792737 8 ==== osd_op_reply(36 600.00000000 [delete] v43'14 uv14 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99600c0040 con 0x7f9968011830 2024-04-25T01:18:16.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.209+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 <== osd.2 v2:172.21.15.90:6808/906792737 9 ==== osd_op_reply(37 603.00000000 [delete] v43'15 uv15 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99600c0040 con 0x7f9968011830 2024-04-25T01:18:16.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.209+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 <== osd.4 v2:172.21.15.103:6821/1971061680 10 ==== osd_op_reply(39 200.00000000 [delete] v43'21 uv21 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9968002600 con 0x562f31cb76c0 2024-04-25T01:18:16.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.209+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 <== osd.4 v2:172.21.15.103:6821/1971061680 11 ==== osd_op_reply(40 200.00000001 [delete] v43'22 uv22 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9968002600 con 0x562f31cb76c0 2024-04-25T01:18:16.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.209+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 <== osd.4 v2:172.21.15.103:6821/1971061680 12 ==== osd_op_reply(41 606.00000000 [delete] v43'23 uv23 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9968002600 con 0x562f31cb76c0 2024-04-25T01:18:16.193 INFO:tasks.ceph.mds.c.smithi090.stdout:starting mds.c at 2024-04-25T01:18:16.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.210+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 <== osd.2 v2:172.21.15.90:6808/906792737 10 ==== osd_op_reply(42 607.00000000 [delete] v43'3 uv3 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99600c0040 con 0x7f9968011830 2024-04-25T01:18:16.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.210+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 <== osd.3 v2:172.21.15.90:6812/8683104 8 ==== osd_op_reply(45 604.00000000 [delete] v43'4 uv4 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99600c0040 con 0x7f99680273f0 2024-04-25T01:18:16.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.210+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 <== osd.3 v2:172.21.15.90:6812/8683104 9 ==== osd_op_reply(46 500.00000000 [delete] v43'5 uv5 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99600c0040 con 0x7f99680273f0 2024-04-25T01:18:16.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.211+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 <== osd.4 v2:172.21.15.103:6821/1971061680 13 ==== osd_op_reply(49 mds0_inotable [delete] v43'4 uv4 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9968002600 con 0x562f31cb76c0 2024-04-25T01:18:16.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.211+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 <== osd.1 v2:172.21.15.90:6800/2685473653 5 ==== osd_op_reply(53 400.00000000 [delete] v43'2 uv2 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9968002600 con 0x7f9958016930 2024-04-25T01:18:16.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.212+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 <== osd.4 v2:172.21.15.103:6821/1971061680 14 ==== osd_op_reply(50 100.00000000 [delete] v43'5 uv5 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9968002600 con 0x562f31cb76c0 2024-04-25T01:18:16.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.213+0000 7f9967e00640 1 -- 172.21.15.90:0/1505982586 <== osd.5 v2:172.21.15.103:6801/1348846168 3 ==== osd_op_reply(51 mds0_sessionmap [delete] v43'4 uv4 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f99580087e0 con 0x7f9968015920 2024-04-25T01:18:16.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.213+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 <== osd.3 v2:172.21.15.90:6812/8683104 10 ==== osd_op_reply(52 609.00000000 [delete] v43'3 uv3 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99600c0040 con 0x7f99680273f0 2024-04-25T01:18:16.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.213+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 <== osd.2 v2:172.21.15.90:6808/906792737 11 ==== osd_op_reply(54 100.00000000.inode [delete] v43'7 uv7 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f99600c0040 con 0x7f9968011830 2024-04-25T01:18:16.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.213+0000 7f996cc00640 1 -- 172.21.15.90:0/1505982586 <== osd.2 v2:172.21.15.90:6808/906792737 12 ==== osd_op_reply(55 1.00000000 [delete] v43'8 uv8 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f99600c0040 con 0x7f9968011830 2024-04-25T01:18:16.196 INFO:teuthology.orchestra.run.smithi090.stdout:Removed 23 objects 2024-04-25T01:18:16.197 INFO:teuthology.orchestra.run.smithi090.stdout:successfully purged pool cephfs_metadata 2024-04-25T01:18:16.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.213+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x7f9968016040 msgr2=0x7f9968017940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:16.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.213+0000 7f996fa34f40 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x7f9968016040 0x7f9968017940 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.213+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x7f9958016930 msgr2=0x7f9958015380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:16.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.213+0000 7f996fa34f40 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x7f9958016930 0x7f9958015380 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.213+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 >> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] conn(0x7f9968011830 msgr2=0x7f99680102b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:16.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.213+0000 7f996fa34f40 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] conn(0x7f9968011830 0x7f99680102b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.213+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x7f99680273f0 msgr2=0x7f9968027870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:16.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.213+0000 7f996fa34f40 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x7f99680273f0 0x7f9968027870 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.213+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x562f31cb76c0 msgr2=0x562f31d749c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:16.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.213+0000 7f996fa34f40 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x562f31cb76c0 0x562f31d749c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.213+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x7f9968015920 msgr2=0x7f99680143a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:16.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.213+0000 7f996fa34f40 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x7f9968015920 0x7f99680143a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.213+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x7f9960078020 msgr2=0x7f99600a4310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:16.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.213+0000 7f996fa34f40 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x7f9960078020 0x7f99600a4310 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.213+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x7f9968024370 msgr2=0x7f99680247f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:16.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.213+0000 7f996fa34f40 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x7f9968024370 0x7f99680247f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.213+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f994c06a810 msgr2=0x7f994c06ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:16.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.213+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 reap_dead start 2024-04-25T01:18:16.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.214+0000 7f996fa34f40 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f994c06a810 0x7f994c06ccd0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x562f31cce350 tx=0x7f995800c040 comp rx=0 tx=0).stop 2024-04-25T01:18:16.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.214+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 reap_dead start 2024-04-25T01:18:16.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.214+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562f31c45210 msgr2=0x562f31ccc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:16.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.214+0000 7f996fa34f40 1 --2- 172.21.15.90:0/1505982586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562f31c45210 0x562f31ccc950 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f9960025d20 tx=0x7f9960002fb0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.214+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 reap_dead start 2024-04-25T01:18:16.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.214+0000 7f996d600640 1 -- 172.21.15.90:0/1505982586 reap_dead start 2024-04-25T01:18:16.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.214+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 shutdown_connections 2024-04-25T01:18:16.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.214+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 >> 172.21.15.90:0/1505982586 conn(0x562f31bf8b50 msgr2=0x562f31c3faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:16.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.214+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 shutdown_connections 2024-04-25T01:18:16.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.214+0000 7f996fa34f40 1 -- 172.21.15.90:0/1505982586 wait complete. 2024-04-25T01:18:16.200 INFO:tasks.ceph.mds.d.smithi103.stdout:starting mds.d at 2024-04-25T01:18:16.200 INFO:tasks.ceph.mds.b.smithi103.stdout:starting mds.b at 2024-04-25T01:18:16.202 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:18:16.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.388+0000 7f92ed200640 1 Processor -- start 2024-04-25T01:18:16.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.388+0000 7f92ed200640 1 -- start start 2024-04-25T01:18:16.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.389+0000 7f92ed200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92e80d0240 0x7f92e806d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.389+0000 7f92ed200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92e80ccb10 con 0x7f92e8071dd0 2024-04-25T01:18:16.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.389+0000 7f92ed200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92e806ddc0 con 0x7f92e815a980 2024-04-25T01:18:16.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.389+0000 7f92ed200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f92e8072260 con 0x7f92e80d0240 2024-04-25T01:18:16.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.390+0000 7f92e7e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f92e8071dd0 0x7f92e80ccfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53370/0 (socket says 172.21.15.90:53370) 2024-04-25T01:18:16.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.390+0000 7f92e7e00640 1 -- 172.21.15.90:0/1422531552 learned_addr learned my addr 172.21.15.90:0/1422531552 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:16.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.390+0000 7f92dfe00640 1 --2- 172.21.15.90:0/1422531552 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92e80d0240 0x7f92e806d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:16.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.390+0000 7f92e6a00640 1 -- 172.21.15.90:0/1422531552 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4222984004 0 0) 0x7f92e0001490 con 0x7f92e8071dd0 2024-04-25T01:18:16.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.390+0000 7f92e6a00640 1 -- 172.21.15.90:0/1422531552 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92cc003520 con 0x7f92e8071dd0 2024-04-25T01:18:16.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.390+0000 7f92e6a00640 1 -- 172.21.15.90:0/1422531552 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1326191093 0 0) 0x7f92cc003520 con 0x7f92e8071dd0 2024-04-25T01:18:16.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.390+0000 7f92e6a00640 1 -- 172.21.15.90:0/1422531552 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92e80d0240 msgr2=0x7f92e806d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:16.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.390+0000 7f92e6a00640 1 --2- 172.21.15.90:0/1422531552 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92e80d0240 0x7f92e806d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.390+0000 7f92e6a00640 1 -- 172.21.15.90:0/1422531552 >> v1:172.21.15.103:6789/0 conn(0x7f92e815a980 legacy=0x7f92e80cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:16.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.390+0000 7f92e6a00640 1 -- 172.21.15.90:0/1422531552 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92e806e0a0 con 0x7f92e8071dd0 2024-04-25T01:18:16.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.390+0000 7f92e6a00640 1 -- 172.21.15.90:0/1422531552 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f92e0001bc0 con 0x7f92e8071dd0 2024-04-25T01:18:16.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.390+0000 7f92dfe00640 1 --2- 172.21.15.90:0/1422531552 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92e80d0240 0x7f92e806d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:18:16.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.390+0000 7f92e6a00640 1 -- 172.21.15.90:0/1422531552 >> v1:172.21.15.90:6789/0 conn(0x7f92e8071dd0 legacy=0x7f92e80ccfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:16.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.391+0000 7f92e6a00640 1 --2- 172.21.15.90:0/1422531552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92cc0039e0 0x7f92cc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.391+0000 7f92e6a00640 1 --2- 172.21.15.90:0/1422531552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92cc004390 0x7f92cc004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.391+0000 7f92e6a00640 1 --2- 172.21.15.90:0/1422531552 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92e80d0240 0x7f92cc005120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.391+0000 7f92e6a00640 1 -- 172.21.15.90:0/1422531552 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f92e8072260 con 0x7f92cc0039e0 2024-04-25T01:18:16.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.391+0000 7f92e6a00640 1 -- 172.21.15.90:0/1422531552 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f92cc00ba10 con 0x7f92cc004390 2024-04-25T01:18:16.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.391+0000 7f92e6a00640 1 -- 172.21.15.90:0/1422531552 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f92cc00bc10 con 0x7f92e80d0240 2024-04-25T01:18:16.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.391+0000 7f92e7400640 1 --2- 172.21.15.90:0/1422531552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92cc0039e0 0x7f92cc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:16.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.391+0000 7f92dfe00640 1 --2- 172.21.15.90:0/1422531552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92cc004390 0x7f92cc004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:16.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.391+0000 7f92e7e00640 1 --2- 172.21.15.90:0/1422531552 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92e80d0240 0x7f92cc005120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:16.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.391+0000 7f92e7400640 1 -- 172.21.15.90:0/1422531552 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92e80d0240 msgr2=0x7f92cc005120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:16.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.391+0000 7f92e7400640 1 --2- 172.21.15.90:0/1422531552 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92e80d0240 0x7f92cc005120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.391+0000 7f92e7400640 1 -- 172.21.15.90:0/1422531552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92cc004390 msgr2=0x7f92cc004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:16.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.391+0000 7f92e7400640 1 --2- 172.21.15.90:0/1422531552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92cc004390 0x7f92cc004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.391+0000 7f92e7400640 1 -- 172.21.15.90:0/1422531552 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92cc003520 con 0x7f92cc0039e0 2024-04-25T01:18:16.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.391+0000 7f92e7e00640 1 --2- 172.21.15.90:0/1422531552 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92e80d0240 0x7f92cc005120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:16.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.392+0000 7f92e7400640 1 --2- 172.21.15.90:0/1422531552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92cc0039e0 0x7f92cc003dc0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f92d8001860 tx=0x7f92cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7faf07d0f99a00a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:16.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.392+0000 7f92e6a00640 1 -- 172.21.15.90:0/1422531552 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d8037070 con 0x7f92cc0039e0 2024-04-25T01:18:16.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.392+0000 7f92e6a00640 1 -- 172.21.15.90:0/1422531552 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92d802ac30 con 0x7f92cc0039e0 2024-04-25T01:18:16.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.393+0000 7f92e6a00640 1 -- 172.21.15.90:0/1422531552 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d8001430 con 0x7f92cc0039e0 2024-04-25T01:18:16.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.393+0000 7f92ed200640 1 -- 172.21.15.90:0/1422531552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92cc0039e0 msgr2=0x7f92cc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:16.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.393+0000 7f92ed200640 1 --2- 172.21.15.90:0/1422531552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92cc0039e0 0x7f92cc003dc0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f92d8001860 tx=0x7f92cc005830 comp rx=0 tx=0).stop 2024-04-25T01:18:16.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.393+0000 7f92e7e00640 1 -- 172.21.15.90:0/1422531552 reap_dead start 2024-04-25T01:18:16.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.393+0000 7f92ed200640 1 -- 172.21.15.90:0/1422531552 shutdown_connections 2024-04-25T01:18:16.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.393+0000 7f92ed200640 1 -- 172.21.15.90:0/1422531552 >> 172.21.15.90:0/1422531552 conn(0x7f92e806b320 msgr2=0x7f92e806b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:16.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.393+0000 7f92ed200640 1 -- 172.21.15.90:0/1422531552 shutdown_connections 2024-04-25T01:18:16.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.393+0000 7f92ed200640 1 -- 172.21.15.90:0/1422531552 wait complete. 2024-04-25T01:18:16.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.395+0000 7f92ed200640 1 Processor -- start 2024-04-25T01:18:16.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.395+0000 7f92ed200640 1 -- start start 2024-04-25T01:18:16.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.396+0000 7f92ed200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92cc0039e0 0x7f92e8144cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.396+0000 7f92ed200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92e80ad8c0 0x7f92e80add40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.396+0000 7f92ed200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92e8153c20 0x7f92e81540a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.396+0000 7f92ed200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f92e806e660 con 0x7f92e80ad8c0 2024-04-25T01:18:16.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.396+0000 7f92ed200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f92e80aede0 con 0x7f92cc0039e0 2024-04-25T01:18:16.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.396+0000 7f92ed200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f92e815ad60 con 0x7f92e8153c20 2024-04-25T01:18:16.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.396+0000 7f92dfe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92e80ad8c0 0x7f92e80add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:16.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.396+0000 7f92dfe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92e80ad8c0 0x7f92e80add40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40952/0 (socket says 172.21.15.90:40952) 2024-04-25T01:18:16.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.396+0000 7f92dfe00640 1 -- 172.21.15.90:0/2417008836 learned_addr learned my addr 172.21.15.90:0/2417008836 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:16.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.396+0000 7f92e7400640 1 --2- 172.21.15.90:0/2417008836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92cc0039e0 0x7f92e8144cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:16.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.396+0000 7f92e7e00640 1 --2- 172.21.15.90:0/2417008836 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92e8153c20 0x7f92e81540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:16.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.397+0000 7f92dfe00640 1 -- 172.21.15.90:0/2417008836 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92e8153c20 msgr2=0x7f92e81540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:16.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.397+0000 7f92dfe00640 1 --2- 172.21.15.90:0/2417008836 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92e8153c20 0x7f92e81540a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.397+0000 7f92dfe00640 1 -- 172.21.15.90:0/2417008836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92cc0039e0 msgr2=0x7f92e8144cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:16.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.397+0000 7f92dfe00640 1 --2- 172.21.15.90:0/2417008836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92cc0039e0 0x7f92e8144cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.397+0000 7f92dfe00640 1 -- 172.21.15.90:0/2417008836 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92e8154ff0 con 0x7f92e80ad8c0 2024-04-25T01:18:16.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.397+0000 7f92e7e00640 1 --2- 172.21.15.90:0/2417008836 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92e8153c20 0x7f92e81540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:16.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.397+0000 7f92e7400640 1 --2- 172.21.15.90:0/2417008836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92cc0039e0 0x7f92e8144cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:16.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.397+0000 7f92dfe00640 1 --2- 172.21.15.90:0/2417008836 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92e80ad8c0 0x7f92e80add40 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f92d0011600 tx=0x7f92d0011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:16.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.397+0000 7f92df400640 1 -- 172.21.15.90:0/2417008836 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d000f920 con 0x7f92e80ad8c0 2024-04-25T01:18:16.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.397+0000 7f92df400640 1 -- 172.21.15.90:0/2417008836 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92d000fac0 con 0x7f92e80ad8c0 2024-04-25T01:18:16.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.397+0000 7f92ed200640 1 -- 172.21.15.90:0/2417008836 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92e8154a30 con 0x7f92e80ad8c0 2024-04-25T01:18:16.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.397+0000 7f92df400640 1 -- 172.21.15.90:0/2417008836 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d000fe70 con 0x7f92e80ad8c0 2024-04-25T01:18:16.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.397+0000 7f92ed200640 1 -- 172.21.15.90:0/2417008836 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92e8145670 con 0x7f92e80ad8c0 2024-04-25T01:18:16.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.398+0000 7f92ed200640 1 -- 172.21.15.90:0/2417008836 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92e804f500 con 0x7f92e80ad8c0 2024-04-25T01:18:16.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.399+0000 7f92df400640 1 -- 172.21.15.90:0/2417008836 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f92d0015460 con 0x7f92e80ad8c0 2024-04-25T01:18:16.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.400+0000 7f92df400640 1 --2- 172.21.15.90:0/2417008836 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f92c406be60 0x7f92c406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.400+0000 7f92df400640 1 -- 172.21.15.90:0/2417008836 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6245+0+0 (secure 0 0 0) 0x7f92d008b9e0 con 0x7f92e80ad8c0 2024-04-25T01:18:16.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.401+0000 7f92e7400640 1 --2- 172.21.15.90:0/2417008836 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f92c406be60 0x7f92c406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:16.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.401+0000 7f92e7400640 1 --2- 172.21.15.90:0/2417008836 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f92c406be60 0x7f92c406e320 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f92d8034000 tx=0x7f92d8009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:16.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.405+0000 7f92df400640 1 -- 172.21.15.90:0/2417008836 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f92d0019030 con 0x7f92e80ad8c0 2024-04-25T01:18:16.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.601+0000 7f92ed200640 1 -- 172.21.15.90:0/2417008836 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f92e8159f60 con 0x7f92e80ad8c0 2024-04-25T01:18:16.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.602+0000 7f92df400640 1 -- 172.21.15.90:0/2417008836 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) ==== 74+0+17069 (secure 0 0 0) 0x7f92d005a0d0 con 0x7f92e80ad8c0 2024-04-25T01:18:16.586 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:16.586 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":47,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:18:16.361970+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:18:00.504992+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T01:18:01.714313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.16","mappings":[{"from":4,"to":2}]},{"pgid":"6.20","mappings":[{"from":4,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:6837/431390375":"2024-04-26T01:18:15.602554+0000","172.21.15.90:6836/431390375":"2024-04-26T01:18:15.602554+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:18:16.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.605+0000 7f92ed200640 1 -- 172.21.15.90:0/2417008836 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f92c406be60 msgr2=0x7f92c406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:16.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.605+0000 7f92ed200640 1 --2- 172.21.15.90:0/2417008836 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f92c406be60 0x7f92c406e320 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f92d8034000 tx=0x7f92d8009030 comp rx=0 tx=0).stop 2024-04-25T01:18:16.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.605+0000 7f92ed200640 1 -- 172.21.15.90:0/2417008836 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92e80ad8c0 msgr2=0x7f92e80add40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:16.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.605+0000 7f92ed200640 1 --2- 172.21.15.90:0/2417008836 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92e80ad8c0 0x7f92e80add40 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f92d0011600 tx=0x7f92d0011ac0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.605+0000 7f92ed200640 1 -- 172.21.15.90:0/2417008836 shutdown_connections 2024-04-25T01:18:16.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.605+0000 7f92ed200640 1 --2- 172.21.15.90:0/2417008836 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f92c406be60 0x7f92c406e320 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.605+0000 7f92ed200640 1 --2- 172.21.15.90:0/2417008836 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f92e8153c20 0x7f92e81540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.605+0000 7f92ed200640 1 --2- 172.21.15.90:0/2417008836 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92e80ad8c0 0x7f92e80add40 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.605+0000 7f92ed200640 1 --2- 172.21.15.90:0/2417008836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92cc0039e0 0x7f92e8144cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.605+0000 7f92ed200640 1 -- 172.21.15.90:0/2417008836 >> 172.21.15.90:0/2417008836 conn(0x7f92e806b320 msgr2=0x7f92e80b3c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:16.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.606+0000 7f92ed200640 1 -- 172.21.15.90:0/2417008836 shutdown_connections 2024-04-25T01:18:16.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.606+0000 7f92ed200640 1 -- 172.21.15.90:0/2417008836 wait complete. 2024-04-25T01:18:16.610 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:16:15.894126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:18:00.504992+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:18:01.714313+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:18:16.611 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:18:16.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.812+0000 7fb44be00640 1 Processor -- start 2024-04-25T01:18:16.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.813+0000 7fb44be00640 1 -- start start 2024-04-25T01:18:16.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.814+0000 7fb44be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb44c15cd60 0x7fb44c0cfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.814+0000 7fb44be00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb44c0b4df0 con 0x7fb44c15bcd0 2024-04-25T01:18:16.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.814+0000 7fb44be00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb44c0b4b10 con 0x7fb44c0d0200 2024-04-25T01:18:16.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.814+0000 7fb44be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb44c15c8d0 con 0x7fb44c15cd60 2024-04-25T01:18:16.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.814+0000 7fb44aa00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fb44c15bcd0 0x7fb44c0cc230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53374/0 (socket says 172.21.15.90:53374) 2024-04-25T01:18:16.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.814+0000 7fb44aa00640 1 -- 172.21.15.90:0/2997763954 learned_addr learned my addr 172.21.15.90:0/2997763954 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.815+0000 7fb44a000640 1 -- 172.21.15.90:0/2997763954 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3215659051 0 0) 0x7fb44c0b4df0 con 0x7fb44c15bcd0 2024-04-25T01:18:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.815+0000 7fb442000640 1 --2- 172.21.15.90:0/2997763954 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb44c15cd60 0x7fb44c0cfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.815+0000 7fb44a000640 1 -- 172.21.15.90:0/2997763954 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb430003520 con 0x7fb44c15bcd0 2024-04-25T01:18:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.815+0000 7fb44a000640 1 -- 172.21.15.90:0/2997763954 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1678064361 0 0) 0x7fb430003520 con 0x7fb44c15bcd0 2024-04-25T01:18:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.815+0000 7fb44a000640 1 -- 172.21.15.90:0/2997763954 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb44c15cd60 msgr2=0x7fb44c0cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.815+0000 7fb44a000640 1 --2- 172.21.15.90:0/2997763954 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb44c15cd60 0x7fb44c0cfcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.815+0000 7fb44a000640 1 -- 172.21.15.90:0/2997763954 >> v1:172.21.15.103:6789/0 conn(0x7fb44c0d0200 legacy=0x7fb44c15f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.815+0000 7fb44a000640 1 -- 172.21.15.90:0/2997763954 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb44c15c280 con 0x7fb44c15bcd0 2024-04-25T01:18:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.815+0000 7fb44a000640 1 -- 172.21.15.90:0/2997763954 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7fb43c001670 con 0x7fb44c15bcd0 2024-04-25T01:18:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.815+0000 7fb44a000640 1 -- 172.21.15.90:0/2997763954 >> v1:172.21.15.90:6789/0 conn(0x7fb44c15bcd0 legacy=0x7fb44c0cc230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.815+0000 7fb44a000640 1 --2- 172.21.15.90:0/2997763954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4300039e0 0x7fb430003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.815+0000 7fb44a000640 1 --2- 172.21.15.90:0/2997763954 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb44c15cd60 0x7fb430004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.815+0000 7fb44a000640 1 --2- 172.21.15.90:0/2997763954 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb430004d40 0x7fb4300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.815+0000 7fb44a000640 1 -- 172.21.15.90:0/2997763954 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb44c15c8d0 con 0x7fb430004d40 2024-04-25T01:18:16.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.815+0000 7fb44a000640 1 -- 172.21.15.90:0/2997763954 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb43000baf0 con 0x7fb4300039e0 2024-04-25T01:18:16.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.815+0000 7fb44a000640 1 -- 172.21.15.90:0/2997763954 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb43000bcf0 con 0x7fb44c15cd60 2024-04-25T01:18:16.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.816+0000 7fb44b400640 1 --2- 172.21.15.90:0/2997763954 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb430004d40 0x7fb4300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:16.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.816+0000 7fb44aa00640 1 --2- 172.21.15.90:0/2997763954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4300039e0 0x7fb430003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:16.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.816+0000 7fb442000640 1 --2- 172.21.15.90:0/2997763954 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb44c15cd60 0x7fb430004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:16.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.816+0000 7fb44b400640 1 -- 172.21.15.90:0/2997763954 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb44c15cd60 msgr2=0x7fb430004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:16.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.816+0000 7fb44b400640 1 --2- 172.21.15.90:0/2997763954 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb44c15cd60 0x7fb430004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.816+0000 7fb44b400640 1 -- 172.21.15.90:0/2997763954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4300039e0 msgr2=0x7fb430003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:16.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.816+0000 7fb44b400640 1 --2- 172.21.15.90:0/2997763954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4300039e0 0x7fb430003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.816+0000 7fb44b400640 1 -- 172.21.15.90:0/2997763954 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb430003520 con 0x7fb430004d40 2024-04-25T01:18:16.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.816+0000 7fb44aa00640 1 --2- 172.21.15.90:0/2997763954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4300039e0 0x7fb430003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:16.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.816+0000 7fb442000640 1 --2- 172.21.15.90:0/2997763954 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb44c15cd60 0x7fb430004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:16.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.817+0000 7fb44b400640 1 --2- 172.21.15.90:0/2997763954 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb430004d40 0x7fb4300051c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fb4440044b0 tx=0x7fb430005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f967f2ae493f43e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:16.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.817+0000 7fb44a000640 1 -- 172.21.15.90:0/2997763954 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb444037070 con 0x7fb430004d40 2024-04-25T01:18:16.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.817+0000 7fb44a000640 1 -- 172.21.15.90:0/2997763954 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb44402ad80 con 0x7fb430004d40 2024-04-25T01:18:16.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.817+0000 7fb44a000640 1 -- 172.21.15.90:0/2997763954 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb444004080 con 0x7fb430004d40 2024-04-25T01:18:16.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.818+0000 7fb44be00640 1 -- 172.21.15.90:0/2997763954 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb430004d40 msgr2=0x7fb4300051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:16.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.818+0000 7fb44be00640 1 --2- 172.21.15.90:0/2997763954 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb430004d40 0x7fb4300051c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fb4440044b0 tx=0x7fb430005830 comp rx=0 tx=0).stop 2024-04-25T01:18:16.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.818+0000 7fb44b400640 1 -- 172.21.15.90:0/2997763954 reap_dead start 2024-04-25T01:18:16.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.818+0000 7fb44be00640 1 -- 172.21.15.90:0/2997763954 shutdown_connections 2024-04-25T01:18:16.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.818+0000 7fb44be00640 1 -- 172.21.15.90:0/2997763954 >> 172.21.15.90:0/2997763954 conn(0x7fb44c06b560 msgr2=0x7fb44c06b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:16.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.818+0000 7fb44be00640 1 -- 172.21.15.90:0/2997763954 shutdown_connections 2024-04-25T01:18:16.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.818+0000 7fb44be00640 1 -- 172.21.15.90:0/2997763954 wait complete. 2024-04-25T01:18:16.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.820+0000 7fb44be00640 1 Processor -- start 2024-04-25T01:18:16.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.820+0000 7fb44be00640 1 -- start start 2024-04-25T01:18:16.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.821+0000 7fb44be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb430004d40 0x7fb44c0aedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.821+0000 7fb44be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb44c0ad8c0 0x7fb44c145300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.821+0000 7fb44be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb44c14fef0 0x7fb44c14dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.821+0000 7fb44be00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb44c0b4790 con 0x7fb430004d40 2024-04-25T01:18:16.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.821+0000 7fb44be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb44c06e590 con 0x7fb44c0ad8c0 2024-04-25T01:18:16.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.821+0000 7fb44be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb44c0cfcc0 con 0x7fb44c14fef0 2024-04-25T01:18:16.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.821+0000 7fb44aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb430004d40 0x7fb44c0aedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:16.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.821+0000 7fb44aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb430004d40 0x7fb44c0aedf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40982/0 (socket says 172.21.15.90:40982) 2024-04-25T01:18:16.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.821+0000 7fb44aa00640 1 -- 172.21.15.90:0/128872213 learned_addr learned my addr 172.21.15.90:0/128872213 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:16.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.822+0000 7fb442000640 1 --2- 172.21.15.90:0/128872213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb44c0ad8c0 0x7fb44c145300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:16.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.822+0000 7fb44aa00640 1 -- 172.21.15.90:0/128872213 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb44c14fef0 msgr2=0x7fb44c14dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:16.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.822+0000 7fb44aa00640 1 --2- 172.21.15.90:0/128872213 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb44c14fef0 0x7fb44c14dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.822+0000 7fb44aa00640 1 -- 172.21.15.90:0/128872213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb44c0ad8c0 msgr2=0x7fb44c145300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:16.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.822+0000 7fb44aa00640 1 --2- 172.21.15.90:0/128872213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb44c0ad8c0 0x7fb44c145300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:16.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.822+0000 7fb44aa00640 1 -- 172.21.15.90:0/128872213 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb44c145840 con 0x7fb430004d40 2024-04-25T01:18:16.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.822+0000 7fb442000640 1 --2- 172.21.15.90:0/128872213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb44c0ad8c0 0x7fb44c145300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:16.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.822+0000 7fb44aa00640 1 --2- 172.21.15.90:0/128872213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb430004d40 0x7fb44c0aedf0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fb43c00d860 tx=0x7fb43c00dd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:16.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.822+0000 7fb443400640 1 -- 172.21.15.90:0/128872213 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb43c019070 con 0x7fb430004d40 2024-04-25T01:18:16.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.822+0000 7fb443400640 1 -- 172.21.15.90:0/128872213 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb43c0074e0 con 0x7fb430004d40 2024-04-25T01:18:16.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.822+0000 7fb443400640 1 -- 172.21.15.90:0/128872213 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb43c0049e0 con 0x7fb430004d40 2024-04-25T01:18:16.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.822+0000 7fb44be00640 1 -- 172.21.15.90:0/128872213 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb44c0add50 con 0x7fb430004d40 2024-04-25T01:18:16.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.822+0000 7fb44be00640 1 -- 172.21.15.90:0/128872213 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb44c0ae310 con 0x7fb430004d40 2024-04-25T01:18:16.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.823+0000 7fb44be00640 1 -- 172.21.15.90:0/128872213 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb44c04f500 con 0x7fb430004d40 2024-04-25T01:18:16.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.824+0000 7fb443400640 1 -- 172.21.15.90:0/128872213 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb43c01c070 con 0x7fb430004d40 2024-04-25T01:18:16.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.825+0000 7fb443400640 1 --2- 172.21.15.90:0/128872213 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb42806be30 0x7fb42806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:16.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.825+0000 7fb443400640 1 -- 172.21.15.90:0/128872213 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6245+0+0 (secure 0 0 0) 0x7fb43c08ac60 con 0x7fb430004d40 2024-04-25T01:18:16.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.825+0000 7fb442000640 1 --2- 172.21.15.90:0/128872213 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb42806be30 0x7fb42806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:16.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.826+0000 7fb442000640 1 --2- 172.21.15.90:0/128872213 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb42806be30 0x7fb42806e2f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb44c158970 tx=0x7fb434008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:16.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:16.830+0000 7fb443400640 1 -- 172.21.15.90:0/128872213 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb44c145840 con 0x7fb430004d40 2024-04-25T01:18:17.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.026+0000 7fb44be00640 1 -- 172.21.15.90:0/128872213 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb44c160090 con 0x7fb430004d40 2024-04-25T01:18:17.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.026+0000 7fb443400640 1 -- 172.21.15.90:0/128872213 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) ==== 93+0+10 (secure 0 0 0) 0x7fb43c059350 con 0x7fb430004d40 2024-04-25T01:18:17.010 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T01:18:17.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.029+0000 7fb44be00640 1 -- 172.21.15.90:0/128872213 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb42806be30 msgr2=0x7fb42806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:17.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.029+0000 7fb44be00640 1 --2- 172.21.15.90:0/128872213 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb42806be30 0x7fb42806e2f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb44c158970 tx=0x7fb434008040 comp rx=0 tx=0).stop 2024-04-25T01:18:17.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.029+0000 7fb44be00640 1 -- 172.21.15.90:0/128872213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb430004d40 msgr2=0x7fb44c0aedf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:17.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.029+0000 7fb44be00640 1 --2- 172.21.15.90:0/128872213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb430004d40 0x7fb44c0aedf0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fb43c00d860 tx=0x7fb43c00dd20 comp rx=0 tx=0).stop 2024-04-25T01:18:17.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.029+0000 7fb44be00640 1 -- 172.21.15.90:0/128872213 shutdown_connections 2024-04-25T01:18:17.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.029+0000 7fb44be00640 1 --2- 172.21.15.90:0/128872213 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb42806be30 0x7fb42806e2f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:17.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.029+0000 7fb44be00640 1 --2- 172.21.15.90:0/128872213 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb44c14fef0 0x7fb44c14dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:17.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.029+0000 7fb44be00640 1 --2- 172.21.15.90:0/128872213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb44c0ad8c0 0x7fb44c145300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:17.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.029+0000 7fb44be00640 1 --2- 172.21.15.90:0/128872213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb430004d40 0x7fb44c0aedf0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:17.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.029+0000 7fb44be00640 1 -- 172.21.15.90:0/128872213 >> 172.21.15.90:0/128872213 conn(0x7fb44c06b560 msgr2=0x7fb44c15d140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:17.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.029+0000 7fb44be00640 1 -- 172.21.15.90:0/128872213 shutdown_connections 2024-04-25T01:18:17.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.029+0000 7fb44be00640 1 -- 172.21.15.90:0/128872213 wait complete. 2024-04-25T01:18:17.029 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:18:17.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.247+0000 7f3656600640 1 Processor -- start 2024-04-25T01:18:17.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.248+0000 7f3656600640 1 -- start start 2024-04-25T01:18:17.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.249+0000 7f3656600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3650141c80 0x7f36500ce060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:17.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.249+0000 7f3656600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36500ced40 con 0x7f365013ef30 2024-04-25T01:18:17.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.249+0000 7f3656600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36500cf020 con 0x7f36500ce5a0 2024-04-25T01:18:17.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.249+0000 7f3656600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f36501423d0 con 0x7f3650141c80 2024-04-25T01:18:17.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.249+0000 7f364f400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f365013ef30 0x7f3650141570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53380/0 (socket says 172.21.15.90:53380) 2024-04-25T01:18:17.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.249+0000 7f364f400640 1 -- 172.21.15.90:0/3294951907 learned_addr learned my addr 172.21.15.90:0/3294951907 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:17.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.250+0000 7f364ea00640 1 --2- 172.21.15.90:0/3294951907 >> v2:172.21.15.90:3300/0 conn(0x7f3650141c80 0x7f36500ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:17.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.250+0000 7f364ea00640 1 -- 172.21.15.90:0/3294951907 >> v1:172.21.15.103:6789/0 conn(0x7f36500ce5a0 legacy=0x7f3650071410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:17.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.250+0000 7f364e000640 1 -- 172.21.15.90:0/3294951907 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 882866698 0 0) 0x7f36500ced40 con 0x7f365013ef30 2024-04-25T01:18:17.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.250+0000 7f364ea00640 1 -- 172.21.15.90:0/3294951907 >> v1:172.21.15.90:6789/0 conn(0x7f365013ef30 legacy=0x7f3650141570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:17.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.250+0000 7f364ea00640 1 -- 172.21.15.90:0/3294951907 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36500cf300 con 0x7f3650141c80 2024-04-25T01:18:17.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.251+0000 7f364ea00640 1 --2- 172.21.15.90:0/3294951907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3650141c80 0x7f36500ce060 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f36380097a0 tx=0x7f363802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbe97a0c6af50ec4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:17.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.251+0000 7f364e000640 1 -- 172.21.15.90:0/3294951907 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3638037070 con 0x7f3650141c80 2024-04-25T01:18:17.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.251+0000 7f364e000640 1 -- 172.21.15.90:0/3294951907 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3638032440 con 0x7f3650141c80 2024-04-25T01:18:17.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.252+0000 7f364e000640 1 -- 172.21.15.90:0/3294951907 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3638032810 con 0x7f3650141c80 2024-04-25T01:18:17.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.252+0000 7f3656600640 1 -- 172.21.15.90:0/3294951907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3650141c80 msgr2=0x7f36500ce060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:17.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.252+0000 7f3656600640 1 --2- 172.21.15.90:0/3294951907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3650141c80 0x7f36500ce060 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f36380097a0 tx=0x7f363802a1e0 comp rx=0 tx=0).stop 2024-04-25T01:18:17.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.252+0000 7f3656600640 1 -- 172.21.15.90:0/3294951907 shutdown_connections 2024-04-25T01:18:17.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.252+0000 7f3656600640 1 --2- 172.21.15.90:0/3294951907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3650141c80 0x7f36500ce060 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:17.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.252+0000 7f3656600640 1 -- 172.21.15.90:0/3294951907 >> 172.21.15.90:0/3294951907 conn(0x7f36500b17b0 msgr2=0x7f36500b1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:17.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.252+0000 7f3656600640 1 -- 172.21.15.90:0/3294951907 shutdown_connections 2024-04-25T01:18:17.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.252+0000 7f3656600640 1 -- 172.21.15.90:0/3294951907 wait complete. 2024-04-25T01:18:17.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.254+0000 7f3656600640 1 Processor -- start 2024-04-25T01:18:17.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.254+0000 7f3656600640 1 -- start start 2024-04-25T01:18:17.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.255+0000 7f3656600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f36500ce5a0 0x7f365013ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:17.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.255+0000 7f3656600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f365013ef30 0x7f36501362e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:17.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.255+0000 7f3656600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3650141c80 0x7f3650138e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:17.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.255+0000 7f3656600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f36500cea30 con 0x7f3650141c80 2024-04-25T01:18:17.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.255+0000 7f3656600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3650070f80 con 0x7f365013ef30 2024-04-25T01:18:17.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.255+0000 7f3656600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3650071b20 con 0x7f36500ce5a0 2024-04-25T01:18:17.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.255+0000 7f364fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3650141c80 0x7f3650138e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:17.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.256+0000 7f364fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3650141c80 0x7f3650138e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40994/0 (socket says 172.21.15.90:40994) 2024-04-25T01:18:17.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.256+0000 7f364fe00640 1 -- 172.21.15.90:0/2183211134 learned_addr learned my addr 172.21.15.90:0/2183211134 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:17.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.256+0000 7f364ea00640 1 --2- 172.21.15.90:0/2183211134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f365013ef30 0x7f36501362e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:17.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.256+0000 7f364fe00640 1 -- 172.21.15.90:0/2183211134 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f36500ce5a0 msgr2=0x7f365013ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:17.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.256+0000 7f364f400640 1 --2- 172.21.15.90:0/2183211134 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f36500ce5a0 0x7f365013ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:17.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.256+0000 7f364fe00640 1 --2- 172.21.15.90:0/2183211134 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f36500ce5a0 0x7f365013ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:17.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.256+0000 7f364fe00640 1 -- 172.21.15.90:0/2183211134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f365013ef30 msgr2=0x7f36501362e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:17.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.256+0000 7f364fe00640 1 --2- 172.21.15.90:0/2183211134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f365013ef30 0x7f36501362e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:17.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.256+0000 7f364fe00640 1 -- 172.21.15.90:0/2183211134 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f365006de40 con 0x7f3650141c80 2024-04-25T01:18:17.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.256+0000 7f364ea00640 1 --2- 172.21.15.90:0/2183211134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f365013ef30 0x7f36501362e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:17.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.256+0000 7f364fe00640 1 --2- 172.21.15.90:0/2183211134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3650141c80 0x7f3650138e00 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f364800c7b0 tx=0x7f364800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:17.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.257+0000 7f3647400640 1 -- 172.21.15.90:0/2183211134 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3648004b60 con 0x7f3650141c80 2024-04-25T01:18:17.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.257+0000 7f3647400640 1 -- 172.21.15.90:0/2183211134 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3648004d00 con 0x7f3650141c80 2024-04-25T01:18:17.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.257+0000 7f3647400640 1 -- 172.21.15.90:0/2183211134 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f364800aa30 con 0x7f3650141c80 2024-04-25T01:18:17.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.257+0000 7f3656600640 1 -- 172.21.15.90:0/2183211134 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f365013b440 con 0x7f3650141c80 2024-04-25T01:18:17.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.257+0000 7f3656600640 1 -- 172.21.15.90:0/2183211134 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f365013ba00 con 0x7f3650141c80 2024-04-25T01:18:17.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.258+0000 7f3656600640 1 -- 172.21.15.90:0/2183211134 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36500ced40 con 0x7f3650141c80 2024-04-25T01:18:17.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.259+0000 7f3647400640 1 -- 172.21.15.90:0/2183211134 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f364800abd0 con 0x7f3650141c80 2024-04-25T01:18:17.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.261+0000 7f3647400640 1 --2- 172.21.15.90:0/2183211134 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f362c06be30 0x7f362c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:17.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.261+0000 7f3647400640 1 -- 172.21.15.90:0/2183211134 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6245+0+0 (secure 0 0 0) 0x7f364808b370 con 0x7f3650141c80 2024-04-25T01:18:17.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.261+0000 7f364f400640 1 --2- 172.21.15.90:0/2183211134 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f362c06be30 0x7f362c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:17.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.262+0000 7f364f400640 1 --2- 172.21.15.90:0/2183211134 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f362c06be30 0x7f362c06e2f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3640007fd0 tx=0x7f3640009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:17.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.265+0000 7f3647400640 1 -- 172.21.15.90:0/2183211134 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f36500ced40 con 0x7f3650141c80 2024-04-25T01:18:17.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.467+0000 7f3656600640 1 -- 172.21.15.90:0/2183211134 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3650145950 con 0x7f3650141c80 2024-04-25T01:18:17.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.467+0000 7f3647400640 1 -- 172.21.15.90:0/2183211134 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v47) ==== 104+0+11 (secure 0 0 0) 0x7f3648020070 con 0x7f3650141c80 2024-04-25T01:18:17.451 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:18:17.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.470+0000 7f3656600640 1 -- 172.21.15.90:0/2183211134 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f362c06be30 msgr2=0x7f362c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:17.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.470+0000 7f3656600640 1 --2- 172.21.15.90:0/2183211134 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f362c06be30 0x7f362c06e2f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3640007fd0 tx=0x7f3640009040 comp rx=0 tx=0).stop 2024-04-25T01:18:17.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.470+0000 7f3656600640 1 -- 172.21.15.90:0/2183211134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3650141c80 msgr2=0x7f3650138e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:17.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.470+0000 7f3656600640 1 --2- 172.21.15.90:0/2183211134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3650141c80 0x7f3650138e00 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f364800c7b0 tx=0x7f364800cc70 comp rx=0 tx=0).stop 2024-04-25T01:18:17.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.470+0000 7f3656600640 1 -- 172.21.15.90:0/2183211134 shutdown_connections 2024-04-25T01:18:17.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.470+0000 7f3656600640 1 --2- 172.21.15.90:0/2183211134 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f362c06be30 0x7f362c06e2f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:17.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.470+0000 7f3656600640 1 --2- 172.21.15.90:0/2183211134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3650141c80 0x7f3650138e00 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:17.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.470+0000 7f3656600640 1 --2- 172.21.15.90:0/2183211134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f365013ef30 0x7f36501362e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:17.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.470+0000 7f3656600640 1 --2- 172.21.15.90:0/2183211134 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f36500ce5a0 0x7f365013ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:17.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.470+0000 7f3656600640 1 -- 172.21.15.90:0/2183211134 >> 172.21.15.90:0/2183211134 conn(0x7f36500b17b0 msgr2=0x7f36501394d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:17.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.471+0000 7f3656600640 1 -- 172.21.15.90:0/2183211134 shutdown_connections 2024-04-25T01:18:17.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.471+0000 7f3656600640 1 -- 172.21.15.90:0/2183211134 wait complete. 2024-04-25T01:18:17.471 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:18:17.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.681+0000 7f6f41600640 1 Processor -- start 2024-04-25T01:18:17.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.682+0000 7f6f41600640 1 -- start start 2024-04-25T01:18:17.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.683+0000 7f6f41600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f3c06ea40 0x7f6f3c15be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:17.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.683+0000 7f6f41600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6f3c0b2ab0 0x7f6f3c0713e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:17.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.683+0000 7f6f41600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f3c0b3250 con 0x7f6f3c155460 2024-04-25T01:18:17.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.683+0000 7f6f41600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f6f3c0b3530 con 0x7f6f3c0b2ab0 2024-04-25T01:18:17.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.683+0000 7f6f41600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6f3c0b3810 con 0x7f6f3c06ea40 2024-04-25T01:18:17.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.683+0000 7f6f3aa00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f6f3c155460 0x7f6f3c06e330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53384/0 (socket says 172.21.15.90:53384) 2024-04-25T01:18:17.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.683+0000 7f6f3b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6f3c0b2ab0 0x7f6f3c0713e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:17.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.684+0000 7f6f3aa00640 1 -- 172.21.15.90:0/1743035874 learned_addr learned my addr 172.21.15.90:0/1743035874 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:17.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.684+0000 7f6f3a000640 1 --2- 172.21.15.90:0/1743035874 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f3c06ea40 0x7f6f3c15be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:17.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.684+0000 7f6f3b400640 1 -- 172.21.15.90:0/1743035874 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f3c06ea40 msgr2=0x7f6f3c15be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:17.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.684+0000 7f6f3b400640 1 --2- 172.21.15.90:0/1743035874 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f3c06ea40 0x7f6f3c15be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:17.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.684+0000 7f6f3b400640 1 -- 172.21.15.90:0/1743035874 >> v1:172.21.15.90:6789/0 conn(0x7f6f3c155460 legacy=0x7f6f3c06e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:17.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.684+0000 7f6f3b400640 1 -- 172.21.15.90:0/1743035874 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f3c070ec0 con 0x7f6f3c0b2ab0 2024-04-25T01:18:17.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.684+0000 7f6f3a000640 1 --2- 172.21.15.90:0/1743035874 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f3c06ea40 0x7f6f3c15be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:17.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.685+0000 7f6f3b400640 1 --2- 172.21.15.90:0/1743035874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f3c0b2ab0 0x7f6f3c0713e0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f6f340097a0 tx=0x7f6f3402c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b48d4d777f7f516f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:17.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.685+0000 7f6f39600640 1 -- 172.21.15.90:0/1743035874 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f34039070 con 0x7f6f3c0b2ab0 2024-04-25T01:18:17.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.685+0000 7f6f39600640 1 -- 172.21.15.90:0/1743035874 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f34034640 con 0x7f6f3c0b2ab0 2024-04-25T01:18:17.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.685+0000 7f6f39600640 1 -- 172.21.15.90:0/1743035874 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f34034a10 con 0x7f6f3c0b2ab0 2024-04-25T01:18:17.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.686+0000 7f6f41600640 1 -- 172.21.15.90:0/1743035874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f3c0b2ab0 msgr2=0x7f6f3c0713e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:17.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.686+0000 7f6f41600640 1 --2- 172.21.15.90:0/1743035874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f3c0b2ab0 0x7f6f3c0713e0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f6f340097a0 tx=0x7f6f3402c390 comp rx=0 tx=0).stop 2024-04-25T01:18:17.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.686+0000 7f6f41600640 1 -- 172.21.15.90:0/1743035874 shutdown_connections 2024-04-25T01:18:17.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.686+0000 7f6f41600640 1 --2- 172.21.15.90:0/1743035874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f3c0b2ab0 0x7f6f3c0713e0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:17.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.686+0000 7f6f41600640 1 --2- 172.21.15.90:0/1743035874 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f3c06ea40 0x7f6f3c15be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:17.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.686+0000 7f6f41600640 1 -- 172.21.15.90:0/1743035874 >> 172.21.15.90:0/1743035874 conn(0x7f6f3c0b17b0 msgr2=0x7f6f3c0b1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:17.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.686+0000 7f6f41600640 1 -- 172.21.15.90:0/1743035874 shutdown_connections 2024-04-25T01:18:17.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.686+0000 7f6f41600640 1 -- 172.21.15.90:0/1743035874 wait complete. 2024-04-25T01:18:17.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.688+0000 7f6f41600640 1 Processor -- start 2024-04-25T01:18:17.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.688+0000 7f6f41600640 1 -- start start 2024-04-25T01:18:17.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.689+0000 7f6f41600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f3c06ea40 0x7f6f3c1f3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:17.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.689+0000 7f6f41600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f3c0b2ab0 0x7f6f3c156030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:17.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.689+0000 7f6f41600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f3c155460 0x7f6f3c15d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:17.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.689+0000 7f6f41600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6f3c15c480 con 0x7f6f3c0b2ab0 2024-04-25T01:18:17.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.689+0000 7f6f41600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6f3c0b2ed0 con 0x7f6f3c155460 2024-04-25T01:18:17.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.689+0000 7f6f41600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6f3c15c600 con 0x7f6f3c06ea40 2024-04-25T01:18:17.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.689+0000 7f6f3a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f3c0b2ab0 0x7f6f3c156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:17.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.689+0000 7f6f3a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f3c0b2ab0 0x7f6f3c156030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41010/0 (socket says 172.21.15.90:41010) 2024-04-25T01:18:17.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.689+0000 7f6f3a000640 1 -- 172.21.15.90:0/1846004428 learned_addr learned my addr 172.21.15.90:0/1846004428 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:17.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.689+0000 7f6f3a000640 1 -- 172.21.15.90:0/1846004428 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f3c06ea40 msgr2=0x7f6f3c1f3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:17.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.690+0000 7f6f3a000640 1 --2- 172.21.15.90:0/1846004428 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f3c06ea40 0x7f6f3c1f3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:17.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.690+0000 7f6f3a000640 1 -- 172.21.15.90:0/1846004428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f3c155460 msgr2=0x7f6f3c15d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:17.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.690+0000 7f6f3a000640 1 --2- 172.21.15.90:0/1846004428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f3c155460 0x7f6f3c15d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:17.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.690+0000 7f6f3a000640 1 -- 172.21.15.90:0/1846004428 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f3c157e10 con 0x7f6f3c0b2ab0 2024-04-25T01:18:17.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.690+0000 7f6f3a000640 1 --2- 172.21.15.90:0/1846004428 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f3c0b2ab0 0x7f6f3c156030 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f6f2400cf80 tx=0x7f6f2400a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:17.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.690+0000 7f6f32a00640 1 -- 172.21.15.90:0/1846004428 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2401a070 con 0x7f6f3c0b2ab0 2024-04-25T01:18:17.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.690+0000 7f6f32a00640 1 -- 172.21.15.90:0/1846004428 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f2400d840 con 0x7f6f3c0b2ab0 2024-04-25T01:18:17.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.690+0000 7f6f41600640 1 -- 172.21.15.90:0/1846004428 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f3c1580f0 con 0x7f6f3c0b2ab0 2024-04-25T01:18:17.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.690+0000 7f6f32a00640 1 -- 172.21.15.90:0/1846004428 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2400dc30 con 0x7f6f3c0b2ab0 2024-04-25T01:18:17.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.690+0000 7f6f41600640 1 -- 172.21.15.90:0/1846004428 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f3c1586b0 con 0x7f6f3c0b2ab0 2024-04-25T01:18:17.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.691+0000 7f6f41600640 1 -- 172.21.15.90:0/1846004428 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f3c15db70 con 0x7f6f3c0b2ab0 2024-04-25T01:18:17.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.692+0000 7f6f32a00640 1 -- 172.21.15.90:0/1846004428 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6f2401d070 con 0x7f6f3c0b2ab0 2024-04-25T01:18:17.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.693+0000 7f6f32a00640 1 --2- 172.21.15.90:0/1846004428 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f6f1806be30 0x7f6f1806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:17.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.693+0000 7f6f32a00640 1 -- 172.21.15.90:0/1846004428 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6245+0+0 (secure 0 0 0) 0x7f6f2408b7e0 con 0x7f6f3c0b2ab0 2024-04-25T01:18:17.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.693+0000 7f6f3aa00640 1 --2- 172.21.15.90:0/1846004428 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f6f1806be30 0x7f6f1806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:17.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.694+0000 7f6f3aa00640 1 --2- 172.21.15.90:0/1846004428 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f6f1806be30 0x7f6f1806e2f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c000f60 tx=0x7f6f2c007a80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:17.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.698+0000 7f6f32a00640 1 -- 172.21.15.90:0/1846004428 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6f24059920 con 0x7f6f3c0b2ab0 2024-04-25T01:18:17.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.894+0000 7f6f41600640 1 -- 172.21.15.90:0/1846004428 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6f3c06d610 con 0x7f6f3c0b2ab0 2024-04-25T01:18:17.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.895+0000 7f6f32a00640 1 -- 172.21.15.90:0/1846004428 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v47) ==== 100+0+11 (secure 0 0 0) 0x7f6f2405dc20 con 0x7f6f3c0b2ab0 2024-04-25T01:18:17.878 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-25T01:18:17.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.897+0000 7f6f41600640 1 -- 172.21.15.90:0/1846004428 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f6f1806be30 msgr2=0x7f6f1806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:17.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.897+0000 7f6f41600640 1 --2- 172.21.15.90:0/1846004428 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f6f1806be30 0x7f6f1806e2f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c000f60 tx=0x7f6f2c007a80 comp rx=0 tx=0).stop 2024-04-25T01:18:17.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.897+0000 7f6f41600640 1 -- 172.21.15.90:0/1846004428 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f3c0b2ab0 msgr2=0x7f6f3c156030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:17.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.897+0000 7f6f41600640 1 --2- 172.21.15.90:0/1846004428 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f3c0b2ab0 0x7f6f3c156030 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f6f2400cf80 tx=0x7f6f2400a7d0 comp rx=0 tx=0).stop 2024-04-25T01:18:17.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.897+0000 7f6f41600640 1 -- 172.21.15.90:0/1846004428 shutdown_connections 2024-04-25T01:18:17.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.897+0000 7f6f41600640 1 --2- 172.21.15.90:0/1846004428 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f6f1806be30 0x7f6f1806e2f0 secure :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c000f60 tx=0x7f6f2c007a80 comp rx=0 tx=0).stop 2024-04-25T01:18:17.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.897+0000 7f6f41600640 1 --2- 172.21.15.90:0/1846004428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f3c155460 0x7f6f3c15d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:17.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.897+0000 7f6f41600640 1 --2- 172.21.15.90:0/1846004428 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f3c0b2ab0 0x7f6f3c156030 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:17.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.897+0000 7f6f41600640 1 --2- 172.21.15.90:0/1846004428 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f3c06ea40 0x7f6f3c1f3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:17.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.897+0000 7f6f41600640 1 -- 172.21.15.90:0/1846004428 >> 172.21.15.90:0/1846004428 conn(0x7f6f3c0b17b0 msgr2=0x7f6f3c153ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:17.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.898+0000 7f6f41600640 1 -- 172.21.15.90:0/1846004428 shutdown_connections 2024-04-25T01:18:17.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:17.898+0000 7f6f41600640 1 -- 172.21.15.90:0/1846004428 wait complete. 2024-04-25T01:18:17.899 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'recovery_fs' 2024-04-25T01:18:17.900 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create recovery_fs_metadata --pg_num_min 64 2024-04-25T01:18:18.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.119+0000 7f8df1c00640 1 Processor -- start 2024-04-25T01:18:18.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.120+0000 7f8df1c00640 1 -- start start 2024-04-25T01:18:18.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.121+0000 7f8df1c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8dec1445e0 0x7f8dec0cfc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:18.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.121+0000 7f8df1c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8dec071320 0x7f8dec0d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:18.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.121+0000 7f8df1c00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dec1449c0 con 0x7f8dec0728c0 2024-04-25T01:18:18.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.121+0000 7f8df1c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f8dec072430 con 0x7f8dec1445e0 2024-04-25T01:18:18.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.121+0000 7f8df1c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8dec0b2d70 con 0x7f8dec071320 2024-04-25T01:18:18.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.122+0000 7f8deb400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8dec071320 0x7f8dec0d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:18.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.122+0000 7f8deaa00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f8dec1445e0 0x7f8dec0cfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:18.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.122+0000 7f8deb400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8dec071320 0x7f8dec0d01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.90:39796/0 (socket says 172.21.15.90:39796) 2024-04-25T01:18:18.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.122+0000 7f8deb400640 1 -- 172.21.15.90:0/1750306975 learned_addr learned my addr 172.21.15.90:0/1750306975 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:18.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.122+0000 7f8deb400640 1 -- 172.21.15.90:0/1750306975 >> v2:172.21.15.103:3300/0 conn(0x7f8dec1445e0 msgr2=0x7f8dec0cfc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:18.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.122+0000 7f8deb400640 1 --2- 172.21.15.90:0/1750306975 >> v2:172.21.15.103:3300/0 conn(0x7f8dec1445e0 0x7f8dec0cfc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:18.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.122+0000 7f8deb400640 1 -- 172.21.15.90:0/1750306975 >> v1:172.21.15.103:6789/0 conn(0x7f8dec0728c0 legacy=0x7f8dec070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:18.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.122+0000 7f8deb400640 1 -- 172.21.15.90:0/1750306975 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dec071820 con 0x7f8dec071320 2024-04-25T01:18:18.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.124+0000 7f8deb400640 1 --2- 172.21.15.90:0/1750306975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8dec071320 0x7f8dec0d01d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8de4009e50 tx=0x7f8de402a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35fc3843abb3ca55 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:18.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.124+0000 7f8dea000640 1 -- 172.21.15.90:0/1750306975 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de4036070 con 0x7f8dec071320 2024-04-25T01:18:18.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.124+0000 7f8dea000640 1 -- 172.21.15.90:0/1750306975 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8de4031600 con 0x7f8dec071320 2024-04-25T01:18:18.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.124+0000 7f8dea000640 1 -- 172.21.15.90:0/1750306975 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de40319d0 con 0x7f8dec071320 2024-04-25T01:18:18.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.125+0000 7f8df1c00640 1 -- 172.21.15.90:0/1750306975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8dec071320 msgr2=0x7f8dec0d01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:18.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.125+0000 7f8df1c00640 1 --2- 172.21.15.90:0/1750306975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8dec071320 0x7f8dec0d01d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8de4009e50 tx=0x7f8de402a710 comp rx=0 tx=0).stop 2024-04-25T01:18:18.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.125+0000 7f8df1c00640 1 -- 172.21.15.90:0/1750306975 shutdown_connections 2024-04-25T01:18:18.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.125+0000 7f8df1c00640 1 --2- 172.21.15.90:0/1750306975 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8dec071320 0x7f8dec0d01d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:18.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.125+0000 7f8df1c00640 1 --2- 172.21.15.90:0/1750306975 >> v2:172.21.15.103:3300/0 conn(0x7f8dec1445e0 0x7f8dec0cfc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:18.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.125+0000 7f8df1c00640 1 -- 172.21.15.90:0/1750306975 >> 172.21.15.90:0/1750306975 conn(0x7f8dec06b320 msgr2=0x7f8dec06b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:18.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.125+0000 7f8df1c00640 1 -- 172.21.15.90:0/1750306975 shutdown_connections 2024-04-25T01:18:18.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.125+0000 7f8df1c00640 1 -- 172.21.15.90:0/1750306975 wait complete. 2024-04-25T01:18:18.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.127+0000 7f8df1c00640 1 Processor -- start 2024-04-25T01:18:18.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.127+0000 7f8df1c00640 1 -- start start 2024-04-25T01:18:18.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.128+0000 7f8df1c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8dec071320 0x7f8dec15a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:18.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.128+0000 7f8df1c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8dec0728c0 0x7f8dec13dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:18.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.128+0000 7f8df1c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8dec1445e0 0x7f8dec0ce3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:18.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.128+0000 7f8df1c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8dec0720c0 con 0x7f8dec0728c0 2024-04-25T01:18:18.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.128+0000 7f8df1c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8dec071de0 con 0x7f8dec071320 2024-04-25T01:18:18.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.128+0000 7f8df1c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8dec06cbc0 con 0x7f8dec1445e0 2024-04-25T01:18:18.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.128+0000 7f8de2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8dec0728c0 0x7f8dec13dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:18.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.128+0000 7f8de2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8dec0728c0 0x7f8dec13dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41026/0 (socket says 172.21.15.90:41026) 2024-04-25T01:18:18.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.128+0000 7f8de2000640 1 -- 172.21.15.90:0/2071527313 learned_addr learned my addr 172.21.15.90:0/2071527313 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:18.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.129+0000 7f8deaa00640 1 --2- 172.21.15.90:0/2071527313 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8dec071320 0x7f8dec15a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:18.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.129+0000 7f8deb400640 1 --2- 172.21.15.90:0/2071527313 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8dec1445e0 0x7f8dec0ce3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:18.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.129+0000 7f8de2000640 1 -- 172.21.15.90:0/2071527313 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8dec1445e0 msgr2=0x7f8dec0ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:18.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.129+0000 7f8de2000640 1 --2- 172.21.15.90:0/2071527313 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8dec1445e0 0x7f8dec0ce3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:18.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.129+0000 7f8de2000640 1 -- 172.21.15.90:0/2071527313 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8dec071320 msgr2=0x7f8dec15a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:18.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.129+0000 7f8de2000640 1 --2- 172.21.15.90:0/2071527313 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8dec071320 0x7f8dec15a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:18.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.129+0000 7f8de2000640 1 -- 172.21.15.90:0/2071527313 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dec166f90 con 0x7f8dec0728c0 2024-04-25T01:18:18.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.129+0000 7f8de2000640 1 --2- 172.21.15.90:0/2071527313 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8dec0728c0 0x7f8dec13dd60 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f8dd400c470 tx=0x7f8dd4001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:18.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.129+0000 7f8de3400640 1 -- 172.21.15.90:0/2071527313 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd40052d0 con 0x7f8dec0728c0 2024-04-25T01:18:18.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.129+0000 7f8de3400640 1 -- 172.21.15.90:0/2071527313 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dd4005470 con 0x7f8dec0728c0 2024-04-25T01:18:18.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.129+0000 7f8de3400640 1 -- 172.21.15.90:0/2071527313 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd4007260 con 0x7f8dec0728c0 2024-04-25T01:18:18.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.129+0000 7f8df1c00640 1 -- 172.21.15.90:0/2071527313 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dec167270 con 0x7f8dec0728c0 2024-04-25T01:18:18.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.129+0000 7f8df1c00640 1 -- 172.21.15.90:0/2071527313 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dec15ad80 con 0x7f8dec0728c0 2024-04-25T01:18:18.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.131+0000 7f8df1c00640 1 -- 172.21.15.90:0/2071527313 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dec068190 con 0x7f8dec0728c0 2024-04-25T01:18:18.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.131+0000 7f8de3400640 1 -- 172.21.15.90:0/2071527313 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8dd4007400 con 0x7f8dec0728c0 2024-04-25T01:18:18.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.132+0000 7f8de3400640 1 --2- 172.21.15.90:0/2071527313 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f8dc806be60 0x7f8dc806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:18.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.132+0000 7f8de3400640 1 -- 172.21.15.90:0/2071527313 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6245+0+0 (secure 0 0 0) 0x7f8dd408a5c0 con 0x7f8dec0728c0 2024-04-25T01:18:18.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.132+0000 7f8deaa00640 1 --2- 172.21.15.90:0/2071527313 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f8dc806be60 0x7f8dc806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:18.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.133+0000 7f8deaa00640 1 --2- 172.21.15.90:0/2071527313 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f8dc806be60 0x7f8dc806e320 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc009970 tx=0x7f8ddc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:18.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.136+0000 7f8de3400640 1 -- 172.21.15.90:0/2071527313 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8dd4058d40 con 0x7f8dec0728c0 2024-04-25T01:18:18.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.333+0000 7f8df1c00640 1 -- 172.21.15.90:0/2071527313 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "recovery_fs_metadata", "pg_num_min": 64} v 0) -- 0x7f8dec06eab0 con 0x7f8dec0728c0 2024-04-25T01:18:18.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.439+0000 7f8de3400640 1 -- 172.21.15.90:0/2071527313 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "recovery_fs_metadata", "pg_num_min": 64}]=0 pool 'recovery_fs_metadata' created v48) ==== 148+0+0 (secure 0 0 0) 0x7f8dd4058640 con 0x7f8dec0728c0 2024-04-25T01:18:18.422 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'recovery_fs_metadata' created 2024-04-25T01:18:18.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.442+0000 7f8df1c00640 1 -- 172.21.15.90:0/2071527313 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f8dc806be60 msgr2=0x7f8dc806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:18.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.442+0000 7f8df1c00640 1 --2- 172.21.15.90:0/2071527313 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f8dc806be60 0x7f8dc806e320 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc009970 tx=0x7f8ddc008040 comp rx=0 tx=0).stop 2024-04-25T01:18:18.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.442+0000 7f8df1c00640 1 -- 172.21.15.90:0/2071527313 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8dec0728c0 msgr2=0x7f8dec13dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:18.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.442+0000 7f8df1c00640 1 --2- 172.21.15.90:0/2071527313 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8dec0728c0 0x7f8dec13dd60 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f8dd400c470 tx=0x7f8dd4001e70 comp rx=0 tx=0).stop 2024-04-25T01:18:18.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.442+0000 7f8df1c00640 1 -- 172.21.15.90:0/2071527313 shutdown_connections 2024-04-25T01:18:18.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.442+0000 7f8df1c00640 1 --2- 172.21.15.90:0/2071527313 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f8dc806be60 0x7f8dc806e320 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:18.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.442+0000 7f8df1c00640 1 --2- 172.21.15.90:0/2071527313 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8dec1445e0 0x7f8dec0ce3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:18.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.442+0000 7f8df1c00640 1 --2- 172.21.15.90:0/2071527313 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8dec0728c0 0x7f8dec13dd60 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:18.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.442+0000 7f8df1c00640 1 --2- 172.21.15.90:0/2071527313 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8dec071320 0x7f8dec15a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:18.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.442+0000 7f8df1c00640 1 -- 172.21.15.90:0/2071527313 >> 172.21.15.90:0/2071527313 conn(0x7f8dec06b320 msgr2=0x7f8dec0b3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:18.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.442+0000 7f8df1c00640 1 -- 172.21.15.90:0/2071527313 shutdown_connections 2024-04-25T01:18:18.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.442+0000 7f8df1c00640 1 -- 172.21.15.90:0/2071527313 wait complete. 2024-04-25T01:18:18.453 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T01:18:18.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.656+0000 7ff4ba400640 1 Processor -- start 2024-04-25T01:18:18.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.656+0000 7ff4ba400640 1 -- start start 2024-04-25T01:18:18.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.657+0000 7ff4ba400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff4b40afbd0 0x7ff4b40b4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:18.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.658+0000 7ff4ba400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4b406ef30 0x7ff4b4161b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:18.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.658+0000 7ff4ba400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4b40b4880 con 0x7ff4b4162090 2024-04-25T01:18:18.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.658+0000 7ff4ba400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7ff4b40b5120 con 0x7ff4b40afbd0 2024-04-25T01:18:18.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.658+0000 7ff4ba400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff4b40b4b60 con 0x7ff4b406ef30 2024-04-25T01:18:18.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.658+0000 7ff4b3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff4b40afbd0 0x7ff4b40b4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:18.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.658+0000 7ff4b3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff4b40afbd0 0x7ff4b40b4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41034/0 (socket says 172.21.15.90:41034) 2024-04-25T01:18:18.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.658+0000 7ff4b3e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff4b4162090 0x7ff4b415e510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53396/0 (socket says 172.21.15.90:53396) 2024-04-25T01:18:18.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.658+0000 7ff4b3400640 1 -- 172.21.15.90:0/3477332937 learned_addr learned my addr 172.21.15.90:0/3477332937 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:18.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.658+0000 7ff4b2a00640 1 --2- 172.21.15.90:0/3477332937 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4b406ef30 0x7ff4b4161b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:18.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.658+0000 7ff4b3400640 1 -- 172.21.15.90:0/3477332937 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4b406ef30 msgr2=0x7ff4b4161b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:18.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.658+0000 7ff4b3400640 1 --2- 172.21.15.90:0/3477332937 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4b406ef30 0x7ff4b4161b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:18.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.658+0000 7ff4b3400640 1 -- 172.21.15.90:0/3477332937 >> v1:172.21.15.90:6789/0 conn(0x7ff4b4162090 legacy=0x7ff4b415e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:18.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.658+0000 7ff4b3400640 1 -- 172.21.15.90:0/3477332937 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4b40b4e40 con 0x7ff4b40afbd0 2024-04-25T01:18:18.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.658+0000 7ff4b2000640 1 -- 172.21.15.90:0/3477332937 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 313809525 0 0) 0x7ff4b40b4880 con 0x7ff4b4162090 2024-04-25T01:18:18.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.660+0000 7ff4b3400640 1 --2- 172.21.15.90:0/3477332937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4b40afbd0 0x7ff4b40b4030 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ff4a4009e50 tx=0x7ff4a402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aebd92c204e87282 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:18.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.660+0000 7ff4b2000640 1 -- 172.21.15.90:0/3477332937 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a4036070 con 0x7ff4b40afbd0 2024-04-25T01:18:18.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.660+0000 7ff4b2000640 1 -- 172.21.15.90:0/3477332937 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4a4031600 con 0x7ff4b40afbd0 2024-04-25T01:18:18.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.660+0000 7ff4b2000640 1 -- 172.21.15.90:0/3477332937 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a40319d0 con 0x7ff4b40afbd0 2024-04-25T01:18:18.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.660+0000 7ff4ba400640 1 -- 172.21.15.90:0/3477332937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4b40afbd0 msgr2=0x7ff4b40b4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:18.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.660+0000 7ff4ba400640 1 --2- 172.21.15.90:0/3477332937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4b40afbd0 0x7ff4b40b4030 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ff4a4009e50 tx=0x7ff4a402a680 comp rx=0 tx=0).stop 2024-04-25T01:18:18.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.660+0000 7ff4ba400640 1 -- 172.21.15.90:0/3477332937 shutdown_connections 2024-04-25T01:18:18.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.660+0000 7ff4ba400640 1 --2- 172.21.15.90:0/3477332937 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4b406ef30 0x7ff4b4161b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:18.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.660+0000 7ff4ba400640 1 --2- 172.21.15.90:0/3477332937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4b40afbd0 0x7ff4b40b4030 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:18.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.661+0000 7ff4ba400640 1 -- 172.21.15.90:0/3477332937 >> 172.21.15.90:0/3477332937 conn(0x7ff4b40b2570 msgr2=0x7ff4b40b2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:18.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.661+0000 7ff4ba400640 1 -- 172.21.15.90:0/3477332937 shutdown_connections 2024-04-25T01:18:18.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.661+0000 7ff4ba400640 1 -- 172.21.15.90:0/3477332937 wait complete. 2024-04-25T01:18:18.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.662+0000 7ff4ba400640 1 Processor -- start 2024-04-25T01:18:18.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.662+0000 7ff4ba400640 1 -- start start 2024-04-25T01:18:18.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.663+0000 7ff4ba400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4b406ef30 0x7ff4b414cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:18.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.663+0000 7ff4ba400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4b40afbd0 0x7ff4b40cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:18.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.663+0000 7ff4ba400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4b4162090 0x7ff4b4159540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:18.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.663+0000 7ff4ba400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b406c7f0 con 0x7ff4b406ef30 2024-04-25T01:18:18.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.663+0000 7ff4ba400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b40b4570 con 0x7ff4b40afbd0 2024-04-25T01:18:18.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.663+0000 7ff4ba400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff4b406c970 con 0x7ff4b4162090 2024-04-25T01:18:18.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.664+0000 7ff4b3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4b406ef30 0x7ff4b414cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:18.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.664+0000 7ff4b3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4b406ef30 0x7ff4b414cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41040/0 (socket says 172.21.15.90:41040) 2024-04-25T01:18:18.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.664+0000 7ff4b3400640 1 -- 172.21.15.90:0/3667159583 learned_addr learned my addr 172.21.15.90:0/3667159583 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:18.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.664+0000 7ff4b3e00640 1 --2- 172.21.15.90:0/3667159583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4b4162090 0x7ff4b4159540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:18.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.664+0000 7ff4b2a00640 1 --2- 172.21.15.90:0/3667159583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4b40afbd0 0x7ff4b40cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:18.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.664+0000 7ff4b3400640 1 -- 172.21.15.90:0/3667159583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4b4162090 msgr2=0x7ff4b4159540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:18.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.664+0000 7ff4b3400640 1 --2- 172.21.15.90:0/3667159583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4b4162090 0x7ff4b4159540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:18.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.664+0000 7ff4b3400640 1 -- 172.21.15.90:0/3667159583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4b40afbd0 msgr2=0x7ff4b40cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:18.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.664+0000 7ff4b3400640 1 --2- 172.21.15.90:0/3667159583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4b40afbd0 0x7ff4b40cdfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:18.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.664+0000 7ff4b3400640 1 -- 172.21.15.90:0/3667159583 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4b40ccc60 con 0x7ff4b406ef30 2024-04-25T01:18:18.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.665+0000 7ff4b3400640 1 --2- 172.21.15.90:0/3667159583 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4b406ef30 0x7ff4b414cbe0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7ff4a4034040 tx=0x7ff4a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:18.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.665+0000 7ff4ab400640 1 -- 172.21.15.90:0/3667159583 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a4041020 con 0x7ff4b406ef30 2024-04-25T01:18:18.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.665+0000 7ff4ab400640 1 -- 172.21.15.90:0/3667159583 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4a4037040 con 0x7ff4b406ef30 2024-04-25T01:18:18.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.665+0000 7ff4ab400640 1 -- 172.21.15.90:0/3667159583 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a4036050 con 0x7ff4b406ef30 2024-04-25T01:18:18.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.665+0000 7ff4ba400640 1 -- 172.21.15.90:0/3667159583 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4b40ccf40 con 0x7ff4b406ef30 2024-04-25T01:18:18.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.665+0000 7ff4ba400640 1 -- 172.21.15.90:0/3667159583 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4b40cd500 con 0x7ff4b406ef30 2024-04-25T01:18:18.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.666+0000 7ff4ba400640 1 -- 172.21.15.90:0/3667159583 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4b404f5a0 con 0x7ff4b406ef30 2024-04-25T01:18:18.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.667+0000 7ff4ab400640 1 -- 172.21.15.90:0/3667159583 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff4a4007c70 con 0x7ff4b406ef30 2024-04-25T01:18:18.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.668+0000 7ff4ab400640 1 --2- 172.21.15.90:0/3667159583 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff49406be30 0x7ff49406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:18.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.668+0000 7ff4ab400640 1 -- 172.21.15.90:0/3667159583 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6620+0+0 (secure 0 0 0) 0x7ff4a40abcc0 con 0x7ff4b406ef30 2024-04-25T01:18:18.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.669+0000 7ff4b2a00640 1 --2- 172.21.15.90:0/3667159583 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff49406be30 0x7ff49406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:18.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.669+0000 7ff4b2a00640 1 --2- 172.21.15.90:0/3667159583 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff49406be30 0x7ff49406e2f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff4b40b0900 tx=0x7ff4a0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:18.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.674+0000 7ff4ab400640 1 -- 172.21.15.90:0/3667159583 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff4a40b2080 con 0x7ff4b406ef30 2024-04-25T01:18:18.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.874+0000 7ff4ba400640 1 -- 172.21.15.90:0/3667159583 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7ff4b40adc70 con 0x7ff4b406ef30 2024-04-25T01:18:18.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.875+0000 7ff4ab400640 1 -- 172.21.15.90:0/3667159583 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' already exists v48) ==== 177+0+0 (secure 0 0 0) 0x7ff4a404b060 con 0x7ff4b406ef30 2024-04-25T01:18:18.858 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data' already exists 2024-04-25T01:18:18.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.877+0000 7ff4ba400640 1 -- 172.21.15.90:0/3667159583 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff49406be30 msgr2=0x7ff49406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:18.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.877+0000 7ff4ba400640 1 --2- 172.21.15.90:0/3667159583 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff49406be30 0x7ff49406e2f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff4b40b0900 tx=0x7ff4a0008040 comp rx=0 tx=0).stop 2024-04-25T01:18:18.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.877+0000 7ff4ba400640 1 -- 172.21.15.90:0/3667159583 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4b406ef30 msgr2=0x7ff4b414cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:18.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.877+0000 7ff4ba400640 1 --2- 172.21.15.90:0/3667159583 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4b406ef30 0x7ff4b414cbe0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7ff4a4034040 tx=0x7ff4a4004a40 comp rx=0 tx=0).stop 2024-04-25T01:18:18.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.877+0000 7ff4ba400640 1 -- 172.21.15.90:0/3667159583 shutdown_connections 2024-04-25T01:18:18.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.877+0000 7ff4ba400640 1 --2- 172.21.15.90:0/3667159583 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff49406be30 0x7ff49406e2f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:18.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.877+0000 7ff4ba400640 1 --2- 172.21.15.90:0/3667159583 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff4b4162090 0x7ff4b4159540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:18.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.877+0000 7ff4ba400640 1 --2- 172.21.15.90:0/3667159583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4b40afbd0 0x7ff4b40cdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:18.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.877+0000 7ff4ba400640 1 --2- 172.21.15.90:0/3667159583 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4b406ef30 0x7ff4b414cbe0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:18.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.877+0000 7ff4ba400640 1 -- 172.21.15.90:0/3667159583 >> 172.21.15.90:0/3667159583 conn(0x7ff4b40b2570 msgr2=0x7ff4b40b2950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:18.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.878+0000 7ff4ba400640 1 -- 172.21.15.90:0/3667159583 shutdown_connections 2024-04-25T01:18:18.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:18.878+0000 7ff4ba400640 1 -- 172.21.15.90:0/3667159583 wait complete. 2024-04-25T01:18:18.879 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new recovery_fs recovery_fs_metadata cephfs_data --recover --allow-dangerous-metadata-overlay 2024-04-25T01:18:19.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.106+0000 7f3984200640 1 Processor -- start 2024-04-25T01:18:19.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.107+0000 7f3984200640 1 -- start start 2024-04-25T01:18:19.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.108+0000 7f3984200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f397c166d00 0x7f397c1690f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:19.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.108+0000 7f3984200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f397c1629b0 con 0x7f397c072620 2024-04-25T01:18:19.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.108+0000 7f3984200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f397c0af170 con 0x7f397c1616f0 2024-04-25T01:18:19.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.108+0000 7f3984200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f397c071c10 con 0x7f397c166d00 2024-04-25T01:18:19.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.109+0000 7f3980e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f397c072620 0x7f397c1622a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53398/0 (socket says 172.21.15.90:53398) 2024-04-25T01:18:19.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.109+0000 7f3980e00640 1 -- 172.21.15.90:0/3372165319 learned_addr learned my addr 172.21.15.90:0/3372165319 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:19.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.109+0000 7f3982200640 1 --2- 172.21.15.90:0/3372165319 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f397c166d00 0x7f397c1690f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:19.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.109+0000 7f397be00640 1 -- 172.21.15.90:0/3372165319 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1855719534 0 0) 0x7f397c1629b0 con 0x7f397c072620 2024-04-25T01:18:19.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.109+0000 7f397be00640 1 -- 172.21.15.90:0/3372165319 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3964003520 con 0x7f397c072620 2024-04-25T01:18:19.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.109+0000 7f397be00640 1 -- 172.21.15.90:0/3372165319 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3695875787 0 0) 0x7f3964003520 con 0x7f397c072620 2024-04-25T01:18:19.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.110+0000 7f397be00640 1 -- 172.21.15.90:0/3372165319 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f397c166d00 msgr2=0x7f397c1690f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:19.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.110+0000 7f397be00640 1 --2- 172.21.15.90:0/3372165319 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f397c166d00 0x7f397c1690f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:19.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.110+0000 7f397be00640 1 -- 172.21.15.90:0/3372165319 >> v1:172.21.15.103:6789/0 conn(0x7f397c1616f0 legacy=0x7f397c161b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:19.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.110+0000 7f397be00640 1 -- 172.21.15.90:0/3372165319 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f397c071ef0 con 0x7f397c072620 2024-04-25T01:18:19.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.110+0000 7f397be00640 1 -- 172.21.15.90:0/3372165319 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f3968001670 con 0x7f397c072620 2024-04-25T01:18:19.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.110+0000 7f397be00640 1 -- 172.21.15.90:0/3372165319 >> v1:172.21.15.90:6789/0 conn(0x7f397c072620 legacy=0x7f397c1622a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:19.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.110+0000 7f397be00640 1 --2- 172.21.15.90:0/3372165319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39640039e0 0x7f3964003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:19.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.110+0000 7f397be00640 1 --2- 172.21.15.90:0/3372165319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3964004800 0x7f3964004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:19.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.110+0000 7f397be00640 1 --2- 172.21.15.90:0/3372165319 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f397c166d00 0x7f39640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:19.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.110+0000 7f397be00640 1 -- 172.21.15.90:0/3372165319 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f397c071c10 con 0x7f3964004800 2024-04-25T01:18:19.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.110+0000 7f397be00640 1 -- 172.21.15.90:0/3372165319 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f396400baa0 con 0x7f39640039e0 2024-04-25T01:18:19.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.110+0000 7f397be00640 1 -- 172.21.15.90:0/3372165319 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f396400bca0 con 0x7f397c166d00 2024-04-25T01:18:19.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.110+0000 7f3980e00640 1 --2- 172.21.15.90:0/3372165319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3964004800 0x7f3964004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:19.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.110+0000 7f3981800640 1 --2- 172.21.15.90:0/3372165319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39640039e0 0x7f3964003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:19.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.110+0000 7f3980e00640 1 -- 172.21.15.90:0/3372165319 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f397c166d00 msgr2=0x7f39640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:19.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.110+0000 7f3982200640 1 --2- 172.21.15.90:0/3372165319 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f397c166d00 0x7f39640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:19.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.110+0000 7f3980e00640 1 --2- 172.21.15.90:0/3372165319 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f397c166d00 0x7f39640051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:19.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.110+0000 7f3980e00640 1 -- 172.21.15.90:0/3372165319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39640039e0 msgr2=0x7f3964003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:19.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.110+0000 7f3980e00640 1 --2- 172.21.15.90:0/3372165319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39640039e0 0x7f3964003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:19.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.110+0000 7f3980e00640 1 -- 172.21.15.90:0/3372165319 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3964003520 con 0x7f3964004800 2024-04-25T01:18:19.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.111+0000 7f3981800640 1 --2- 172.21.15.90:0/3372165319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39640039e0 0x7f3964003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:19.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.111+0000 7f3980e00640 1 --2- 172.21.15.90:0/3372165319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3964004800 0x7f3964004c80 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f3968001eb0 tx=0x7f3964005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f00e93b059cdcae5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:19.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.112+0000 7f397be00640 1 -- 172.21.15.90:0/3372165319 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3968007d30 con 0x7f3964004800 2024-04-25T01:18:19.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.112+0000 7f397be00640 1 -- 172.21.15.90:0/3372165319 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3968005af0 con 0x7f3964004800 2024-04-25T01:18:19.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.112+0000 7f397be00640 1 -- 172.21.15.90:0/3372165319 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3968007070 con 0x7f3964004800 2024-04-25T01:18:19.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.112+0000 7f3984200640 1 -- 172.21.15.90:0/3372165319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3964004800 msgr2=0x7f3964004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:19.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.112+0000 7f3984200640 1 --2- 172.21.15.90:0/3372165319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3964004800 0x7f3964004c80 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f3968001eb0 tx=0x7f3964005830 comp rx=0 tx=0).stop 2024-04-25T01:18:19.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.112+0000 7f3982200640 1 -- 172.21.15.90:0/3372165319 reap_dead start 2024-04-25T01:18:19.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.112+0000 7f3984200640 1 -- 172.21.15.90:0/3372165319 shutdown_connections 2024-04-25T01:18:19.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.112+0000 7f3984200640 1 -- 172.21.15.90:0/3372165319 >> 172.21.15.90:0/3372165319 conn(0x7f397c0b16b0 msgr2=0x7f397c0b1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:19.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.113+0000 7f3984200640 1 -- 172.21.15.90:0/3372165319 shutdown_connections 2024-04-25T01:18:19.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.113+0000 7f3984200640 1 -- 172.21.15.90:0/3372165319 wait complete. 2024-04-25T01:18:19.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.114+0000 7f3984200640 1 Processor -- start 2024-04-25T01:18:19.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.114+0000 7f3984200640 1 -- start start 2024-04-25T01:18:19.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.115+0000 7f3984200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3964004800 0x7f397c1609a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:19.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.115+0000 7f3984200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f397c15f400 0x7f397c147800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:19.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.115+0000 7f3984200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f397c147f90 0x7f397c153bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:19.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.115+0000 7f3984200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f397c0adc40 con 0x7f3964004800 2024-04-25T01:18:19.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.115+0000 7f3984200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f397c0ad960 con 0x7f397c15f400 2024-04-25T01:18:19.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.115+0000 7f3984200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f397c0718c0 con 0x7f397c147f90 2024-04-25T01:18:19.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.115+0000 7f3981800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3964004800 0x7f397c1609a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:19.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.116+0000 7f3981800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3964004800 0x7f397c1609a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41066/0 (socket says 172.21.15.90:41066) 2024-04-25T01:18:19.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.116+0000 7f3981800640 1 -- 172.21.15.90:0/3821312926 learned_addr learned my addr 172.21.15.90:0/3821312926 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:19.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.116+0000 7f3982200640 1 --2- 172.21.15.90:0/3821312926 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f397c147f90 0x7f397c153bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:19.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.116+0000 7f3981800640 1 -- 172.21.15.90:0/3821312926 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f397c147f90 msgr2=0x7f397c153bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:19.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.116+0000 7f3981800640 1 --2- 172.21.15.90:0/3821312926 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f397c147f90 0x7f397c153bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:19.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.116+0000 7f3981800640 1 -- 172.21.15.90:0/3821312926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f397c15f400 msgr2=0x7f397c147800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:19.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.116+0000 7f3981800640 1 --2- 172.21.15.90:0/3821312926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f397c15f400 0x7f397c147800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:19.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.116+0000 7f3981800640 1 -- 172.21.15.90:0/3821312926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f397c160ee0 con 0x7f3964004800 2024-04-25T01:18:19.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.116+0000 7f3981800640 1 --2- 172.21.15.90:0/3821312926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3964004800 0x7f397c1609a0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f397000d770 tx=0x7f397000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:19.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.116+0000 7f3979600640 1 -- 172.21.15.90:0/3821312926 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3970005270 con 0x7f3964004800 2024-04-25T01:18:19.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.117+0000 7f3979600640 1 -- 172.21.15.90:0/3821312926 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3970017070 con 0x7f3964004800 2024-04-25T01:18:19.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.117+0000 7f3979600640 1 -- 172.21.15.90:0/3821312926 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f397000b6b0 con 0x7f3964004800 2024-04-25T01:18:19.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.117+0000 7f3984200640 1 -- 172.21.15.90:0/3821312926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f397c147d40 con 0x7f3964004800 2024-04-25T01:18:19.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.117+0000 7f3984200640 1 -- 172.21.15.90:0/3821312926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f397c15fbe0 con 0x7f3964004800 2024-04-25T01:18:19.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.118+0000 7f3984200640 1 -- 172.21.15.90:0/3821312926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f397c068190 con 0x7f3964004800 2024-04-25T01:18:19.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.119+0000 7f3979600640 1 -- 172.21.15.90:0/3821312926 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f397000b850 con 0x7f3964004800 2024-04-25T01:18:19.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.120+0000 7f3979600640 1 --2- 172.21.15.90:0/3821312926 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f395c06be30 0x7f395c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:19.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.120+0000 7f3979600640 1 -- 172.21.15.90:0/3821312926 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6620+0+0 (secure 0 0 0) 0x7f397008b0b0 con 0x7f3964004800 2024-04-25T01:18:19.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.120+0000 7f3980e00640 1 --2- 172.21.15.90:0/3821312926 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f395c06be30 0x7f395c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:19.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.121+0000 7f3980e00640 1 --2- 172.21.15.90:0/3821312926 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f395c06be30 0x7f395c06e2f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3968032000 tx=0x7f3968009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:19.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.125+0000 7f3979600640 1 -- 172.21.15.90:0/3821312926 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f3970019310 con 0x7f3964004800 2024-04-25T01:18:19.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.369+0000 7f3984200640 1 -- 172.21.15.90:0/3821312926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "recovery_fs", "metadata": "recovery_fs_metadata", "data": "cephfs_data", "recover": true, "allow_dangerous_metadata_overlay": true} v 0) -- 0x7f397c06dad0 con 0x7f3964004800 2024-04-25T01:18:19.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.440+0000 7f3979600640 1 -- 172.21.15.90:0/3821312926 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "recovery_fs", "metadata": "recovery_fs_metadata", "data": "cephfs_data", "recover": true, "allow_dangerous_metadata_overlay": true}]=0 new fs with metadata pool 7 and data pool 6 v21) ==== 241+0+0 (secure 0 0 0) 0x7f3970001230 con 0x7f3964004800 2024-04-25T01:18:19.424 INFO:teuthology.orchestra.run.smithi090.stderr:new fs with metadata pool 7 and data pool 6 2024-04-25T01:18:19.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.443+0000 7f3984200640 1 -- 172.21.15.90:0/3821312926 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f395c06be30 msgr2=0x7f395c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:19.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.443+0000 7f3984200640 1 --2- 172.21.15.90:0/3821312926 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f395c06be30 0x7f395c06e2f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3968032000 tx=0x7f3968009030 comp rx=0 tx=0).stop 2024-04-25T01:18:19.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.443+0000 7f3984200640 1 -- 172.21.15.90:0/3821312926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3964004800 msgr2=0x7f397c1609a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:19.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.443+0000 7f3984200640 1 --2- 172.21.15.90:0/3821312926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3964004800 0x7f397c1609a0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f397000d770 tx=0x7f397000dc30 comp rx=0 tx=0).stop 2024-04-25T01:18:19.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.443+0000 7f3984200640 1 -- 172.21.15.90:0/3821312926 shutdown_connections 2024-04-25T01:18:19.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.443+0000 7f3984200640 1 --2- 172.21.15.90:0/3821312926 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f395c06be30 0x7f395c06e2f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:19.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.443+0000 7f3984200640 1 --2- 172.21.15.90:0/3821312926 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f397c147f90 0x7f397c153bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:19.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.443+0000 7f3984200640 1 --2- 172.21.15.90:0/3821312926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f397c15f400 0x7f397c147800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:19.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.443+0000 7f3984200640 1 --2- 172.21.15.90:0/3821312926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3964004800 0x7f397c1609a0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:19.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.443+0000 7f3984200640 1 -- 172.21.15.90:0/3821312926 >> 172.21.15.90:0/3821312926 conn(0x7f397c0b16b0 msgr2=0x7f397c06e2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:19.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.443+0000 7f3984200640 1 -- 172.21.15.90:0/3821312926 shutdown_connections 2024-04-25T01:18:19.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.443+0000 7f3984200640 1 -- 172.21.15.90:0/3821312926 wait complete. 2024-04-25T01:18:19.444 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:18:19.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.639+0000 7f80b0800640 1 Processor -- start 2024-04-25T01:18:19.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.639+0000 7f80b0800640 1 -- start start 2024-04-25T01:18:19.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.640+0000 7f80b0800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f80a8145640 0x7f80a815ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:19.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.640+0000 7f80b0800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f80a80b2a20 0x7f80a80cfc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:19.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.640+0000 7f80b0800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80a806cbc0 0x7f80a80d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:19.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.641+0000 7f80b0800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f80a80cdf70 con 0x7f80a8145640 2024-04-25T01:18:19.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.641+0000 7f80b0800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f80a815d360 con 0x7f80a80b2a20 2024-04-25T01:18:19.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.641+0000 7f80b0800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f80a8145a20 con 0x7f80a806cbc0 2024-04-25T01:18:19.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.641+0000 7f80ade00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f80a8145640 0x7f80a815ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:19.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.641+0000 7f80ade00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f80a8145640 0x7f80a815ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41072/0 (socket says 172.21.15.90:41072) 2024-04-25T01:18:19.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.641+0000 7f80ade00640 1 -- 172.21.15.90:0/1068749294 learned_addr learned my addr 172.21.15.90:0/1068749294 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:19.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.641+0000 7f80ad400640 1 --2- 172.21.15.90:0/1068749294 >> v2:172.21.15.103:3300/0 conn(0x7f80a80b2a20 0x7f80a80cfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:19.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.641+0000 7f80ae800640 1 --2- 172.21.15.90:0/1068749294 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80a806cbc0 0x7f80a80d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:19.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.641+0000 7f80ade00640 1 -- 172.21.15.90:0/1068749294 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80a806cbc0 msgr2=0x7f80a80d01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:19.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.641+0000 7f80ade00640 1 --2- 172.21.15.90:0/1068749294 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80a806cbc0 0x7f80a80d01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:19.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.641+0000 7f80ade00640 1 -- 172.21.15.90:0/1068749294 >> v2:172.21.15.103:3300/0 conn(0x7f80a80b2a20 msgr2=0x7f80a80cfc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:19.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.641+0000 7f80ade00640 1 --2- 172.21.15.90:0/1068749294 >> v2:172.21.15.103:3300/0 conn(0x7f80a80b2a20 0x7f80a80cfc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:19.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.641+0000 7f80ade00640 1 -- 172.21.15.90:0/1068749294 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80a806d140 con 0x7f80a8145640 2024-04-25T01:18:19.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.643+0000 7f80ade00640 1 --2- 172.21.15.90:0/1068749294 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80a8145640 0x7f80a815ce20 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f80a8072f30 tx=0x7f809c02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2ff616b4cae9e50 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:19.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.643+0000 7f80aca00640 1 -- 172.21.15.90:0/1068749294 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f809c037070 con 0x7f80a8145640 2024-04-25T01:18:19.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.643+0000 7f80aca00640 1 -- 172.21.15.90:0/1068749294 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f809c032440 con 0x7f80a8145640 2024-04-25T01:18:19.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.643+0000 7f80aca00640 1 -- 172.21.15.90:0/1068749294 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f809c032810 con 0x7f80a8145640 2024-04-25T01:18:19.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.644+0000 7f80b0800640 1 -- 172.21.15.90:0/1068749294 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80a8145640 msgr2=0x7f80a815ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:19.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.644+0000 7f80b0800640 1 --2- 172.21.15.90:0/1068749294 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80a8145640 0x7f80a815ce20 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f80a8072f30 tx=0x7f809c02a140 comp rx=0 tx=0).stop 2024-04-25T01:18:19.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.644+0000 7f80b0800640 1 -- 172.21.15.90:0/1068749294 shutdown_connections 2024-04-25T01:18:19.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.644+0000 7f80b0800640 1 --2- 172.21.15.90:0/1068749294 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80a806cbc0 0x7f80a80d01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:19.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.644+0000 7f80b0800640 1 --2- 172.21.15.90:0/1068749294 >> v2:172.21.15.103:3300/0 conn(0x7f80a80b2a20 0x7f80a80cfc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:19.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.644+0000 7f80b0800640 1 --2- 172.21.15.90:0/1068749294 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80a8145640 0x7f80a815ce20 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:19.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.644+0000 7f80b0800640 1 -- 172.21.15.90:0/1068749294 >> 172.21.15.90:0/1068749294 conn(0x7f80a806b320 msgr2=0x7f80a806b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:19.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.644+0000 7f80b0800640 1 -- 172.21.15.90:0/1068749294 shutdown_connections 2024-04-25T01:18:19.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.644+0000 7f80b0800640 1 -- 172.21.15.90:0/1068749294 wait complete. 2024-04-25T01:18:19.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.646+0000 7f80b0800640 1 Processor -- start 2024-04-25T01:18:19.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.646+0000 7f80b0800640 1 -- start start 2024-04-25T01:18:19.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.647+0000 7f80b0800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80a806cbc0 0x7f80a813afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:19.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.647+0000 7f80b0800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80a80b2a20 0x7f80a81f3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:19.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.647+0000 7f80b0800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80a8145640 0x7f80a8142300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:19.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.647+0000 7f80b0800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f80a806d9e0 con 0x7f80a80b2a20 2024-04-25T01:18:19.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.647+0000 7f80b0800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f80a806d700 con 0x7f80a8145640 2024-04-25T01:18:19.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.647+0000 7f80b0800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f80a815f9c0 con 0x7f80a806cbc0 2024-04-25T01:18:19.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.648+0000 7f80ad400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80a80b2a20 0x7f80a81f3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:19.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.648+0000 7f80ad400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80a80b2a20 0x7f80a81f3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41080/0 (socket says 172.21.15.90:41080) 2024-04-25T01:18:19.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.648+0000 7f80ad400640 1 -- 172.21.15.90:0/225383488 learned_addr learned my addr 172.21.15.90:0/225383488 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:19.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.648+0000 7f80ae800640 1 --2- 172.21.15.90:0/225383488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80a8145640 0x7f80a8142300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:19.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.648+0000 7f80ade00640 1 --2- 172.21.15.90:0/225383488 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80a806cbc0 0x7f80a813afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:19.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.648+0000 7f80ad400640 1 -- 172.21.15.90:0/225383488 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80a806cbc0 msgr2=0x7f80a813afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:19.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.648+0000 7f80ad400640 1 --2- 172.21.15.90:0/225383488 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80a806cbc0 0x7f80a813afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:19.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.648+0000 7f80ad400640 1 -- 172.21.15.90:0/225383488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80a8145640 msgr2=0x7f80a8142300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:19.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.648+0000 7f80ad400640 1 --2- 172.21.15.90:0/225383488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80a8145640 0x7f80a8142300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:19.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.648+0000 7f80ad400640 1 -- 172.21.15.90:0/225383488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80a813b520 con 0x7f80a80b2a20 2024-04-25T01:18:19.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.649+0000 7f80ad400640 1 --2- 172.21.15.90:0/225383488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80a80b2a20 0x7f80a81f3d00 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f809800c820 tx=0x7f809800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:19.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.649+0000 7f80a2000640 1 -- 172.21.15.90:0/225383488 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8098015070 con 0x7f80a80b2a20 2024-04-25T01:18:19.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.649+0000 7f80a2000640 1 -- 172.21.15.90:0/225383488 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8098004240 con 0x7f80a80b2a20 2024-04-25T01:18:19.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.649+0000 7f80a2000640 1 -- 172.21.15.90:0/225383488 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f809800ce10 con 0x7f80a80b2a20 2024-04-25T01:18:19.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.649+0000 7f80b0800640 1 -- 172.21.15.90:0/225383488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80a815a860 con 0x7f80a80b2a20 2024-04-25T01:18:19.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.649+0000 7f80b0800640 1 -- 172.21.15.90:0/225383488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80a815ae20 con 0x7f80a80b2a20 2024-04-25T01:18:19.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.650+0000 7f80b0800640 1 -- 172.21.15.90:0/225383488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80a8068190 con 0x7f80a80b2a20 2024-04-25T01:18:19.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.651+0000 7f80a2000640 1 -- 172.21.15.90:0/225383488 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f80980077d0 con 0x7f80a80b2a20 2024-04-25T01:18:19.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.653+0000 7f80a2000640 1 --2- 172.21.15.90:0/225383488 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f808c06be30 0x7f808c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:19.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.653+0000 7f80a2000640 1 -- 172.21.15.90:0/225383488 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 6698+0+0 (secure 0 0 0) 0x7f809808b0d0 con 0x7f80a80b2a20 2024-04-25T01:18:19.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.653+0000 7f80ade00640 1 --2- 172.21.15.90:0/225383488 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f808c06be30 0x7f808c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:19.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.654+0000 7f80ade00640 1 --2- 172.21.15.90:0/225383488 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f808c06be30 0x7f808c06e2f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f809c034000 tx=0x7f809c02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:19.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.657+0000 7f80a2000640 1 -- 172.21.15.90:0/225383488 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f80980596a0 con 0x7f80a80b2a20 2024-04-25T01:18:19.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.856+0000 7f80b0800640 1 -- 172.21.15.90:0/225383488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f80a815ff20 con 0x7f80a80b2a20 2024-04-25T01:18:19.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.858+0000 7f80a2000640 1 -- 172.21.15.90:0/225383488 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) ==== 74+0+18960 (secure 0 0 0) 0x7f8098058fa0 con 0x7f80a80b2a20 2024-04-25T01:18:19.841 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:19.842 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":49,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:18:19.370752+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:18:00.504992+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T01:18:01.714313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"recovery_fs_metadata","create_time":"2024-04-25T01:18:18.384671+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.16","mappings":[{"from":4,"to":2}]},{"pgid":"6.20","mappings":[{"from":4,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:6837/431390375":"2024-04-26T01:18:15.602554+0000","172.21.15.90:6836/431390375":"2024-04-26T01:18:15.602554+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:18:19.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.860+0000 7f80b0800640 1 -- 172.21.15.90:0/225383488 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f808c06be30 msgr2=0x7f808c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:19.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.860+0000 7f80b0800640 1 --2- 172.21.15.90:0/225383488 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f808c06be30 0x7f808c06e2f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f809c034000 tx=0x7f809c02e000 comp rx=0 tx=0).stop 2024-04-25T01:18:19.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.860+0000 7f80b0800640 1 -- 172.21.15.90:0/225383488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80a80b2a20 msgr2=0x7f80a81f3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:19.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.860+0000 7f80b0800640 1 --2- 172.21.15.90:0/225383488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80a80b2a20 0x7f80a81f3d00 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f809800c820 tx=0x7f809800cce0 comp rx=0 tx=0).stop 2024-04-25T01:18:19.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.861+0000 7f80b0800640 1 -- 172.21.15.90:0/225383488 shutdown_connections 2024-04-25T01:18:19.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.861+0000 7f80b0800640 1 --2- 172.21.15.90:0/225383488 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f808c06be30 0x7f808c06e2f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:19.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.861+0000 7f80b0800640 1 --2- 172.21.15.90:0/225383488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80a8145640 0x7f80a8142300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:19.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.861+0000 7f80b0800640 1 --2- 172.21.15.90:0/225383488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80a80b2a20 0x7f80a81f3d00 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:19.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.861+0000 7f80b0800640 1 --2- 172.21.15.90:0/225383488 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f80a806cbc0 0x7f80a813afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:19.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.861+0000 7f80b0800640 1 -- 172.21.15.90:0/225383488 >> 172.21.15.90:0/225383488 conn(0x7f80a806b320 msgr2=0x7f80a81602d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:19.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.861+0000 7f80b0800640 1 -- 172.21.15.90:0/225383488 shutdown_connections 2024-04-25T01:18:19.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:19.861+0000 7f80b0800640 1 -- 172.21.15.90:0/225383488 wait complete. 2024-04-25T01:18:19.864 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:18:19.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:18:19.990 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:18:20.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.082+0000 7f1be9e00640 1 Processor -- start 2024-04-25T01:18:20.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.082+0000 7f1be9e00640 1 -- start start 2024-04-25T01:18:20.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.083+0000 7f1be9e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1be4161800 0x7f1be406d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:20.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.083+0000 7f1be9e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1be40d2160 0x7f1be40d25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:20.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.083+0000 7f1be9e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1be40d4b60 con 0x7f1be406bb50 2024-04-25T01:18:20.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.083+0000 7f1be9e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f1be40d4e40 con 0x7f1be40d2160 2024-04-25T01:18:20.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.083+0000 7f1be9e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1be40d2b00 con 0x7f1be4161800 2024-04-25T01:18:20.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.083+0000 7f1bda000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1be40d2160 0x7f1be40d25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:20.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.083+0000 7f1be3400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f1be406bb50 0x7f1be406bfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53412/0 (socket says 172.21.15.90:53412) 2024-04-25T01:18:20.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.083+0000 7f1be3400640 1 -- 172.21.15.90:0/2825364089 learned_addr learned my addr 172.21.15.90:0/2825364089 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:20.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.084+0000 7f1be2a00640 1 --2- 172.21.15.90:0/2825364089 >> v2:172.21.15.103:3300/0 conn(0x7f1be4161800 0x7f1be406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:20.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.084+0000 7f1bda000640 1 -- 172.21.15.90:0/2825364089 >> v2:172.21.15.103:3300/0 conn(0x7f1be4161800 msgr2=0x7f1be406d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:20.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.084+0000 7f1bda000640 1 --2- 172.21.15.90:0/2825364089 >> v2:172.21.15.103:3300/0 conn(0x7f1be4161800 0x7f1be406d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:20.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.084+0000 7f1bda000640 1 -- 172.21.15.90:0/2825364089 >> v1:172.21.15.90:6789/0 conn(0x7f1be406bb50 legacy=0x7f1be406bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:20.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.084+0000 7f1bda000640 1 -- 172.21.15.90:0/2825364089 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1be40b1800 con 0x7f1be40d2160 2024-04-25T01:18:20.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.084+0000 7f1be2000640 1 -- 172.21.15.90:0/2825364089 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1567667958 0 0) 0x7f1be40d4b60 con 0x7f1be406bb50 2024-04-25T01:18:20.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.085+0000 7f1bda000640 1 --2- 172.21.15.90:0/2825364089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1be40d2160 0x7f1be40d25c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f1be4073340 tx=0x7f1bcc02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9807027b8875cf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:20.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.085+0000 7f1be2000640 1 -- 172.21.15.90:0/2825364089 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc037070 con 0x7f1be40d2160 2024-04-25T01:18:20.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.085+0000 7f1be2000640 1 -- 172.21.15.90:0/2825364089 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bcc032440 con 0x7f1be40d2160 2024-04-25T01:18:20.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.085+0000 7f1be2000640 1 -- 172.21.15.90:0/2825364089 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc032810 con 0x7f1be40d2160 2024-04-25T01:18:20.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.086+0000 7f1be9e00640 1 -- 172.21.15.90:0/2825364089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1be40d2160 msgr2=0x7f1be40d25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:20.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.086+0000 7f1be9e00640 1 --2- 172.21.15.90:0/2825364089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1be40d2160 0x7f1be40d25c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f1be4073340 tx=0x7f1bcc02a230 comp rx=0 tx=0).stop 2024-04-25T01:18:20.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.086+0000 7f1be9e00640 1 -- 172.21.15.90:0/2825364089 shutdown_connections 2024-04-25T01:18:20.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.086+0000 7f1be9e00640 1 --2- 172.21.15.90:0/2825364089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1be40d2160 0x7f1be40d25c0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:20.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.086+0000 7f1be9e00640 1 --2- 172.21.15.90:0/2825364089 >> v2:172.21.15.103:3300/0 conn(0x7f1be4161800 0x7f1be406d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:20.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.086+0000 7f1be9e00640 1 -- 172.21.15.90:0/2825364089 >> 172.21.15.90:0/2825364089 conn(0x7f1be40d3ea0 msgr2=0x7f1be40d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:20.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.086+0000 7f1be9e00640 1 -- 172.21.15.90:0/2825364089 shutdown_connections 2024-04-25T01:18:20.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.086+0000 7f1be9e00640 1 -- 172.21.15.90:0/2825364089 wait complete. 2024-04-25T01:18:20.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.088+0000 7f1be9e00640 1 Processor -- start 2024-04-25T01:18:20.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.088+0000 7f1be9e00640 1 -- start start 2024-04-25T01:18:20.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.089+0000 7f1be9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1be406bb50 0x7f1be414f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:20.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.089+0000 7f1be9e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1be40d2160 0x7f1be415b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:20.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.089+0000 7f1be9e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1be4161800 0x7f1be41f3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:20.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.089+0000 7f1be9e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1be406cfb0 con 0x7f1be406bb50 2024-04-25T01:18:20.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.089+0000 7f1be9e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1be40b1b70 con 0x7f1be40d2160 2024-04-25T01:18:20.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.089+0000 7f1be9e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1be406d130 con 0x7f1be4161800 2024-04-25T01:18:20.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.089+0000 7f1be2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1be406bb50 0x7f1be414f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:20.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.089+0000 7f1be2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1be406bb50 0x7f1be414f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41090/0 (socket says 172.21.15.90:41090) 2024-04-25T01:18:20.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.089+0000 7f1be2a00640 1 -- 172.21.15.90:0/544364780 learned_addr learned my addr 172.21.15.90:0/544364780 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:20.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.090+0000 7f1be3400640 1 --2- 172.21.15.90:0/544364780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1be4161800 0x7f1be41f3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:20.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.090+0000 7f1bda000640 1 --2- 172.21.15.90:0/544364780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1be40d2160 0x7f1be415b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:20.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.090+0000 7f1be2a00640 1 -- 172.21.15.90:0/544364780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1be4161800 msgr2=0x7f1be41f3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:20.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.090+0000 7f1be2a00640 1 --2- 172.21.15.90:0/544364780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1be4161800 0x7f1be41f3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:20.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.090+0000 7f1be2a00640 1 -- 172.21.15.90:0/544364780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1be40d2160 msgr2=0x7f1be415b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:20.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.090+0000 7f1be2a00640 1 --2- 172.21.15.90:0/544364780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1be40d2160 0x7f1be415b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:20.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.090+0000 7f1be2a00640 1 -- 172.21.15.90:0/544364780 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1be40c9120 con 0x7f1be406bb50 2024-04-25T01:18:20.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.090+0000 7f1bda000640 1 --2- 172.21.15.90:0/544364780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1be40d2160 0x7f1be415b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:20.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.090+0000 7f1be2a00640 1 --2- 172.21.15.90:0/544364780 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1be406bb50 0x7f1be414f380 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1bd400c7f0 tx=0x7f1bd400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:20.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.090+0000 7f1bdbe00640 1 -- 172.21.15.90:0/544364780 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd4015070 con 0x7f1be406bb50 2024-04-25T01:18:20.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.090+0000 7f1bdbe00640 1 -- 172.21.15.90:0/544364780 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bd4004220 con 0x7f1be406bb50 2024-04-25T01:18:20.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.090+0000 7f1bdbe00640 1 -- 172.21.15.90:0/544364780 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd4007620 con 0x7f1be406bb50 2024-04-25T01:18:20.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.090+0000 7f1be9e00640 1 -- 172.21.15.90:0/544364780 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1be40c9400 con 0x7f1be406bb50 2024-04-25T01:18:20.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.091+0000 7f1be9e00640 1 -- 172.21.15.90:0/544364780 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1be415be80 con 0x7f1be406bb50 2024-04-25T01:18:20.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.092+0000 7f1be9e00640 1 -- 172.21.15.90:0/544364780 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1be4068190 con 0x7f1be406bb50 2024-04-25T01:18:20.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.093+0000 7f1bdbe00640 1 -- 172.21.15.90:0/544364780 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1bd40079d0 con 0x7f1be406bb50 2024-04-25T01:18:20.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.094+0000 7f1bdbe00640 1 --2- 172.21.15.90:0/544364780 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1bc006be30 0x7f1bc006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:20.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.094+0000 7f1bdbe00640 1 -- 172.21.15.90:0/544364780 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 6698+0+0 (secure 0 0 0) 0x7f1bd408b600 con 0x7f1be406bb50 2024-04-25T01:18:20.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.094+0000 7f1bda000640 1 --2- 172.21.15.90:0/544364780 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1bc006be30 0x7f1bc006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:20.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.095+0000 7f1bda000640 1 --2- 172.21.15.90:0/544364780 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1bc006be30 0x7f1bc006e2f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc034000 tx=0x7f1bcc02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:20.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.098+0000 7f1bdbe00640 1 -- 172.21.15.90:0/544364780 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1bd4059b30 con 0x7f1be406bb50 2024-04-25T01:18:20.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.311+0000 7f1be9e00640 1 -- 172.21.15.90:0/544364780 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1be408dfc0 con 0x7f1be406bb50 2024-04-25T01:18:20.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.315+0000 7f1bdbe00640 1 -- 172.21.15.90:0/544364780 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) ==== 74+0+18960 (secure 0 0 0) 0x7f1bd4059430 con 0x7f1be406bb50 2024-04-25T01:18:20.298 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:20.299 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":49,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:18:19.370752+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:18:00.504992+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T01:18:01.714313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"recovery_fs_metadata","create_time":"2024-04-25T01:18:18.384671+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.16","mappings":[{"from":4,"to":2}]},{"pgid":"6.20","mappings":[{"from":4,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:6837/431390375":"2024-04-26T01:18:15.602554+0000","172.21.15.90:6836/431390375":"2024-04-26T01:18:15.602554+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:18:20.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.317+0000 7f1be9e00640 1 -- 172.21.15.90:0/544364780 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1bc006be30 msgr2=0x7f1bc006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:20.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.317+0000 7f1be9e00640 1 --2- 172.21.15.90:0/544364780 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1bc006be30 0x7f1bc006e2f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc034000 tx=0x7f1bcc02e000 comp rx=0 tx=0).stop 2024-04-25T01:18:20.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.317+0000 7f1be9e00640 1 -- 172.21.15.90:0/544364780 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1be406bb50 msgr2=0x7f1be414f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:20.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.317+0000 7f1be9e00640 1 --2- 172.21.15.90:0/544364780 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1be406bb50 0x7f1be414f380 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1bd400c7f0 tx=0x7f1bd400ccb0 comp rx=0 tx=0).stop 2024-04-25T01:18:20.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.317+0000 7f1be9e00640 1 -- 172.21.15.90:0/544364780 shutdown_connections 2024-04-25T01:18:20.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.317+0000 7f1be9e00640 1 --2- 172.21.15.90:0/544364780 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1bc006be30 0x7f1bc006e2f0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:20.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.317+0000 7f1be9e00640 1 --2- 172.21.15.90:0/544364780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1be4161800 0x7f1be41f3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:20.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.317+0000 7f1be9e00640 1 --2- 172.21.15.90:0/544364780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1be40d2160 0x7f1be415b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:20.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.317+0000 7f1be9e00640 1 --2- 172.21.15.90:0/544364780 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1be406bb50 0x7f1be414f380 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:20.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.317+0000 7f1be9e00640 1 -- 172.21.15.90:0/544364780 >> 172.21.15.90:0/544364780 conn(0x7f1be40d3ea0 msgr2=0x7f1be415e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:20.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.318+0000 7f1be9e00640 1 -- 172.21.15.90:0/544364780 shutdown_connections 2024-04-25T01:18:20.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.318+0000 7f1be9e00640 1 -- 172.21.15.90:0/544364780 wait complete. 2024-04-25T01:18:20.319 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set recovery_fs standby_count_wanted 0 2024-04-25T01:18:20.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.584+0000 7fa827200640 1 Processor -- start 2024-04-25T01:18:20.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.584+0000 7fa827200640 1 -- start start 2024-04-25T01:18:20.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.585+0000 7fa827200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa820070ba0 0x7fa82006d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:20.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.586+0000 7fa827200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa820142950 con 0x7fa8200720d0 2024-04-25T01:18:20.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.586+0000 7fa827200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa820071030 con 0x7fa820142570 2024-04-25T01:18:20.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.586+0000 7fa827200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa820071c40 con 0x7fa820070ba0 2024-04-25T01:18:20.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.586+0000 7fa825200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa8200720d0 0x7fa82015fc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53418/0 (socket says 172.21.15.90:53418) 2024-04-25T01:18:20.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.586+0000 7fa825200640 1 -- 172.21.15.90:0/3601450132 learned_addr learned my addr 172.21.15.90:0/3601450132 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:20.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.586+0000 7fa81f400640 1 --2- 172.21.15.90:0/3601450132 >> v2:172.21.15.103:3300/0 conn(0x7fa820070ba0 0x7fa82006d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:20.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.586+0000 7fa81ea00640 1 -- 172.21.15.90:0/3601450132 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3827218278 0 0) 0x7fa820142950 con 0x7fa8200720d0 2024-04-25T01:18:20.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.586+0000 7fa81ea00640 1 -- 172.21.15.90:0/3601450132 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa804003520 con 0x7fa8200720d0 2024-04-25T01:18:20.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.587+0000 7fa81ea00640 1 -- 172.21.15.90:0/3601450132 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1859175952 0 0) 0x7fa804003520 con 0x7fa8200720d0 2024-04-25T01:18:20.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.587+0000 7fa81ea00640 1 -- 172.21.15.90:0/3601450132 >> v2:172.21.15.103:3300/0 conn(0x7fa820070ba0 msgr2=0x7fa82006d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:20.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.587+0000 7fa81ea00640 1 --2- 172.21.15.90:0/3601450132 >> v2:172.21.15.103:3300/0 conn(0x7fa820070ba0 0x7fa82006d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:20.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.587+0000 7fa81ea00640 1 -- 172.21.15.90:0/3601450132 >> v1:172.21.15.103:6789/0 conn(0x7fa820142570 legacy=0x7fa82006cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:20.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.587+0000 7fa81ea00640 1 -- 172.21.15.90:0/3601450132 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa820071310 con 0x7fa8200720d0 2024-04-25T01:18:20.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.587+0000 7fa81ea00640 1 -- 172.21.15.90:0/3601450132 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7fa818001670 con 0x7fa8200720d0 2024-04-25T01:18:20.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.587+0000 7fa81ea00640 1 -- 172.21.15.90:0/3601450132 >> v1:172.21.15.90:6789/0 conn(0x7fa8200720d0 legacy=0x7fa82015fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:20.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.587+0000 7fa81ea00640 1 --2- 172.21.15.90:0/3601450132 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa804003a00 0x7fa804003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:20.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.587+0000 7fa81ea00640 1 --2- 172.21.15.90:0/3601450132 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa804004800 0x7fa804004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:20.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.587+0000 7fa81ea00640 1 --2- 172.21.15.90:0/3601450132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8040051c0 0x7fa80400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:20.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.587+0000 7fa81ea00640 1 -- 172.21.15.90:0/3601450132 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa820071c40 con 0x7fa8040051c0 2024-04-25T01:18:20.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.587+0000 7fa81ea00640 1 -- 172.21.15.90:0/3601450132 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa80400c0c0 con 0x7fa804003a00 2024-04-25T01:18:20.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.587+0000 7fa81ea00640 1 -- 172.21.15.90:0/3601450132 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa80400c2c0 con 0x7fa804004800 2024-04-25T01:18:20.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.587+0000 7fa825200640 1 --2- 172.21.15.90:0/3601450132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8040051c0 0x7fa80400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:20.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.587+0000 7fa81fe00640 1 --2- 172.21.15.90:0/3601450132 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa804003a00 0x7fa804003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:20.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.588+0000 7fa81f400640 1 --2- 172.21.15.90:0/3601450132 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa804004800 0x7fa804004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:20.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.588+0000 7fa825200640 1 -- 172.21.15.90:0/3601450132 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa804004800 msgr2=0x7fa804004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:20.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.588+0000 7fa825200640 1 --2- 172.21.15.90:0/3601450132 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa804004800 0x7fa804004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:20.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.588+0000 7fa825200640 1 -- 172.21.15.90:0/3601450132 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa804003a00 msgr2=0x7fa804003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:20.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.588+0000 7fa825200640 1 --2- 172.21.15.90:0/3601450132 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa804003a00 0x7fa804003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:20.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.588+0000 7fa825200640 1 -- 172.21.15.90:0/3601450132 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa804003520 con 0x7fa8040051c0 2024-04-25T01:18:20.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.589+0000 7fa825200640 1 -- 172.21.15.90:0/3601450132 reap_dead start 2024-04-25T01:18:20.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.589+0000 7fa825200640 1 --2- 172.21.15.90:0/3601450132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8040051c0 0x7fa80400b9c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa81800c110 tx=0x7fa804005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffdd6006be252f9d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:20.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.589+0000 7fa81ea00640 1 -- 172.21.15.90:0/3601450132 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa818007c40 con 0x7fa8040051c0 2024-04-25T01:18:20.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.589+0000 7fa81ea00640 1 -- 172.21.15.90:0/3601450132 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa818005b10 con 0x7fa8040051c0 2024-04-25T01:18:20.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.589+0000 7fa81ea00640 1 -- 172.21.15.90:0/3601450132 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa818007070 con 0x7fa8040051c0 2024-04-25T01:18:20.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.589+0000 7fa827200640 1 -- 172.21.15.90:0/3601450132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8040051c0 msgr2=0x7fa80400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:20.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.589+0000 7fa827200640 1 --2- 172.21.15.90:0/3601450132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8040051c0 0x7fa80400b9c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa81800c110 tx=0x7fa804005830 comp rx=0 tx=0).stop 2024-04-25T01:18:20.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.590+0000 7fa827200640 1 -- 172.21.15.90:0/3601450132 shutdown_connections 2024-04-25T01:18:20.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.590+0000 7fa827200640 1 --2- 172.21.15.90:0/3601450132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8040051c0 0x7fa80400b9c0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:20.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.590+0000 7fa827200640 1 -- 172.21.15.90:0/3601450132 >> 172.21.15.90:0/3601450132 conn(0x7fa82006b140 msgr2=0x7fa82006b550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:20.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.590+0000 7fa827200640 1 -- 172.21.15.90:0/3601450132 shutdown_connections 2024-04-25T01:18:20.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.590+0000 7fa827200640 1 -- 172.21.15.90:0/3601450132 wait complete. 2024-04-25T01:18:20.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.591+0000 7fa827200640 1 Processor -- start 2024-04-25T01:18:20.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.592+0000 7fa827200640 1 -- start start 2024-04-25T01:18:20.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.592+0000 7fa827200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8040051c0 0x7fa820159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:20.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.592+0000 7fa827200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa8201f36a0 0x7fa8201f74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:20.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.592+0000 7fa827200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8201f3b00 0x7fa8201f7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:20.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.593+0000 7fa827200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa820072c60 con 0x7fa8040051c0 2024-04-25T01:18:20.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.593+0000 7fa827200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa8200718d0 con 0x7fa8201f3b00 2024-04-25T01:18:20.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.593+0000 7fa827200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa820072de0 con 0x7fa8201f36a0 2024-04-25T01:18:20.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.593+0000 7fa81fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8040051c0 0x7fa820159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:20.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.593+0000 7fa81fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8040051c0 0x7fa820159cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41106/0 (socket says 172.21.15.90:41106) 2024-04-25T01:18:20.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.593+0000 7fa81fe00640 1 -- 172.21.15.90:0/678251832 learned_addr learned my addr 172.21.15.90:0/678251832 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:20.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.593+0000 7fa81fe00640 1 -- 172.21.15.90:0/678251832 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa8201f36a0 msgr2=0x7fa8201f74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:20.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.593+0000 7fa81fe00640 1 --2- 172.21.15.90:0/678251832 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa8201f36a0 0x7fa8201f74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:20.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.593+0000 7fa81fe00640 1 -- 172.21.15.90:0/678251832 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8201f3b00 msgr2=0x7fa8201f7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:20.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.593+0000 7fa81fe00640 1 --2- 172.21.15.90:0/678251832 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8201f3b00 0x7fa8201f7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:20.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.593+0000 7fa81fe00640 1 -- 172.21.15.90:0/678251832 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8201f8320 con 0x7fa8040051c0 2024-04-25T01:18:20.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.593+0000 7fa81fe00640 1 --2- 172.21.15.90:0/678251832 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8040051c0 0x7fa820159cd0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fa81000e7b0 tx=0x7fa81000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:20.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.594+0000 7fa817e00640 1 -- 172.21.15.90:0/678251832 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa810003b80 con 0x7fa8040051c0 2024-04-25T01:18:20.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.594+0000 7fa817e00640 1 -- 172.21.15.90:0/678251832 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa810018070 con 0x7fa8040051c0 2024-04-25T01:18:20.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.594+0000 7fa817e00640 1 -- 172.21.15.90:0/678251832 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8100050d0 con 0x7fa8040051c0 2024-04-25T01:18:20.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.594+0000 7fa827200640 1 -- 172.21.15.90:0/678251832 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa82015a210 con 0x7fa8040051c0 2024-04-25T01:18:20.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.594+0000 7fa827200640 1 -- 172.21.15.90:0/678251832 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa820158ca0 con 0x7fa8040051c0 2024-04-25T01:18:20.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.595+0000 7fa827200640 1 -- 172.21.15.90:0/678251832 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa82004f5a0 con 0x7fa8040051c0 2024-04-25T01:18:20.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.596+0000 7fa817e00640 1 -- 172.21.15.90:0/678251832 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa810005270 con 0x7fa8040051c0 2024-04-25T01:18:20.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.597+0000 7fa817e00640 1 --2- 172.21.15.90:0/678251832 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa80006be30 0x7fa80006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:20.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.597+0000 7fa817e00640 1 -- 172.21.15.90:0/678251832 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7fa81008b170 con 0x7fa8040051c0 2024-04-25T01:18:20.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.597+0000 7fa81f400640 1 --2- 172.21.15.90:0/678251832 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa80006be30 0x7fa80006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:20.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.598+0000 7fa81f400640 1 --2- 172.21.15.90:0/678251832 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa80006be30 0x7fa80006e2f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa82006ce30 tx=0x7fa80c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:20.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.601+0000 7fa817e00640 1 -- 172.21.15.90:0/678251832 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa81001a310 con 0x7fa8040051c0 2024-04-25T01:18:20.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:20.841+0000 7fa827200640 1 -- 172.21.15.90:0/678251832 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "recovery_fs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fa82015a440 con 0x7fa8040051c0 2024-04-25T01:18:21.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.446+0000 7fa817e00640 1 -- 172.21.15.90:0/678251832 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "recovery_fs", "var": "standby_count_wanted", "val": "0"}]=0 v22) ==== 123+0+0 (secure 0 0 0) 0x7fa810059740 con 0x7fa8040051c0 2024-04-25T01:18:21.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.450+0000 7fa827200640 1 -- 172.21.15.90:0/678251832 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa80006be30 msgr2=0x7fa80006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:21.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.450+0000 7fa827200640 1 --2- 172.21.15.90:0/678251832 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa80006be30 0x7fa80006e2f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa82006ce30 tx=0x7fa80c008040 comp rx=0 tx=0).stop 2024-04-25T01:18:21.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.450+0000 7fa827200640 1 -- 172.21.15.90:0/678251832 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8040051c0 msgr2=0x7fa820159cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:21.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.450+0000 7fa827200640 1 --2- 172.21.15.90:0/678251832 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8040051c0 0x7fa820159cd0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fa81000e7b0 tx=0x7fa81000ec70 comp rx=0 tx=0).stop 2024-04-25T01:18:21.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.450+0000 7fa827200640 1 -- 172.21.15.90:0/678251832 shutdown_connections 2024-04-25T01:18:21.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.450+0000 7fa827200640 1 --2- 172.21.15.90:0/678251832 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa80006be30 0x7fa80006e2f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:21.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.450+0000 7fa827200640 1 --2- 172.21.15.90:0/678251832 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8201f3b00 0x7fa8201f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:21.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.450+0000 7fa827200640 1 --2- 172.21.15.90:0/678251832 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa8201f36a0 0x7fa8201f74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:21.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.450+0000 7fa827200640 1 --2- 172.21.15.90:0/678251832 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8040051c0 0x7fa820159cd0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:21.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.450+0000 7fa827200640 1 -- 172.21.15.90:0/678251832 >> 172.21.15.90:0/678251832 conn(0x7fa82006b140 msgr2=0x7fa82006e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:21.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.450+0000 7fa827200640 1 -- 172.21.15.90:0/678251832 shutdown_connections 2024-04-25T01:18:21.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.450+0000 7fa827200640 1 -- 172.21.15.90:0/678251832 wait complete. 2024-04-25T01:18:21.454 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-25T01:18:21.454 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set recovery_fs allow_standby_replay false 2024-04-25T01:18:21.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.677+0000 7fe956400640 1 Processor -- start 2024-04-25T01:18:21.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.678+0000 7fe956400640 1 -- start start 2024-04-25T01:18:21.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.679+0000 7fe956400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9500720d0 0x7fe950166f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:21.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.679+0000 7fe956400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe950142950 con 0x7fe950142570 2024-04-25T01:18:21.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.679+0000 7fe956400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe950071030 con 0x7fe950070ba0 2024-04-25T01:18:21.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.679+0000 7fe956400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe950071c40 con 0x7fe9500720d0 2024-04-25T01:18:21.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.679+0000 7fe94f400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe950142570 0x7fe95006cbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53432/0 (socket says 172.21.15.90:53432) 2024-04-25T01:18:21.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.680+0000 7fe94f400640 1 -- 172.21.15.90:0/1143521433 learned_addr learned my addr 172.21.15.90:0/1143521433 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:21.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.680+0000 7fe94fe00640 1 --2- 172.21.15.90:0/1143521433 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9500720d0 0x7fe950166f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:21.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.680+0000 7fe94ea00640 1 -- 172.21.15.90:0/1143521433 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1946774817 0 0) 0x7fe950142950 con 0x7fe950142570 2024-04-25T01:18:21.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.680+0000 7fe94ea00640 1 -- 172.21.15.90:0/1143521433 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe934003520 con 0x7fe950142570 2024-04-25T01:18:21.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.680+0000 7fe94ea00640 1 -- 172.21.15.90:0/1143521433 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2896130439 0 0) 0x7fe934003520 con 0x7fe950142570 2024-04-25T01:18:21.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.680+0000 7fe94ea00640 1 -- 172.21.15.90:0/1143521433 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9500720d0 msgr2=0x7fe950166f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:21.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.680+0000 7fe94ea00640 1 --2- 172.21.15.90:0/1143521433 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9500720d0 0x7fe950166f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:21.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.680+0000 7fe94ea00640 1 -- 172.21.15.90:0/1143521433 >> v1:172.21.15.103:6789/0 conn(0x7fe950070ba0 legacy=0x7fe95006d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:21.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.680+0000 7fe94ea00640 1 -- 172.21.15.90:0/1143521433 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe950071310 con 0x7fe950142570 2024-04-25T01:18:21.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.680+0000 7fe94ea00640 1 -- 172.21.15.90:0/1143521433 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7fe9400019f0 con 0x7fe950142570 2024-04-25T01:18:21.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.680+0000 7fe94ea00640 1 -- 172.21.15.90:0/1143521433 >> v1:172.21.15.90:6789/0 conn(0x7fe950142570 legacy=0x7fe95006cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:21.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.680+0000 7fe94ea00640 1 --2- 172.21.15.90:0/1143521433 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9340039e0 0x7fe934003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:21.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.681+0000 7fe94ea00640 1 --2- 172.21.15.90:0/1143521433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe934004800 0x7fe934004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:21.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.681+0000 7fe94ea00640 1 --2- 172.21.15.90:0/1143521433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9340051c0 0x7fe93400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:21.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.681+0000 7fe94ea00640 1 -- 172.21.15.90:0/1143521433 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe950071c40 con 0x7fe934004800 2024-04-25T01:18:21.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.681+0000 7fe94ea00640 1 -- 172.21.15.90:0/1143521433 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe93400c0c0 con 0x7fe9340051c0 2024-04-25T01:18:21.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.681+0000 7fe94ea00640 1 -- 172.21.15.90:0/1143521433 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe93400c2c0 con 0x7fe9340039e0 2024-04-25T01:18:21.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.681+0000 7fe946a00640 1 --2- 172.21.15.90:0/1143521433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe934004800 0x7fe934004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:21.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.681+0000 7fe94f400640 1 --2- 172.21.15.90:0/1143521433 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9340039e0 0x7fe934003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:21.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.681+0000 7fe94fe00640 1 --2- 172.21.15.90:0/1143521433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9340051c0 0x7fe93400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:21.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.681+0000 7fe946a00640 1 -- 172.21.15.90:0/1143521433 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9340039e0 msgr2=0x7fe934003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:21.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.681+0000 7fe946a00640 1 --2- 172.21.15.90:0/1143521433 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9340039e0 0x7fe934003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:21.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.681+0000 7fe946a00640 1 -- 172.21.15.90:0/1143521433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9340051c0 msgr2=0x7fe93400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:21.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.681+0000 7fe946a00640 1 --2- 172.21.15.90:0/1143521433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9340051c0 0x7fe93400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:21.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.681+0000 7fe946a00640 1 -- 172.21.15.90:0/1143521433 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe934003520 con 0x7fe934004800 2024-04-25T01:18:21.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.681+0000 7fe94fe00640 1 --2- 172.21.15.90:0/1143521433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9340051c0 0x7fe93400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:21.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.682+0000 7fe946a00640 1 --2- 172.21.15.90:0/1143521433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe934004800 0x7fe934004c80 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe938001750 tx=0x7fe934005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48763a9ef22d9fe3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:21.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.683+0000 7fe94ea00640 1 -- 172.21.15.90:0/1143521433 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe938001370 con 0x7fe934004800 2024-04-25T01:18:21.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.683+0000 7fe94ea00640 1 -- 172.21.15.90:0/1143521433 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9380052d0 con 0x7fe934004800 2024-04-25T01:18:21.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.683+0000 7fe94ea00640 1 -- 172.21.15.90:0/1143521433 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe93802abe0 con 0x7fe934004800 2024-04-25T01:18:21.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.683+0000 7fe956400640 1 -- 172.21.15.90:0/1143521433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe934004800 msgr2=0x7fe934004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:21.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.683+0000 7fe956400640 1 --2- 172.21.15.90:0/1143521433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe934004800 0x7fe934004c80 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe938001750 tx=0x7fe934005830 comp rx=0 tx=0).stop 2024-04-25T01:18:21.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.683+0000 7fe94fe00640 1 -- 172.21.15.90:0/1143521433 reap_dead start 2024-04-25T01:18:21.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.683+0000 7fe956400640 1 -- 172.21.15.90:0/1143521433 shutdown_connections 2024-04-25T01:18:21.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.683+0000 7fe956400640 1 -- 172.21.15.90:0/1143521433 >> 172.21.15.90:0/1143521433 conn(0x7fe95006b140 msgr2=0x7fe95006b550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:21.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.684+0000 7fe956400640 1 -- 172.21.15.90:0/1143521433 shutdown_connections 2024-04-25T01:18:21.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.684+0000 7fe956400640 1 -- 172.21.15.90:0/1143521433 wait complete. 2024-04-25T01:18:21.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.685+0000 7fe956400640 1 Processor -- start 2024-04-25T01:18:21.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.686+0000 7fe956400640 1 -- start start 2024-04-25T01:18:21.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.686+0000 7fe956400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe934004800 0x7fe950155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:21.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.686+0000 7fe956400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9500cbb00 0x7fe9501e1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:21.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.686+0000 7fe956400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9501e1ce0 0x7fe9501e2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:21.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.687+0000 7fe956400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe950072c60 con 0x7fe934004800 2024-04-25T01:18:21.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.687+0000 7fe956400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe9500718d0 con 0x7fe9501e1ce0 2024-04-25T01:18:21.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.687+0000 7fe956400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe950072de0 con 0x7fe9500cbb00 2024-04-25T01:18:21.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.687+0000 7fe94f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe934004800 0x7fe950155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:21.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.687+0000 7fe94f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe934004800 0x7fe950155360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41124/0 (socket says 172.21.15.90:41124) 2024-04-25T01:18:21.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.687+0000 7fe94f400640 1 -- 172.21.15.90:0/2760073054 learned_addr learned my addr 172.21.15.90:0/2760073054 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:21.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.687+0000 7fe94fe00640 1 --2- 172.21.15.90:0/2760073054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9501e1ce0 0x7fe9501e2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:21.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.687+0000 7fe946a00640 1 --2- 172.21.15.90:0/2760073054 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9500cbb00 0x7fe9501e1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:21.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.687+0000 7fe94f400640 1 -- 172.21.15.90:0/2760073054 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9500cbb00 msgr2=0x7fe9501e1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:21.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.687+0000 7fe94f400640 1 --2- 172.21.15.90:0/2760073054 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9500cbb00 0x7fe9501e1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:21.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.687+0000 7fe94f400640 1 -- 172.21.15.90:0/2760073054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9501e1ce0 msgr2=0x7fe9501e2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:21.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.687+0000 7fe94f400640 1 --2- 172.21.15.90:0/2760073054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9501e1ce0 0x7fe9501e2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:21.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.687+0000 7fe94f400640 1 -- 172.21.15.90:0/2760073054 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9500cbf80 con 0x7fe934004800 2024-04-25T01:18:21.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.687+0000 7fe946a00640 1 --2- 172.21.15.90:0/2760073054 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9500cbb00 0x7fe9501e1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:21.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.688+0000 7fe94f400640 1 --2- 172.21.15.90:0/2760073054 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe934004800 0x7fe950155360 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe940007a60 tx=0x7fe94000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:21.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.688+0000 7fe947e00640 1 -- 172.21.15.90:0/2760073054 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94000ee50 con 0x7fe934004800 2024-04-25T01:18:21.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.688+0000 7fe947e00640 1 -- 172.21.15.90:0/2760073054 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe940018070 con 0x7fe934004800 2024-04-25T01:18:21.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.688+0000 7fe947e00640 1 -- 172.21.15.90:0/2760073054 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe940013660 con 0x7fe934004800 2024-04-25T01:18:21.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.688+0000 7fe956400640 1 -- 172.21.15.90:0/2760073054 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9500cc260 con 0x7fe934004800 2024-04-25T01:18:21.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.688+0000 7fe956400640 1 -- 172.21.15.90:0/2760073054 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9501558a0 con 0x7fe934004800 2024-04-25T01:18:21.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.689+0000 7fe956400640 1 -- 172.21.15.90:0/2760073054 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9500c7ea0 con 0x7fe934004800 2024-04-25T01:18:21.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.690+0000 7fe947e00640 1 -- 172.21.15.90:0/2760073054 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe940013800 con 0x7fe934004800 2024-04-25T01:18:21.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.691+0000 7fe947e00640 1 --2- 172.21.15.90:0/2760073054 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe92c06be30 0x7fe92c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:21.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.691+0000 7fe947e00640 1 -- 172.21.15.90:0/2760073054 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7fe94008ba20 con 0x7fe934004800 2024-04-25T01:18:21.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.692+0000 7fe946a00640 1 --2- 172.21.15.90:0/2760073054 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe92c06be30 0x7fe92c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:21.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.692+0000 7fe946a00640 1 --2- 172.21.15.90:0/2760073054 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe92c06be30 0x7fe92c06e2f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe938032000 tx=0x7fe93802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:21.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.696+0000 7fe947e00640 1 -- 172.21.15.90:0/2760073054 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe940059f50 con 0x7fe934004800 2024-04-25T01:18:21.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:21.933+0000 7fe956400640 1 -- 172.21.15.90:0/2760073054 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "recovery_fs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fe95006e9a0 con 0x7fe934004800 2024-04-25T01:18:22.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.450+0000 7fe947e00640 1 -- 172.21.15.90:0/2760073054 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "recovery_fs", "var": "allow_standby_replay", "val": "false"}]=0 v23) ==== 127+0+0 (secure 0 0 0) 0x7fe940059850 con 0x7fe934004800 2024-04-25T01:18:22.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.453+0000 7fe956400640 1 -- 172.21.15.90:0/2760073054 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe92c06be30 msgr2=0x7fe92c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:22.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.453+0000 7fe956400640 1 --2- 172.21.15.90:0/2760073054 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe92c06be30 0x7fe92c06e2f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe938032000 tx=0x7fe93802e000 comp rx=0 tx=0).stop 2024-04-25T01:18:22.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.453+0000 7fe956400640 1 -- 172.21.15.90:0/2760073054 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe934004800 msgr2=0x7fe950155360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:22.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.453+0000 7fe956400640 1 --2- 172.21.15.90:0/2760073054 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe934004800 0x7fe950155360 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe940007a60 tx=0x7fe94000e530 comp rx=0 tx=0).stop 2024-04-25T01:18:22.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.453+0000 7fe956400640 1 -- 172.21.15.90:0/2760073054 shutdown_connections 2024-04-25T01:18:22.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.453+0000 7fe956400640 1 --2- 172.21.15.90:0/2760073054 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe92c06be30 0x7fe92c06e2f0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:22.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.453+0000 7fe956400640 1 --2- 172.21.15.90:0/2760073054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9501e1ce0 0x7fe9501e2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:22.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.453+0000 7fe956400640 1 --2- 172.21.15.90:0/2760073054 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe9500cbb00 0x7fe9501e1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:22.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.453+0000 7fe956400640 1 --2- 172.21.15.90:0/2760073054 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe934004800 0x7fe950155360 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:22.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.453+0000 7fe956400640 1 -- 172.21.15.90:0/2760073054 >> 172.21.15.90:0/2760073054 conn(0x7fe95006b140 msgr2=0x7fe95006cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:22.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.454+0000 7fe956400640 1 -- 172.21.15.90:0/2760073054 shutdown_connections 2024-04-25T01:18:22.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.454+0000 7fe956400640 1 -- 172.21.15.90:0/2760073054 wait complete. 2024-04-25T01:18:22.458 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'recovery_fs' 2024-04-25T01:18:22.459 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:18:22.570 INFO:tasks.ceph.mon.a.smithi090.stderr:2024-04-25T01:18:22.586+0000 7f0aa8600640 -1 log_channel(cluster) log [ERR] : Health check update: 2 filesystems are offline (MDS_ALL_DOWN) 2024-04-25T01:18:22.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.673+0000 7f9e29800640 1 Processor -- start 2024-04-25T01:18:22.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.673+0000 7f9e29800640 1 -- start start 2024-04-25T01:18:22.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.675+0000 7f9e29800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9e240b2a20 0x7f9e240cfc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:22.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.675+0000 7f9e29800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e2406cbc0 0x7f9e240d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:22.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.675+0000 7f9e29800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e240cdf70 con 0x7f9e24145640 2024-04-25T01:18:22.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.675+0000 7f9e29800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f9e24145a20 con 0x7f9e240b2a20 2024-04-25T01:18:22.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.675+0000 7f9e29800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9e2406d140 con 0x7f9e2406cbc0 2024-04-25T01:18:22.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.675+0000 7f9e22a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f9e24145640 0x7f9e2415ce20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53434/0 (socket says 172.21.15.90:53434) 2024-04-25T01:18:22.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.675+0000 7f9e22a00640 1 -- 172.21.15.90:0/201556443 learned_addr learned my addr 172.21.15.90:0/201556443 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:22.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.675+0000 7f9e22000640 1 --2- 172.21.15.90:0/201556443 >> v2:172.21.15.90:3300/0 conn(0x7f9e240b2a20 0x7f9e240cfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:22.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.675+0000 7f9e22000640 1 -- 172.21.15.90:0/201556443 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e2406cbc0 msgr2=0x7f9e240d01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:22.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.675+0000 7f9e22000640 1 --2- 172.21.15.90:0/201556443 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e2406cbc0 0x7f9e240d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:22.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.675+0000 7f9e22000640 1 -- 172.21.15.90:0/201556443 >> v1:172.21.15.90:6789/0 conn(0x7f9e24145640 legacy=0x7f9e2415ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:22.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.676+0000 7f9e22000640 1 -- 172.21.15.90:0/201556443 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e2406d420 con 0x7f9e240b2a20 2024-04-25T01:18:22.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.677+0000 7f9e22000640 1 --2- 172.21.15.90:0/201556443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e240b2a20 0x7f9e240cfc90 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c0097a0 tx=0x7f9e0c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=438586525856158f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:22.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.677+0000 7f9e21600640 1 -- 172.21.15.90:0/201556443 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c037070 con 0x7f9e240b2a20 2024-04-25T01:18:22.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.677+0000 7f9e21600640 1 -- 172.21.15.90:0/201556443 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e0c032440 con 0x7f9e240b2a20 2024-04-25T01:18:22.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.677+0000 7f9e21600640 1 -- 172.21.15.90:0/201556443 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c032810 con 0x7f9e240b2a20 2024-04-25T01:18:22.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.677+0000 7f9e29800640 1 -- 172.21.15.90:0/201556443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e240b2a20 msgr2=0x7f9e240cfc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:22.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.677+0000 7f9e29800640 1 --2- 172.21.15.90:0/201556443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e240b2a20 0x7f9e240cfc90 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c0097a0 tx=0x7f9e0c02a180 comp rx=0 tx=0).stop 2024-04-25T01:18:22.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.678+0000 7f9e29800640 1 -- 172.21.15.90:0/201556443 shutdown_connections 2024-04-25T01:18:22.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.678+0000 7f9e29800640 1 --2- 172.21.15.90:0/201556443 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e2406cbc0 0x7f9e240d01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:22.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.678+0000 7f9e29800640 1 --2- 172.21.15.90:0/201556443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e240b2a20 0x7f9e240cfc90 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:22.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.678+0000 7f9e29800640 1 -- 172.21.15.90:0/201556443 >> 172.21.15.90:0/201556443 conn(0x7f9e2406b320 msgr2=0x7f9e2406b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:22.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.678+0000 7f9e29800640 1 -- 172.21.15.90:0/201556443 shutdown_connections 2024-04-25T01:18:22.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.678+0000 7f9e29800640 1 -- 172.21.15.90:0/201556443 wait complete. 2024-04-25T01:18:22.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.679+0000 7f9e29800640 1 Processor -- start 2024-04-25T01:18:22.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.679+0000 7f9e29800640 1 -- start start 2024-04-25T01:18:22.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.680+0000 7f9e29800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e2406cbc0 0x7f9e24070c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:22.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.680+0000 7f9e29800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e240b2a20 0x7f9e24157ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:22.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.680+0000 7f9e29800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e24145640 0x7f9e24143160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:22.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.680+0000 7f9e29800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9e240ce450 con 0x7f9e24145640 2024-04-25T01:18:22.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.680+0000 7f9e29800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9e2406d9e0 con 0x7f9e240b2a20 2024-04-25T01:18:22.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.680+0000 7f9e29800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9e240ce5d0 con 0x7f9e2406cbc0 2024-04-25T01:18:22.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.681+0000 7f9e23400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e24145640 0x7f9e24143160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:22.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.681+0000 7f9e23400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e24145640 0x7f9e24143160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41148/0 (socket says 172.21.15.90:41148) 2024-04-25T01:18:22.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.681+0000 7f9e23400640 1 -- 172.21.15.90:0/2739057934 learned_addr learned my addr 172.21.15.90:0/2739057934 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:22.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.681+0000 7f9e22a00640 1 --2- 172.21.15.90:0/2739057934 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e2406cbc0 0x7f9e24070c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:22.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.681+0000 7f9e22000640 1 --2- 172.21.15.90:0/2739057934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e240b2a20 0x7f9e24157ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:22.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.681+0000 7f9e23400640 1 -- 172.21.15.90:0/2739057934 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e2406cbc0 msgr2=0x7f9e24070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:22.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.681+0000 7f9e23400640 1 --2- 172.21.15.90:0/2739057934 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e2406cbc0 0x7f9e24070c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:22.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.681+0000 7f9e23400640 1 -- 172.21.15.90:0/2739057934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e240b2a20 msgr2=0x7f9e24157ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:22.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.681+0000 7f9e23400640 1 --2- 172.21.15.90:0/2739057934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e240b2a20 0x7f9e24157ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:22.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.681+0000 7f9e23400640 1 -- 172.21.15.90:0/2739057934 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e24071150 con 0x7f9e24145640 2024-04-25T01:18:22.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.681+0000 7f9e22000640 1 --2- 172.21.15.90:0/2739057934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e240b2a20 0x7f9e24157ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:22.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.681+0000 7f9e23400640 1 --2- 172.21.15.90:0/2739057934 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e24145640 0x7f9e24143160 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c00b9a0 tx=0x7f9e1c00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:22.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.682+0000 7f9e1aa00640 1 -- 172.21.15.90:0/2739057934 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c019070 con 0x7f9e24145640 2024-04-25T01:18:22.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.682+0000 7f9e1aa00640 1 -- 172.21.15.90:0/2739057934 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e1c00cbc0 con 0x7f9e24145640 2024-04-25T01:18:22.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.682+0000 7f9e29800640 1 -- 172.21.15.90:0/2739057934 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e240715e0 con 0x7f9e24145640 2024-04-25T01:18:22.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.682+0000 7f9e1aa00640 1 -- 172.21.15.90:0/2739057934 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c014440 con 0x7f9e24145640 2024-04-25T01:18:22.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.682+0000 7f9e29800640 1 -- 172.21.15.90:0/2739057934 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e24071ba0 con 0x7f9e24145640 2024-04-25T01:18:22.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.683+0000 7f9e29800640 1 -- 172.21.15.90:0/2739057934 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e24068190 con 0x7f9e24145640 2024-04-25T01:18:22.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.684+0000 7f9e1aa00640 1 -- 172.21.15.90:0/2739057934 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9e1c01c070 con 0x7f9e24145640 2024-04-25T01:18:22.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.685+0000 7f9e1aa00640 1 --2- 172.21.15.90:0/2739057934 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9e0006be30 0x7f9e0006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:22.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.685+0000 7f9e1aa00640 1 -- 172.21.15.90:0/2739057934 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f9e1c08afa0 con 0x7f9e24145640 2024-04-25T01:18:22.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.685+0000 7f9e22a00640 1 --2- 172.21.15.90:0/2739057934 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9e0006be30 0x7f9e0006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:22.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.686+0000 7f9e22a00640 1 --2- 172.21.15.90:0/2739057934 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9e0006be30 0x7f9e0006e2f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9e14000f60 tx=0x7f9e14009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:22.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.690+0000 7f9e1aa00640 1 -- 172.21.15.90:0/2739057934 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9e1c059560 con 0x7f9e24145640 2024-04-25T01:18:22.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.927+0000 7f9e29800640 1 -- 172.21.15.90:0/2739057934 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9e2406bbf0 con 0x7f9e24145640 2024-04-25T01:18:22.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.928+0000 7f9e1aa00640 1 -- 172.21.15.90:0/2739057934 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6477 (secure 0 0 0) 0x7f9e1c058e60 con 0x7f9e24145640 2024-04-25T01:18:22.912 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:22.912 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1873497946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6833","nonce":1873497946},{"type":"v1","addr":"172.21.15.103:6835","nonce":1873497946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/3206865830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":3206865830},{"type":"v1","addr":"172.21.15.90:6835","nonce":3206865830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5598,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2208594231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2208594231},{"type":"v1","addr":"172.21.15.90:6837","nonce":2208594231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5601,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6834/2850373997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":2850373997},{"type":"v1","addr":"172.21.15.103:6834","nonce":2850373997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:18:02.701652+0000","modified":"2024-04-25T01:18:15.603132+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2},{"mdsmap":{"epoch":23,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:18:19.371011+0000","modified":"2024-04-25T01:18:21.934723+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":7,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3}]} 2024-04-25T01:18:22.912 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 23 2024-04-25T01:18:22.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.931+0000 7f9e29800640 1 -- 172.21.15.90:0/2739057934 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9e0006be30 msgr2=0x7f9e0006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:22.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.931+0000 7f9e29800640 1 --2- 172.21.15.90:0/2739057934 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9e0006be30 0x7f9e0006e2f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9e14000f60 tx=0x7f9e14009040 comp rx=0 tx=0).stop 2024-04-25T01:18:22.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.931+0000 7f9e29800640 1 -- 172.21.15.90:0/2739057934 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e24145640 msgr2=0x7f9e24143160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:22.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.931+0000 7f9e29800640 1 --2- 172.21.15.90:0/2739057934 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e24145640 0x7f9e24143160 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c00b9a0 tx=0x7f9e1c00be60 comp rx=0 tx=0).stop 2024-04-25T01:18:22.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.931+0000 7f9e29800640 1 -- 172.21.15.90:0/2739057934 shutdown_connections 2024-04-25T01:18:22.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.932+0000 7f9e29800640 1 --2- 172.21.15.90:0/2739057934 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9e0006be30 0x7f9e0006e2f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:22.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.932+0000 7f9e29800640 1 --2- 172.21.15.90:0/2739057934 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e24145640 0x7f9e24143160 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:22.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.932+0000 7f9e29800640 1 --2- 172.21.15.90:0/2739057934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e240b2a20 0x7f9e24157ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:22.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.932+0000 7f9e29800640 1 --2- 172.21.15.90:0/2739057934 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e2406cbc0 0x7f9e24070c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:22.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.932+0000 7f9e29800640 1 -- 172.21.15.90:0/2739057934 >> 172.21.15.90:0/2739057934 conn(0x7f9e2406b320 msgr2=0x7f9e240af660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:22.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.932+0000 7f9e29800640 1 -- 172.21.15.90:0/2739057934 shutdown_connections 2024-04-25T01:18:22.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:22.932+0000 7f9e29800640 1 -- 172.21.15.90:0/2739057934 wait complete. 2024-04-25T01:18:22.935 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:18:23.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.155+0000 7f551d600640 1 Processor -- start 2024-04-25T01:18:23.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.155+0000 7f551d600640 1 -- start start 2024-04-25T01:18:23.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.156+0000 7f551d600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f55180b2a20 0x7f55180cfc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:23.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.156+0000 7f551d600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55180cdf70 con 0x7f551806cbc0 2024-04-25T01:18:23.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.156+0000 7f551d600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5518145a20 con 0x7f5518145640 2024-04-25T01:18:23.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.156+0000 7f551d600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f551806d140 con 0x7f55180b2a20 2024-04-25T01:18:23.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.157+0000 7f5517400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f551806cbc0 0x7f55180ce450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53440/0 (socket says 172.21.15.90:53440) 2024-04-25T01:18:23.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.157+0000 7f5517400640 1 -- 172.21.15.90:0/610480321 learned_addr learned my addr 172.21.15.90:0/610480321 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:23.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.157+0000 7f5516000640 1 --2- 172.21.15.90:0/610480321 >> v2:172.21.15.103:3300/0 conn(0x7f55180b2a20 0x7f55180cfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:23.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.157+0000 7f5515600640 1 -- 172.21.15.90:0/610480321 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4133562180 0 0) 0x7f55180cdf70 con 0x7f551806cbc0 2024-04-25T01:18:23.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.157+0000 7f5515600640 1 -- 172.21.15.90:0/610480321 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54fc003520 con 0x7f551806cbc0 2024-04-25T01:18:23.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.158+0000 7f5515600640 1 -- 172.21.15.90:0/610480321 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2564383618 0 0) 0x7f54fc003520 con 0x7f551806cbc0 2024-04-25T01:18:23.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.158+0000 7f5515600640 1 -- 172.21.15.90:0/610480321 >> v2:172.21.15.103:3300/0 conn(0x7f55180b2a20 msgr2=0x7f55180cfc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:23.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.158+0000 7f5515600640 1 --2- 172.21.15.90:0/610480321 >> v2:172.21.15.103:3300/0 conn(0x7f55180b2a20 0x7f55180cfc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:23.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.158+0000 7f5515600640 1 -- 172.21.15.90:0/610480321 >> v1:172.21.15.103:6789/0 conn(0x7f5518145640 legacy=0x7f551815ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:23.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.158+0000 7f5515600640 1 -- 172.21.15.90:0/610480321 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f551806d420 con 0x7f551806cbc0 2024-04-25T01:18:23.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.158+0000 7f5515600640 1 -- 172.21.15.90:0/610480321 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f5510001a10 con 0x7f551806cbc0 2024-04-25T01:18:23.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.158+0000 7f5515600640 1 -- 172.21.15.90:0/610480321 >> v1:172.21.15.90:6789/0 conn(0x7f551806cbc0 legacy=0x7f55180ce450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:23.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.158+0000 7f5515600640 1 --2- 172.21.15.90:0/610480321 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54fc0039b0 0x7f54fc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:23.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.158+0000 7f5515600640 1 --2- 172.21.15.90:0/610480321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54fc004800 0x7f54fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:23.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.158+0000 7f5515600640 1 --2- 172.21.15.90:0/610480321 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f54fc0051c0 0x7f54fc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:23.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.158+0000 7f5515600640 1 -- 172.21.15.90:0/610480321 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f551806d140 con 0x7f54fc004800 2024-04-25T01:18:23.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.158+0000 7f5515600640 1 -- 172.21.15.90:0/610480321 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f54fc00c060 con 0x7f54fc0039b0 2024-04-25T01:18:23.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.158+0000 7f5515600640 1 -- 172.21.15.90:0/610480321 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f54fc00c260 con 0x7f54fc0051c0 2024-04-25T01:18:23.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.158+0000 7f5516000640 1 --2- 172.21.15.90:0/610480321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54fc004800 0x7f54fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:23.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.158+0000 7f5516a00640 1 --2- 172.21.15.90:0/610480321 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54fc0039b0 0x7f54fc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:23.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.159+0000 7f5516000640 1 -- 172.21.15.90:0/610480321 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f54fc0051c0 msgr2=0x7f54fc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:23.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.159+0000 7f5516000640 1 --2- 172.21.15.90:0/610480321 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f54fc0051c0 0x7f54fc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:23.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.159+0000 7f5516000640 1 -- 172.21.15.90:0/610480321 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54fc0039b0 msgr2=0x7f54fc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:23.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.159+0000 7f5516000640 1 --2- 172.21.15.90:0/610480321 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54fc0039b0 0x7f54fc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:23.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.159+0000 7f5516000640 1 -- 172.21.15.90:0/610480321 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54fc003520 con 0x7f54fc004800 2024-04-25T01:18:23.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.159+0000 7f5517400640 1 -- 172.21.15.90:0/610480321 reap_dead start 2024-04-25T01:18:23.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.160+0000 7f5516000640 1 --2- 172.21.15.90:0/610480321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54fc004800 0x7f54fc004c80 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f5504009ac0 tx=0x7f54fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ec46873263c1cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:23.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.160+0000 7f5515600640 1 -- 172.21.15.90:0/610480321 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5504036070 con 0x7f54fc004800 2024-04-25T01:18:23.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.160+0000 7f5515600640 1 -- 172.21.15.90:0/610480321 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5504031400 con 0x7f54fc004800 2024-04-25T01:18:23.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.160+0000 7f5515600640 1 -- 172.21.15.90:0/610480321 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5504009690 con 0x7f54fc004800 2024-04-25T01:18:23.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.160+0000 7f551d600640 1 -- 172.21.15.90:0/610480321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54fc004800 msgr2=0x7f54fc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:23.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.160+0000 7f551d600640 1 --2- 172.21.15.90:0/610480321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54fc004800 0x7f54fc004c80 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f5504009ac0 tx=0x7f54fc005830 comp rx=0 tx=0).stop 2024-04-25T01:18:23.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.161+0000 7f551d600640 1 -- 172.21.15.90:0/610480321 shutdown_connections 2024-04-25T01:18:23.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.161+0000 7f551d600640 1 --2- 172.21.15.90:0/610480321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54fc004800 0x7f54fc004c80 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:23.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.161+0000 7f551d600640 1 -- 172.21.15.90:0/610480321 >> 172.21.15.90:0/610480321 conn(0x7f551806b320 msgr2=0x7f551806b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:23.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.161+0000 7f551d600640 1 -- 172.21.15.90:0/610480321 shutdown_connections 2024-04-25T01:18:23.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.161+0000 7f551d600640 1 -- 172.21.15.90:0/610480321 wait complete. 2024-04-25T01:18:23.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.162+0000 7f551d600640 1 Processor -- start 2024-04-25T01:18:23.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.163+0000 7f551d600640 1 -- start start 2024-04-25T01:18:23.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.164+0000 7f551d600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f54fc004800 0x7f55181f4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:23.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.164+0000 7f551d600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55181548d0 0x7f5518156140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:23.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.164+0000 7f551d600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f55181f1fe0 0x7f55181f2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:23.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.164+0000 7f551d600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f551815ce20 con 0x7f55181f1fe0 2024-04-25T01:18:23.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.164+0000 7f551d600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f551806d9e0 con 0x7f55181548d0 2024-04-25T01:18:23.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.164+0000 7f551d600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f551815cfa0 con 0x7f54fc004800 2024-04-25T01:18:23.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.164+0000 7f5517400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f55181f1fe0 0x7f55181f2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:23.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.164+0000 7f5517400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f55181f1fe0 0x7f55181f2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41162/0 (socket says 172.21.15.90:41162) 2024-04-25T01:18:23.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.164+0000 7f5517400640 1 -- 172.21.15.90:0/186571424 learned_addr learned my addr 172.21.15.90:0/186571424 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:23.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.164+0000 7f5517400640 1 -- 172.21.15.90:0/186571424 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f54fc004800 msgr2=0x7f55181f4e20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T01:18:23.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.164+0000 7f5517400640 1 --2- 172.21.15.90:0/186571424 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f54fc004800 0x7f55181f4e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:23.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.164+0000 7f5517400640 1 -- 172.21.15.90:0/186571424 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55181548d0 msgr2=0x7f5518156140 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T01:18:23.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.164+0000 7f5517400640 1 --2- 172.21.15.90:0/186571424 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55181548d0 0x7f5518156140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:23.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.164+0000 7f5517400640 1 -- 172.21.15.90:0/186571424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55181f1d10 con 0x7f55181f1fe0 2024-04-25T01:18:23.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.165+0000 7f5517400640 1 --2- 172.21.15.90:0/186571424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f55181f1fe0 0x7f55181f2460 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f55180ccd80 tx=0x7f551000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:23.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.165+0000 7f550ea00640 1 -- 172.21.15.90:0/186571424 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5510004d60 con 0x7f55181f1fe0 2024-04-25T01:18:23.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.165+0000 7f550ea00640 1 -- 172.21.15.90:0/186571424 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f551000a5b0 con 0x7f55181f1fe0 2024-04-25T01:18:23.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.165+0000 7f550ea00640 1 -- 172.21.15.90:0/186571424 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551000a9a0 con 0x7f55181f1fe0 2024-04-25T01:18:23.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.165+0000 7f551d600640 1 -- 172.21.15.90:0/186571424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5518156a40 con 0x7f55181f1fe0 2024-04-25T01:18:23.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.165+0000 7f551d600640 1 -- 172.21.15.90:0/186571424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5518157000 con 0x7f55181f1fe0 2024-04-25T01:18:23.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.166+0000 7f551d600640 1 -- 172.21.15.90:0/186571424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5518068190 con 0x7f55181f1fe0 2024-04-25T01:18:23.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.167+0000 7f550ea00640 1 -- 172.21.15.90:0/186571424 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f551000ab40 con 0x7f55181f1fe0 2024-04-25T01:18:23.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.168+0000 7f550ea00640 1 --2- 172.21.15.90:0/186571424 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f54f806be30 0x7f54f806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:23.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.168+0000 7f550ea00640 1 -- 172.21.15.90:0/186571424 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f551008a5a0 con 0x7f55181f1fe0 2024-04-25T01:18:23.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.168+0000 7f5516a00640 1 --2- 172.21.15.90:0/186571424 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f54f806be30 0x7f54f806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:23.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.169+0000 7f5516a00640 1 --2- 172.21.15.90:0/186571424 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f54f806be30 0x7f54f806e2f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5508000c00 tx=0x7f5508009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:23.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.174+0000 7f550ea00640 1 -- 172.21.15.90:0/186571424 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f551001a030 con 0x7f55181f1fe0 2024-04-25T01:18:23.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.375+0000 7f551d600640 1 -- 172.21.15.90:0/186571424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f551815c420 con 0x7f55181f1fe0 2024-04-25T01:18:23.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.378+0000 7f550ea00640 1 -- 172.21.15.90:0/186571424 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) ==== 74+0+18947 (secure 0 0 0) 0x7f5510058b70 con 0x7f55181f1fe0 2024-04-25T01:18:23.362 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:23.362 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":50,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:18:20.389626+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:18:00.504992+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T01:18:01.714313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"recovery_fs_metadata","create_time":"2024-04-25T01:18:18.384671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.16","mappings":[{"from":4,"to":2}]},{"pgid":"6.20","mappings":[{"from":4,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:6837/431390375":"2024-04-26T01:18:15.602554+0000","172.21.15.90:6836/431390375":"2024-04-26T01:18:15.602554+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:18:23.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.380+0000 7f551d600640 1 -- 172.21.15.90:0/186571424 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f54f806be30 msgr2=0x7f54f806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:23.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.380+0000 7f551d600640 1 --2- 172.21.15.90:0/186571424 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f54f806be30 0x7f54f806e2f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5508000c00 tx=0x7f5508009040 comp rx=0 tx=0).stop 2024-04-25T01:18:23.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.380+0000 7f551d600640 1 -- 172.21.15.90:0/186571424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f55181f1fe0 msgr2=0x7f55181f2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:23.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.380+0000 7f551d600640 1 --2- 172.21.15.90:0/186571424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f55181f1fe0 0x7f55181f2460 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f55180ccd80 tx=0x7f551000cb90 comp rx=0 tx=0).stop 2024-04-25T01:18:23.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.380+0000 7f551d600640 1 -- 172.21.15.90:0/186571424 shutdown_connections 2024-04-25T01:18:23.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.380+0000 7f551d600640 1 --2- 172.21.15.90:0/186571424 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f54f806be30 0x7f54f806e2f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:23.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.381+0000 7f551d600640 1 --2- 172.21.15.90:0/186571424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f55181f1fe0 0x7f55181f2460 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:23.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.381+0000 7f551d600640 1 --2- 172.21.15.90:0/186571424 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55181548d0 0x7f5518156140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:23.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.381+0000 7f551d600640 1 --2- 172.21.15.90:0/186571424 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f54fc004800 0x7f55181f4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:23.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.381+0000 7f551d600640 1 -- 172.21.15.90:0/186571424 >> 172.21.15.90:0/186571424 conn(0x7f551806b320 msgr2=0x7f55180b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:23.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.381+0000 7f551d600640 1 -- 172.21.15.90:0/186571424 shutdown_connections 2024-04-25T01:18:23.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.381+0000 7f551d600640 1 -- 172.21.15.90:0/186571424 wait complete. 2024-04-25T01:18:23.383 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T01:18:23.383 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:18:23.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.602+0000 7f6f82400640 1 Processor -- start 2024-04-25T01:18:23.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.602+0000 7f6f82400640 1 -- start start 2024-04-25T01:18:23.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.603+0000 7f6f82400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6f7c0b2a20 0x7f6f7c0cfc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:23.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.603+0000 7f6f82400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f7c06cbc0 0x7f6f7c0d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:23.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.603+0000 7f6f82400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f7c0cdf70 con 0x7f6f7c145640 2024-04-25T01:18:23.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.603+0000 7f6f82400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f6f7c145a20 con 0x7f6f7c0b2a20 2024-04-25T01:18:23.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.603+0000 7f6f82400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6f7c06d140 con 0x7f6f7c06cbc0 2024-04-25T01:18:23.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.604+0000 7f6f7b400640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f6f7c145640 0x7f6f7c15ce20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.90:49700/0 (socket says 172.21.15.90:49700) 2024-04-25T01:18:23.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.604+0000 7f6f7b400640 1 -- 172.21.15.90:0/186858210 learned_addr learned my addr 172.21.15.90:0/186858210 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:23.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.604+0000 7f6f7be00640 1 --2- 172.21.15.90:0/186858210 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f7c06cbc0 0x7f6f7c0d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:23.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.604+0000 7f6f73e00640 1 --2- 172.21.15.90:0/186858210 >> v2:172.21.15.103:3300/0 conn(0x7f6f7c0b2a20 0x7f6f7c0cfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:23.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.605+0000 7f6f7aa00640 1 -- 172.21.15.90:0/186858210 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 109974324 0 0) 0x7f6f7c0cdf70 con 0x7f6f7c145640 2024-04-25T01:18:23.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.605+0000 7f6f7aa00640 1 -- 172.21.15.90:0/186858210 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f60003520 con 0x7f6f7c145640 2024-04-25T01:18:23.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.605+0000 7f6f73e00640 1 -- 172.21.15.90:0/186858210 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f7c06cbc0 msgr2=0x7f6f7c0d01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:23.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.605+0000 7f6f73e00640 1 --2- 172.21.15.90:0/186858210 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f7c06cbc0 0x7f6f7c0d01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:23.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.605+0000 7f6f73e00640 1 -- 172.21.15.90:0/186858210 >> v1:172.21.15.103:6789/0 conn(0x7f6f7c145640 legacy=0x7f6f7c15ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:23.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.605+0000 7f6f73e00640 1 -- 172.21.15.90:0/186858210 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f7c06d420 con 0x7f6f7c0b2a20 2024-04-25T01:18:23.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.605+0000 7f6f7be00640 1 --2- 172.21.15.90:0/186858210 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f7c06cbc0 0x7f6f7c0d01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:18:23.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.607+0000 7f6f73e00640 1 --2- 172.21.15.90:0/186858210 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f7c0b2a20 0x7f6f7c0cfc90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6f640097a0 tx=0x7f6f6402a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc1a9e68bec4f79d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:23.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.607+0000 7f6f7aa00640 1 -- 172.21.15.90:0/186858210 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f64037070 con 0x7f6f7c0b2a20 2024-04-25T01:18:23.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.607+0000 7f6f7aa00640 1 -- 172.21.15.90:0/186858210 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f64032440 con 0x7f6f7c0b2a20 2024-04-25T01:18:23.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.607+0000 7f6f7aa00640 1 -- 172.21.15.90:0/186858210 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f64032810 con 0x7f6f7c0b2a20 2024-04-25T01:18:23.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.608+0000 7f6f82400640 1 -- 172.21.15.90:0/186858210 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f7c0b2a20 msgr2=0x7f6f7c0cfc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:23.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.608+0000 7f6f82400640 1 --2- 172.21.15.90:0/186858210 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f7c0b2a20 0x7f6f7c0cfc90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6f640097a0 tx=0x7f6f6402a180 comp rx=0 tx=0).stop 2024-04-25T01:18:23.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.608+0000 7f6f82400640 1 -- 172.21.15.90:0/186858210 shutdown_connections 2024-04-25T01:18:23.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.608+0000 7f6f82400640 1 --2- 172.21.15.90:0/186858210 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f7c06cbc0 0x7f6f7c0d01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:23.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.608+0000 7f6f82400640 1 --2- 172.21.15.90:0/186858210 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f7c0b2a20 0x7f6f7c0cfc90 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:23.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.608+0000 7f6f82400640 1 -- 172.21.15.90:0/186858210 >> 172.21.15.90:0/186858210 conn(0x7f6f7c06b320 msgr2=0x7f6f7c06b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:23.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.608+0000 7f6f82400640 1 -- 172.21.15.90:0/186858210 shutdown_connections 2024-04-25T01:18:23.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.608+0000 7f6f82400640 1 -- 172.21.15.90:0/186858210 wait complete. 2024-04-25T01:18:23.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.610+0000 7f6f82400640 1 Processor -- start 2024-04-25T01:18:23.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.610+0000 7f6f82400640 1 -- start start 2024-04-25T01:18:23.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.611+0000 7f6f82400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f7c06cbc0 0x7f6f7c1544b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:23.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.611+0000 7f6f82400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f7c0b2a20 0x7f6f7c0b3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:23.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.611+0000 7f6f82400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f7c145640 0x7f6f7c139430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:23.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.611+0000 7f6f82400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6f7c0ce450 con 0x7f6f7c06cbc0 2024-04-25T01:18:23.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.611+0000 7f6f82400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6f7c06d9e0 con 0x7f6f7c145640 2024-04-25T01:18:23.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.611+0000 7f6f82400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6f7c0ce5d0 con 0x7f6f7c0b2a20 2024-04-25T01:18:23.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.612+0000 7f6f7b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f7c06cbc0 0x7f6f7c1544b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:23.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.612+0000 7f6f7b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f7c06cbc0 0x7f6f7c1544b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41164/0 (socket says 172.21.15.90:41164) 2024-04-25T01:18:23.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.612+0000 7f6f7b400640 1 -- 172.21.15.90:0/319906326 learned_addr learned my addr 172.21.15.90:0/319906326 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:23.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.612+0000 7f6f7be00640 1 --2- 172.21.15.90:0/319906326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f7c145640 0x7f6f7c139430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:23.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.612+0000 7f6f73e00640 1 --2- 172.21.15.90:0/319906326 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f7c0b2a20 0x7f6f7c0b3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:23.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.612+0000 7f6f7b400640 1 -- 172.21.15.90:0/319906326 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f7c0b2a20 msgr2=0x7f6f7c0b3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:23.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.612+0000 7f6f7b400640 1 --2- 172.21.15.90:0/319906326 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f7c0b2a20 0x7f6f7c0b3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:23.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.612+0000 7f6f7b400640 1 -- 172.21.15.90:0/319906326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f7c145640 msgr2=0x7f6f7c139430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:23.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.612+0000 7f6f7b400640 1 --2- 172.21.15.90:0/319906326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f7c145640 0x7f6f7c139430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:23.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.612+0000 7f6f7b400640 1 -- 172.21.15.90:0/319906326 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f7c1549f0 con 0x7f6f7c06cbc0 2024-04-25T01:18:23.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.612+0000 7f6f7b400640 1 --2- 172.21.15.90:0/319906326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f7c06cbc0 0x7f6f7c1544b0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00c400 tx=0x7f6f6c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:23.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.613+0000 7f6f73400640 1 -- 172.21.15.90:0/319906326 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c017070 con 0x7f6f7c06cbc0 2024-04-25T01:18:23.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.613+0000 7f6f73400640 1 -- 172.21.15.90:0/319906326 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f6c005d00 con 0x7f6f7c06cbc0 2024-04-25T01:18:23.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.613+0000 7f6f73400640 1 -- 172.21.15.90:0/319906326 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c007310 con 0x7f6f7c06cbc0 2024-04-25T01:18:23.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.613+0000 7f6f82400640 1 -- 172.21.15.90:0/319906326 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f7c154c90 con 0x7f6f7c06cbc0 2024-04-25T01:18:23.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.613+0000 7f6f82400640 1 -- 172.21.15.90:0/319906326 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f7c155250 con 0x7f6f7c06cbc0 2024-04-25T01:18:23.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.614+0000 7f6f82400640 1 -- 172.21.15.90:0/319906326 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f7c0d0370 con 0x7f6f7c06cbc0 2024-04-25T01:18:23.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.615+0000 7f6f73400640 1 -- 172.21.15.90:0/319906326 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6f6c01a070 con 0x7f6f7c06cbc0 2024-04-25T01:18:23.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.616+0000 7f6f73400640 1 --2- 172.21.15.90:0/319906326 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f6f5806be30 0x7f6f5806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:23.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.616+0000 7f6f73400640 1 -- 172.21.15.90:0/319906326 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f6f6c08a670 con 0x7f6f7c06cbc0 2024-04-25T01:18:23.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.616+0000 7f6f73e00640 1 --2- 172.21.15.90:0/319906326 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f6f5806be30 0x7f6f5806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:23.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.617+0000 7f6f73e00640 1 --2- 172.21.15.90:0/319906326 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f6f5806be30 0x7f6f5806e2f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6f64034000 tx=0x7f6f6402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:23.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.621+0000 7f6f73400640 1 -- 172.21.15.90:0/319906326 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6f7c0d0370 con 0x7f6f7c06cbc0 2024-04-25T01:18:23.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.816+0000 7f6f82400640 1 -- 172.21.15.90:0/319906326 --> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6f7c068190 con 0x7f6f5806be30 2024-04-25T01:18:23.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.826+0000 7f6f73400640 1 -- 172.21.15.90:0/319906326 <== mgr.4104 v2:172.21.15.90:6832/2198086733 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332509 (secure 0 0 0) 0x7f6f7c068190 con 0x7f6f5806be30 2024-04-25T01:18:23.817 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:23.820 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-25T01:18:23.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.834+0000 7f6f82400640 1 -- 172.21.15.90:0/319906326 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f6f5806be30 msgr2=0x7f6f5806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:23.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.834+0000 7f6f82400640 1 --2- 172.21.15.90:0/319906326 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f6f5806be30 0x7f6f5806e2f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6f64034000 tx=0x7f6f6402e000 comp rx=0 tx=0).stop 2024-04-25T01:18:23.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.834+0000 7f6f82400640 1 -- 172.21.15.90:0/319906326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f7c06cbc0 msgr2=0x7f6f7c1544b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:23.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.834+0000 7f6f82400640 1 --2- 172.21.15.90:0/319906326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f7c06cbc0 0x7f6f7c1544b0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00c400 tx=0x7f6f6c00c8c0 comp rx=0 tx=0).stop 2024-04-25T01:18:23.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.834+0000 7f6f82400640 1 -- 172.21.15.90:0/319906326 shutdown_connections 2024-04-25T01:18:23.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.834+0000 7f6f82400640 1 --2- 172.21.15.90:0/319906326 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f6f5806be30 0x7f6f5806e2f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:23.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.834+0000 7f6f82400640 1 --2- 172.21.15.90:0/319906326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f7c145640 0x7f6f7c139430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:23.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.834+0000 7f6f82400640 1 --2- 172.21.15.90:0/319906326 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6f7c0b2a20 0x7f6f7c0b3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:23.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.834+0000 7f6f82400640 1 --2- 172.21.15.90:0/319906326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f7c06cbc0 0x7f6f7c1544b0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:23.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.834+0000 7f6f82400640 1 -- 172.21.15.90:0/319906326 >> 172.21.15.90:0/319906326 conn(0x7f6f7c06b320 msgr2=0x7f6f7c0cfc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:23.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.834+0000 7f6f82400640 1 -- 172.21.15.90:0/319906326 shutdown_connections 2024-04-25T01:18:23.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:23.834+0000 7f6f82400640 1 -- 172.21.15.90:0/319906326 wait complete. 2024-04-25T01:18:23.838 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2024-04-25T01:18:21.863273+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6881824,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":117,"num_read_kb":100,"num_write":259,"num_write_kb":8604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":704,"ondisk_log_size":704,"up":258,"acting":258,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":214,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238196,"kb_used_data":17848,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754736524,"statfs":{"total":773094113280,"available":772850200576,"internally_reserved":0,"allocated":18276352,"data_stored":15789640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17557,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":44,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.496151"},"pg_stats":[{"pgid":"6.3f","version":"42'8","reported_seq":37,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.422510+0000","last_change":"2024-04-25T01:18:03.253664+0000","last_active":"2024-04-25T01:18:19.422510+0000","last_peered":"2024-04-25T01:18:19.422510+0000","last_clean":"2024-04-25T01:18:19.422510+0000","last_became_active":"2024-04-25T01:18:03.253362+0000","last_became_peered":"2024-04-25T01:18:03.253362+0000","last_unstale":"2024-04-25T01:18:19.422510+0000","last_undegraded":"2024-04-25T01:18:19.422510+0000","last_fullsized":"2024-04-25T01:18:19.422510+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:21:24.721366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3e","version":"42'4","reported_seq":33,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.412764+0000","last_change":"2024-04-25T01:18:03.247432+0000","last_active":"2024-04-25T01:18:19.412764+0000","last_peered":"2024-04-25T01:18:19.412764+0000","last_clean":"2024-04-25T01:18:19.412764+0000","last_became_active":"2024-04-25T01:18:03.247132+0000","last_became_peered":"2024-04-25T01:18:03.247132+0000","last_unstale":"2024-04-25T01:18:19.412764+0000","last_undegraded":"2024-04-25T01:18:19.412764+0000","last_fullsized":"2024-04-25T01:18:19.412764+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:23:13.485645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3d","version":"42'7","reported_seq":36,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.422299+0000","last_change":"2024-04-25T01:18:03.267777+0000","last_active":"2024-04-25T01:18:19.422299+0000","last_peered":"2024-04-25T01:18:19.422299+0000","last_clean":"2024-04-25T01:18:19.422299+0000","last_became_active":"2024-04-25T01:18:03.267444+0000","last_became_peered":"2024-04-25T01:18:03.267444+0000","last_unstale":"2024-04-25T01:18:19.422299+0000","last_undegraded":"2024-04-25T01:18:19.422299+0000","last_fullsized":"2024-04-25T01:18:19.422299+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:06.961610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3c","version":"42'8","reported_seq":37,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.421434+0000","last_change":"2024-04-25T01:18:03.260302+0000","last_active":"2024-04-25T01:18:19.421434+0000","last_peered":"2024-04-25T01:18:19.421434+0000","last_clean":"2024-04-25T01:18:19.421434+0000","last_became_active":"2024-04-25T01:18:03.259967+0000","last_became_peered":"2024-04-25T01:18:03.259967+0000","last_unstale":"2024-04-25T01:18:19.421434+0000","last_undegraded":"2024-04-25T01:18:19.421434+0000","last_fullsized":"2024-04-25T01:18:19.421434+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:57:03.302298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3b","version":"42'4","reported_seq":33,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.407372+0000","last_change":"2024-04-25T01:18:03.257452+0000","last_active":"2024-04-25T01:18:19.407372+0000","last_peered":"2024-04-25T01:18:19.407372+0000","last_clean":"2024-04-25T01:18:19.407372+0000","last_became_active":"2024-04-25T01:18:03.256443+0000","last_became_peered":"2024-04-25T01:18:03.256443+0000","last_unstale":"2024-04-25T01:18:19.407372+0000","last_undegraded":"2024-04-25T01:18:19.407372+0000","last_fullsized":"2024-04-25T01:18:19.407372+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:48:59.966764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3a","version":"42'6","reported_seq":35,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.411910+0000","last_change":"2024-04-25T01:18:03.242630+0000","last_active":"2024-04-25T01:18:19.411910+0000","last_peered":"2024-04-25T01:18:19.411910+0000","last_clean":"2024-04-25T01:18:19.411910+0000","last_became_active":"2024-04-25T01:18:03.242276+0000","last_became_peered":"2024-04-25T01:18:03.242276+0000","last_unstale":"2024-04-25T01:18:19.411910+0000","last_undegraded":"2024-04-25T01:18:19.411910+0000","last_fullsized":"2024-04-25T01:18:19.411910+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:42:01.659435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.39","version":"42'5","reported_seq":33,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.393897+0000","last_change":"2024-04-25T01:18:03.257363+0000","last_active":"2024-04-25T01:18:19.393897+0000","last_peered":"2024-04-25T01:18:19.393897+0000","last_clean":"2024-04-25T01:18:19.393897+0000","last_became_active":"2024-04-25T01:18:03.256685+0000","last_became_peered":"2024-04-25T01:18:03.256685+0000","last_unstale":"2024-04-25T01:18:19.393897+0000","last_undegraded":"2024-04-25T01:18:19.393897+0000","last_fullsized":"2024-04-25T01:18:19.393897+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:19.882784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.38","version":"42'9","reported_seq":38,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.421336+0000","last_change":"2024-04-25T01:18:03.274858+0000","last_active":"2024-04-25T01:18:19.421336+0000","last_peered":"2024-04-25T01:18:19.421336+0000","last_clean":"2024-04-25T01:18:19.421336+0000","last_became_active":"2024-04-25T01:18:03.273775+0000","last_became_peered":"2024-04-25T01:18:03.273775+0000","last_unstale":"2024-04-25T01:18:19.421336+0000","last_undegraded":"2024-04-25T01:18:19.421336+0000","last_fullsized":"2024-04-25T01:18:19.421336+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:20:51.212419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.37","version":"42'11","reported_seq":40,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.414081+0000","last_change":"2024-04-25T01:18:03.265545+0000","last_active":"2024-04-25T01:18:19.414081+0000","last_peered":"2024-04-25T01:18:19.414081+0000","last_clean":"2024-04-25T01:18:19.414081+0000","last_became_active":"2024-04-25T01:18:03.265251+0000","last_became_peered":"2024-04-25T01:18:03.265251+0000","last_unstale":"2024-04-25T01:18:19.414081+0000","last_undegraded":"2024-04-25T01:18:19.414081+0000","last_fullsized":"2024-04-25T01:18:19.414081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:13.833482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.36","version":"42'15","reported_seq":43,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.396223+0000","last_change":"2024-04-25T01:18:03.267089+0000","last_active":"2024-04-25T01:18:19.396223+0000","last_peered":"2024-04-25T01:18:19.396223+0000","last_clean":"2024-04-25T01:18:19.396223+0000","last_became_active":"2024-04-25T01:18:03.266799+0000","last_became_peered":"2024-04-25T01:18:03.266799+0000","last_unstale":"2024-04-25T01:18:19.396223+0000","last_undegraded":"2024-04-25T01:18:19.396223+0000","last_fullsized":"2024-04-25T01:18:19.396223+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:57:10.276501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.35","version":"42'6","reported_seq":34,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.408260+0000","last_change":"2024-04-25T01:18:03.276878+0000","last_active":"2024-04-25T01:18:19.408260+0000","last_peered":"2024-04-25T01:18:19.408260+0000","last_clean":"2024-04-25T01:18:19.408260+0000","last_became_active":"2024-04-25T01:18:03.276396+0000","last_became_peered":"2024-04-25T01:18:03.276396+0000","last_unstale":"2024-04-25T01:18:19.408260+0000","last_undegraded":"2024-04-25T01:18:19.408260+0000","last_fullsized":"2024-04-25T01:18:19.408260+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:39.462581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.34","version":"42'7","reported_seq":36,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.408902+0000","last_change":"2024-04-25T01:18:03.249998+0000","last_active":"2024-04-25T01:18:19.408902+0000","last_peered":"2024-04-25T01:18:19.408902+0000","last_clean":"2024-04-25T01:18:19.408902+0000","last_became_active":"2024-04-25T01:18:03.249714+0000","last_became_peered":"2024-04-25T01:18:03.249714+0000","last_unstale":"2024-04-25T01:18:19.408902+0000","last_undegraded":"2024-04-25T01:18:19.408902+0000","last_fullsized":"2024-04-25T01:18:19.408902+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:19.385178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.33","version":"42'10","reported_seq":38,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.393975+0000","last_change":"2024-04-25T01:18:03.256824+0000","last_active":"2024-04-25T01:18:19.393975+0000","last_peered":"2024-04-25T01:18:19.393975+0000","last_clean":"2024-04-25T01:18:19.393975+0000","last_became_active":"2024-04-25T01:18:03.256477+0000","last_became_peered":"2024-04-25T01:18:03.256477+0000","last_unstale":"2024-04-25T01:18:19.393975+0000","last_undegraded":"2024-04-25T01:18:19.393975+0000","last_fullsized":"2024-04-25T01:18:19.393975+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:19.060283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.32","version":"42'8","reported_seq":37,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.398241+0000","last_change":"2024-04-25T01:18:03.265116+0000","last_active":"2024-04-25T01:18:19.398241+0000","last_peered":"2024-04-25T01:18:19.398241+0000","last_clean":"2024-04-25T01:18:19.398241+0000","last_became_active":"2024-04-25T01:18:03.264863+0000","last_became_peered":"2024-04-25T01:18:03.264863+0000","last_unstale":"2024-04-25T01:18:19.398241+0000","last_undegraded":"2024-04-25T01:18:19.398241+0000","last_fullsized":"2024-04-25T01:18:19.398241+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:35:34.926511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.31","version":"42'7","reported_seq":37,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.409005+0000","last_change":"2024-04-25T01:18:03.258504+0000","last_active":"2024-04-25T01:18:19.409005+0000","last_peered":"2024-04-25T01:18:19.409005+0000","last_clean":"2024-04-25T01:18:19.409005+0000","last_became_active":"2024-04-25T01:18:03.258242+0000","last_became_peered":"2024-04-25T01:18:03.258242+0000","last_unstale":"2024-04-25T01:18:19.409005+0000","last_undegraded":"2024-04-25T01:18:19.409005+0000","last_fullsized":"2024-04-25T01:18:19.409005+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:33:32.271379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.30","version":"42'9","reported_seq":37,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.393559+0000","last_change":"2024-04-25T01:18:03.261774+0000","last_active":"2024-04-25T01:18:19.393559+0000","last_peered":"2024-04-25T01:18:19.393559+0000","last_clean":"2024-04-25T01:18:19.393559+0000","last_became_active":"2024-04-25T01:18:03.261428+0000","last_became_peered":"2024-04-25T01:18:03.261428+0000","last_unstale":"2024-04-25T01:18:19.393559+0000","last_undegraded":"2024-04-25T01:18:19.393559+0000","last_fullsized":"2024-04-25T01:18:19.393559+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:55:31.639411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2f","version":"42'9","reported_seq":39,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.412342+0000","last_change":"2024-04-25T01:18:03.273618+0000","last_active":"2024-04-25T01:18:19.412342+0000","last_peered":"2024-04-25T01:18:19.412342+0000","last_clean":"2024-04-25T01:18:19.412342+0000","last_became_active":"2024-04-25T01:18:03.273161+0000","last_became_peered":"2024-04-25T01:18:03.273161+0000","last_unstale":"2024-04-25T01:18:19.412342+0000","last_undegraded":"2024-04-25T01:18:19.412342+0000","last_fullsized":"2024-04-25T01:18:19.412342+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:42:40.677650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2097152,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":2048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2e","version":"42'10","reported_seq":40,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.406922+0000","last_change":"2024-04-25T01:18:03.265079+0000","last_active":"2024-04-25T01:18:19.406922+0000","last_peered":"2024-04-25T01:18:19.406922+0000","last_clean":"2024-04-25T01:18:19.406922+0000","last_became_active":"2024-04-25T01:18:03.264708+0000","last_became_peered":"2024-04-25T01:18:03.264708+0000","last_unstale":"2024-04-25T01:18:19.406922+0000","last_undegraded":"2024-04-25T01:18:19.406922+0000","last_fullsized":"2024-04-25T01:18:19.406922+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:47:34.607606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2d","version":"42'7","reported_seq":38,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.400124+0000","last_change":"2024-04-25T01:18:03.257519+0000","last_active":"2024-04-25T01:18:19.400124+0000","last_peered":"2024-04-25T01:18:19.400124+0000","last_clean":"2024-04-25T01:18:19.400124+0000","last_became_active":"2024-04-25T01:18:03.257225+0000","last_became_peered":"2024-04-25T01:18:03.257225+0000","last_unstale":"2024-04-25T01:18:19.400124+0000","last_undegraded":"2024-04-25T01:18:19.400124+0000","last_fullsized":"2024-04-25T01:18:19.400124+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:12:04.693125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2c","version":"42'8","reported_seq":36,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.404274+0000","last_change":"2024-04-25T01:18:03.267873+0000","last_active":"2024-04-25T01:18:19.404274+0000","last_peered":"2024-04-25T01:18:19.404274+0000","last_clean":"2024-04-25T01:18:19.404274+0000","last_became_active":"2024-04-25T01:18:03.267601+0000","last_became_peered":"2024-04-25T01:18:03.267601+0000","last_unstale":"2024-04-25T01:18:19.404274+0000","last_undegraded":"2024-04-25T01:18:19.404274+0000","last_fullsized":"2024-04-25T01:18:19.404274+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:38:56.106596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2b","version":"42'11","reported_seq":39,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.404116+0000","last_change":"2024-04-25T01:18:03.256260+0000","last_active":"2024-04-25T01:18:19.404116+0000","last_peered":"2024-04-25T01:18:19.404116+0000","last_clean":"2024-04-25T01:18:19.404116+0000","last_became_active":"2024-04-25T01:18:03.255488+0000","last_became_peered":"2024-04-25T01:18:03.255488+0000","last_unstale":"2024-04-25T01:18:19.404116+0000","last_undegraded":"2024-04-25T01:18:19.404116+0000","last_fullsized":"2024-04-25T01:18:19.404116+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:06:25.097573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2a","version":"42'9","reported_seq":37,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.405642+0000","last_change":"2024-04-25T01:18:03.262912+0000","last_active":"2024-04-25T01:18:19.405642+0000","last_peered":"2024-04-25T01:18:19.405642+0000","last_clean":"2024-04-25T01:18:19.405642+0000","last_became_active":"2024-04-25T01:18:03.262598+0000","last_became_peered":"2024-04-25T01:18:03.262598+0000","last_unstale":"2024-04-25T01:18:19.405642+0000","last_undegraded":"2024-04-25T01:18:19.405642+0000","last_fullsized":"2024-04-25T01:18:19.405642+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:41:12.326642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.29","version":"42'8","reported_seq":38,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.409114+0000","last_change":"2024-04-25T01:18:03.259084+0000","last_active":"2024-04-25T01:18:19.409114+0000","last_peered":"2024-04-25T01:18:19.409114+0000","last_clean":"2024-04-25T01:18:19.409114+0000","last_became_active":"2024-04-25T01:18:03.258651+0000","last_became_peered":"2024-04-25T01:18:03.258651+0000","last_unstale":"2024-04-25T01:18:19.409114+0000","last_undegraded":"2024-04-25T01:18:19.409114+0000","last_fullsized":"2024-04-25T01:18:19.409114+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:08:28.501544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.28","version":"42'6","reported_seq":35,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.421564+0000","last_change":"2024-04-25T01:18:03.270002+0000","last_active":"2024-04-25T01:18:19.421564+0000","last_peered":"2024-04-25T01:18:19.421564+0000","last_clean":"2024-04-25T01:18:19.421564+0000","last_became_active":"2024-04-25T01:18:03.269711+0000","last_became_peered":"2024-04-25T01:18:03.269711+0000","last_unstale":"2024-04-25T01:18:19.421564+0000","last_undegraded":"2024-04-25T01:18:19.421564+0000","last_fullsized":"2024-04-25T01:18:19.421564+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:20:54.301682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.27","version":"42'2","reported_seq":31,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.399760+0000","last_change":"2024-04-25T01:18:03.265266+0000","last_active":"2024-04-25T01:18:19.399760+0000","last_peered":"2024-04-25T01:18:19.399760+0000","last_clean":"2024-04-25T01:18:19.399760+0000","last_became_active":"2024-04-25T01:18:03.264973+0000","last_became_peered":"2024-04-25T01:18:03.264973+0000","last_unstale":"2024-04-25T01:18:19.399760+0000","last_undegraded":"2024-04-25T01:18:19.399760+0000","last_fullsized":"2024-04-25T01:18:19.399760+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:30:00.416602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.26","version":"42'15","reported_seq":44,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.421901+0000","last_change":"2024-04-25T01:18:03.263008+0000","last_active":"2024-04-25T01:18:19.421901+0000","last_peered":"2024-04-25T01:18:19.421901+0000","last_clean":"2024-04-25T01:18:19.421901+0000","last_became_active":"2024-04-25T01:18:03.262081+0000","last_became_peered":"2024-04-25T01:18:03.262081+0000","last_unstale":"2024-04-25T01:18:19.421901+0000","last_undegraded":"2024-04-25T01:18:19.421901+0000","last_fullsized":"2024-04-25T01:18:19.421901+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:32:52.505647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.25","version":"42'6","reported_seq":34,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.408617+0000","last_change":"2024-04-25T01:18:03.263526+0000","last_active":"2024-04-25T01:18:19.408617+0000","last_peered":"2024-04-25T01:18:19.408617+0000","last_clean":"2024-04-25T01:18:19.408617+0000","last_became_active":"2024-04-25T01:18:03.258646+0000","last_became_peered":"2024-04-25T01:18:03.258646+0000","last_unstale":"2024-04-25T01:18:19.408617+0000","last_undegraded":"2024-04-25T01:18:19.408617+0000","last_fullsized":"2024-04-25T01:18:19.408617+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:34:33.950222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.24","version":"42'4","reported_seq":32,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.394633+0000","last_change":"2024-04-25T01:18:03.267937+0000","last_active":"2024-04-25T01:18:19.394633+0000","last_peered":"2024-04-25T01:18:19.394633+0000","last_clean":"2024-04-25T01:18:19.394633+0000","last_became_active":"2024-04-25T01:18:03.266925+0000","last_became_peered":"2024-04-25T01:18:03.266925+0000","last_unstale":"2024-04-25T01:18:19.394633+0000","last_undegraded":"2024-04-25T01:18:19.394633+0000","last_fullsized":"2024-04-25T01:18:19.394633+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:18:05.301920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.23","version":"42'7","reported_seq":37,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.409268+0000","last_change":"2024-04-25T01:18:03.253210+0000","last_active":"2024-04-25T01:18:19.409268+0000","last_peered":"2024-04-25T01:18:19.409268+0000","last_clean":"2024-04-25T01:18:19.409268+0000","last_became_active":"2024-04-25T01:18:03.252930+0000","last_became_peered":"2024-04-25T01:18:03.252930+0000","last_unstale":"2024-04-25T01:18:19.409268+0000","last_undegraded":"2024-04-25T01:18:19.409268+0000","last_fullsized":"2024-04-25T01:18:19.409268+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:56:17.738368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.22","version":"42'8","reported_seq":38,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.410148+0000","last_change":"2024-04-25T01:18:03.255049+0000","last_active":"2024-04-25T01:18:19.410148+0000","last_peered":"2024-04-25T01:18:19.410148+0000","last_clean":"2024-04-25T01:18:19.410148+0000","last_became_active":"2024-04-25T01:18:03.254795+0000","last_became_peered":"2024-04-25T01:18:03.254795+0000","last_unstale":"2024-04-25T01:18:19.410148+0000","last_undegraded":"2024-04-25T01:18:19.410148+0000","last_fullsized":"2024-04-25T01:18:19.410148+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:31:38.615550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.21","version":"42'12","reported_seq":40,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.394480+0000","last_change":"2024-04-25T01:18:03.257602+0000","last_active":"2024-04-25T01:18:19.394480+0000","last_peered":"2024-04-25T01:18:19.394480+0000","last_clean":"2024-04-25T01:18:19.394480+0000","last_became_active":"2024-04-25T01:18:03.257303+0000","last_became_peered":"2024-04-25T01:18:03.257303+0000","last_unstale":"2024-04-25T01:18:19.394480+0000","last_undegraded":"2024-04-25T01:18:19.394480+0000","last_fullsized":"2024-04-25T01:18:19.394480+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:35:34.926511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.20","version":"42'9","reported_seq":32,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.399253+0000","last_change":"2024-04-25T01:18:10.498531+0000","last_active":"2024-04-25T01:18:19.399253+0000","last_peered":"2024-04-25T01:18:19.399253+0000","last_clean":"2024-04-25T01:18:19.399253+0000","last_became_active":"2024-04-25T01:18:10.497858+0000","last_became_peered":"2024-04-25T01:18:10.497858+0000","last_unstale":"2024-04-25T01:18:19.399253+0000","last_undegraded":"2024-04-25T01:18:19.399253+0000","last_fullsized":"2024-04-25T01:18:19.399253+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:50:36.431482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.425409+0000","last_change":"2024-04-25T01:18:19.416518+0000","last_active":"2024-04-25T01:18:19.425409+0000","last_peered":"2024-04-25T01:18:19.425409+0000","last_clean":"2024-04-25T01:18:19.425409+0000","last_became_active":"2024-04-25T01:18:19.415975+0000","last_became_peered":"2024-04-25T01:18:19.415975+0000","last_unstale":"2024-04-25T01:18:19.425409+0000","last_undegraded":"2024-04-25T01:18:19.425409+0000","last_fullsized":"2024-04-25T01:18:19.425409+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:55:53.445775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"42'9","reported_seq":39,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.411290+0000","last_change":"2024-04-25T01:18:03.264199+0000","last_active":"2024-04-25T01:18:19.411290+0000","last_peered":"2024-04-25T01:18:19.411290+0000","last_clean":"2024-04-25T01:18:19.411290+0000","last_became_active":"2024-04-25T01:18:03.260827+0000","last_became_peered":"2024-04-25T01:18:03.260827+0000","last_unstale":"2024-04-25T01:18:19.411290+0000","last_undegraded":"2024-04-25T01:18:19.411290+0000","last_fullsized":"2024-04-25T01:18:19.411290+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:39:15.865514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"43'8","reported_seq":38,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.400042+0000","last_change":"2024-04-25T01:18:02.263556+0000","last_active":"2024-04-25T01:18:19.400042+0000","last_peered":"2024-04-25T01:18:19.400042+0000","last_clean":"2024-04-25T01:18:19.400042+0000","last_became_active":"2024-04-25T01:18:02.263123+0000","last_became_peered":"2024-04-25T01:18:02.263123+0000","last_unstale":"2024-04-25T01:18:19.400042+0000","last_undegraded":"2024-04-25T01:18:19.400042+0000","last_fullsized":"2024-04-25T01:18:19.400042+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:09:21.858773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.430342+0000","last_change":"2024-04-25T01:18:19.426375+0000","last_active":"2024-04-25T01:18:19.430342+0000","last_peered":"2024-04-25T01:18:19.430342+0000","last_clean":"2024-04-25T01:18:19.430342+0000","last_became_active":"2024-04-25T01:18:19.425993+0000","last_became_peered":"2024-04-25T01:18:19.425993+0000","last_unstale":"2024-04-25T01:18:19.430342+0000","last_undegraded":"2024-04-25T01:18:19.430342+0000","last_fullsized":"2024-04-25T01:18:19.430342+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:30:55.006132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"42'15","reported_seq":44,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.412649+0000","last_change":"2024-04-25T01:18:03.269144+0000","last_active":"2024-04-25T01:18:19.412649+0000","last_peered":"2024-04-25T01:18:19.412649+0000","last_clean":"2024-04-25T01:18:19.412649+0000","last_became_active":"2024-04-25T01:18:03.268851+0000","last_became_peered":"2024-04-25T01:18:03.268851+0000","last_unstale":"2024-04-25T01:18:19.412649+0000","last_undegraded":"2024-04-25T01:18:19.412649+0000","last_fullsized":"2024-04-25T01:18:19.412649+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:55:37.223439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"39'1","reported_seq":31,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.413037+0000","last_change":"2024-04-25T01:18:02.255987+0000","last_active":"2024-04-25T01:18:19.413037+0000","last_peered":"2024-04-25T01:18:19.413037+0000","last_clean":"2024-04-25T01:18:19.413037+0000","last_became_active":"2024-04-25T01:18:02.255540+0000","last_became_peered":"2024-04-25T01:18:02.255540+0000","last_unstale":"2024-04-25T01:18:19.413037+0000","last_undegraded":"2024-04-25T01:18:19.413037+0000","last_fullsized":"2024-04-25T01:18:19.413037+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:38:12.422806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.422350+0000","last_change":"2024-04-25T01:18:19.418961+0000","last_active":"2024-04-25T01:18:19.422350+0000","last_peered":"2024-04-25T01:18:19.422350+0000","last_clean":"2024-04-25T01:18:19.422350+0000","last_became_active":"2024-04-25T01:18:19.418629+0000","last_became_peered":"2024-04-25T01:18:19.418629+0000","last_unstale":"2024-04-25T01:18:19.422350+0000","last_undegraded":"2024-04-25T01:18:19.422350+0000","last_fullsized":"2024-04-25T01:18:19.422350+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:54:12.857479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"42'5","reported_seq":35,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.407177+0000","last_change":"2024-04-25T01:18:03.260381+0000","last_active":"2024-04-25T01:18:19.407177+0000","last_peered":"2024-04-25T01:18:19.407177+0000","last_clean":"2024-04-25T01:18:19.407177+0000","last_became_active":"2024-04-25T01:18:03.260114+0000","last_became_peered":"2024-04-25T01:18:03.260114+0000","last_unstale":"2024-04-25T01:18:19.407177+0000","last_undegraded":"2024-04-25T01:18:19.407177+0000","last_fullsized":"2024-04-25T01:18:19.407177+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:10.899004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"43'3","reported_seq":33,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.422765+0000","last_change":"2024-04-25T01:18:02.270338+0000","last_active":"2024-04-25T01:18:19.422765+0000","last_peered":"2024-04-25T01:18:19.422765+0000","last_clean":"2024-04-25T01:18:19.422765+0000","last_became_active":"2024-04-25T01:18:02.270063+0000","last_became_peered":"2024-04-25T01:18:02.270063+0000","last_unstale":"2024-04-25T01:18:19.422765+0000","last_undegraded":"2024-04-25T01:18:19.422765+0000","last_fullsized":"2024-04-25T01:18:19.422765+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:25:09.988571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.431621+0000","last_change":"2024-04-25T01:18:19.430171+0000","last_active":"2024-04-25T01:18:19.431621+0000","last_peered":"2024-04-25T01:18:19.431621+0000","last_clean":"2024-04-25T01:18:19.431621+0000","last_became_active":"2024-04-25T01:18:19.429361+0000","last_became_peered":"2024-04-25T01:18:19.429361+0000","last_unstale":"2024-04-25T01:18:19.431621+0000","last_undegraded":"2024-04-25T01:18:19.431621+0000","last_fullsized":"2024-04-25T01:18:19.431621+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:55:47.861748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1f","version":"42'6","reported_seq":35,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.413019+0000","last_change":"2024-04-25T01:18:03.264063+0000","last_active":"2024-04-25T01:18:19.413019+0000","last_peered":"2024-04-25T01:18:19.413019+0000","last_clean":"2024-04-25T01:18:19.413019+0000","last_became_active":"2024-04-25T01:18:03.263708+0000","last_became_peered":"2024-04-25T01:18:03.263708+0000","last_unstale":"2024-04-25T01:18:19.413019+0000","last_undegraded":"2024-04-25T01:18:19.413019+0000","last_fullsized":"2024-04-25T01:18:19.413019+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:15.465731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"43'3","reported_seq":33,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.399076+0000","last_change":"2024-04-25T01:18:02.263250+0000","last_active":"2024-04-25T01:18:19.399076+0000","last_peered":"2024-04-25T01:18:19.399076+0000","last_clean":"2024-04-25T01:18:19.399076+0000","last_became_active":"2024-04-25T01:18:02.262763+0000","last_became_peered":"2024-04-25T01:18:02.262763+0000","last_unstale":"2024-04-25T01:18:19.399076+0000","last_undegraded":"2024-04-25T01:18:19.399076+0000","last_fullsized":"2024-04-25T01:18:19.399076+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:22:34.107130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.425141+0000","last_change":"2024-04-25T01:18:19.420413+0000","last_active":"2024-04-25T01:18:19.425141+0000","last_peered":"2024-04-25T01:18:19.425141+0000","last_clean":"2024-04-25T01:18:19.425141+0000","last_became_active":"2024-04-25T01:18:19.420145+0000","last_became_peered":"2024-04-25T01:18:19.420145+0000","last_unstale":"2024-04-25T01:18:19.425141+0000","last_undegraded":"2024-04-25T01:18:19.425141+0000","last_fullsized":"2024-04-25T01:18:19.425141+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:00:19.438096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"42'8","reported_seq":37,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.421842+0000","last_change":"2024-04-25T01:18:03.250303+0000","last_active":"2024-04-25T01:18:19.421842+0000","last_peered":"2024-04-25T01:18:19.421842+0000","last_clean":"2024-04-25T01:18:19.421842+0000","last_became_active":"2024-04-25T01:18:03.249995+0000","last_became_peered":"2024-04-25T01:18:03.249995+0000","last_unstale":"2024-04-25T01:18:19.421842+0000","last_undegraded":"2024-04-25T01:18:19.421842+0000","last_fullsized":"2024-04-25T01:18:19.421842+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:11.366143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"39'1","reported_seq":31,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.401035+0000","last_change":"2024-04-25T01:18:02.270642+0000","last_active":"2024-04-25T01:18:19.401035+0000","last_peered":"2024-04-25T01:18:19.401035+0000","last_clean":"2024-04-25T01:18:19.401035+0000","last_became_active":"2024-04-25T01:18:02.270383+0000","last_became_peered":"2024-04-25T01:18:02.270383+0000","last_unstale":"2024-04-25T01:18:19.401035+0000","last_undegraded":"2024-04-25T01:18:19.401035+0000","last_fullsized":"2024-04-25T01:18:19.401035+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:20:17.417113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.428101+0000","last_change":"2024-04-25T01:18:19.424356+0000","last_active":"2024-04-25T01:18:19.428101+0000","last_peered":"2024-04-25T01:18:19.428101+0000","last_clean":"2024-04-25T01:18:19.428101+0000","last_became_active":"2024-04-25T01:18:19.423905+0000","last_became_peered":"2024-04-25T01:18:19.423905+0000","last_unstale":"2024-04-25T01:18:19.428101+0000","last_undegraded":"2024-04-25T01:18:19.428101+0000","last_fullsized":"2024-04-25T01:18:19.428101+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:45:25.608799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"42'10","reported_seq":38,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.404680+0000","last_change":"2024-04-25T01:18:03.262229+0000","last_active":"2024-04-25T01:18:19.404680+0000","last_peered":"2024-04-25T01:18:19.404680+0000","last_clean":"2024-04-25T01:18:19.404680+0000","last_became_active":"2024-04-25T01:18:03.256356+0000","last_became_peered":"2024-04-25T01:18:03.256356+0000","last_unstale":"2024-04-25T01:18:19.404680+0000","last_undegraded":"2024-04-25T01:18:19.404680+0000","last_fullsized":"2024-04-25T01:18:19.404680+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:10:58.007657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"39'1","reported_seq":30,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.394763+0000","last_change":"2024-04-25T01:18:02.267951+0000","last_active":"2024-04-25T01:18:19.394763+0000","last_peered":"2024-04-25T01:18:19.394763+0000","last_clean":"2024-04-25T01:18:19.394763+0000","last_became_active":"2024-04-25T01:18:02.267086+0000","last_became_peered":"2024-04-25T01:18:02.267086+0000","last_unstale":"2024-04-25T01:18:19.394763+0000","last_undegraded":"2024-04-25T01:18:19.394763+0000","last_fullsized":"2024-04-25T01:18:19.394763+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:03:28.616338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.429647+0000","last_change":"2024-04-25T01:18:19.428103+0000","last_active":"2024-04-25T01:18:19.429647+0000","last_peered":"2024-04-25T01:18:19.429647+0000","last_clean":"2024-04-25T01:18:19.429647+0000","last_became_active":"2024-04-25T01:18:19.427765+0000","last_became_peered":"2024-04-25T01:18:19.427765+0000","last_unstale":"2024-04-25T01:18:19.429647+0000","last_undegraded":"2024-04-25T01:18:19.429647+0000","last_fullsized":"2024-04-25T01:18:19.429647+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:08:44.723881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"42'7","reported_seq":37,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.410279+0000","last_change":"2024-04-25T01:18:03.265044+0000","last_active":"2024-04-25T01:18:19.410279+0000","last_peered":"2024-04-25T01:18:19.410279+0000","last_clean":"2024-04-25T01:18:19.410279+0000","last_became_active":"2024-04-25T01:18:03.264645+0000","last_became_peered":"2024-04-25T01:18:03.264645+0000","last_unstale":"2024-04-25T01:18:19.410279+0000","last_undegraded":"2024-04-25T01:18:19.410279+0000","last_fullsized":"2024-04-25T01:18:19.410279+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:41:51.721827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":30,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.421355+0000","last_change":"2024-04-25T01:18:02.262809+0000","last_active":"2024-04-25T01:18:19.421355+0000","last_peered":"2024-04-25T01:18:19.421355+0000","last_clean":"2024-04-25T01:18:19.421355+0000","last_became_active":"2024-04-25T01:18:02.261620+0000","last_became_peered":"2024-04-25T01:18:02.261620+0000","last_unstale":"2024-04-25T01:18:19.421355+0000","last_undegraded":"2024-04-25T01:18:19.421355+0000","last_fullsized":"2024-04-25T01:18:19.421355+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:12.817891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.419684+0000","last_change":"2024-04-25T01:18:19.414293+0000","last_active":"2024-04-25T01:18:19.419684+0000","last_peered":"2024-04-25T01:18:19.419684+0000","last_clean":"2024-04-25T01:18:19.419684+0000","last_became_active":"2024-04-25T01:18:19.414004+0000","last_became_peered":"2024-04-25T01:18:19.414004+0000","last_unstale":"2024-04-25T01:18:19.419684+0000","last_undegraded":"2024-04-25T01:18:19.419684+0000","last_fullsized":"2024-04-25T01:18:19.419684+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:15:26.510336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"42'8","reported_seq":37,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.411498+0000","last_change":"2024-04-25T01:18:03.261875+0000","last_active":"2024-04-25T01:18:19.411498+0000","last_peered":"2024-04-25T01:18:19.411498+0000","last_clean":"2024-04-25T01:18:19.411498+0000","last_became_active":"2024-04-25T01:18:03.261413+0000","last_became_peered":"2024-04-25T01:18:03.261413+0000","last_unstale":"2024-04-25T01:18:19.411498+0000","last_undegraded":"2024-04-25T01:18:19.411498+0000","last_fullsized":"2024-04-25T01:18:19.411498+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:47.778367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":29,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.393687+0000","last_change":"2024-04-25T01:18:02.257390+0000","last_active":"2024-04-25T01:18:19.393687+0000","last_peered":"2024-04-25T01:18:19.393687+0000","last_clean":"2024-04-25T01:18:19.393687+0000","last_became_active":"2024-04-25T01:18:02.257118+0000","last_became_peered":"2024-04-25T01:18:02.257118+0000","last_unstale":"2024-04-25T01:18:19.393687+0000","last_undegraded":"2024-04-25T01:18:19.393687+0000","last_fullsized":"2024-04-25T01:18:19.393687+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:49:10.075668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.430677+0000","last_change":"2024-04-25T01:18:19.426279+0000","last_active":"2024-04-25T01:18:19.430677+0000","last_peered":"2024-04-25T01:18:19.430677+0000","last_clean":"2024-04-25T01:18:19.430677+0000","last_became_active":"2024-04-25T01:18:19.425962+0000","last_became_peered":"2024-04-25T01:18:19.425962+0000","last_unstale":"2024-04-25T01:18:19.430677+0000","last_undegraded":"2024-04-25T01:18:19.430677+0000","last_fullsized":"2024-04-25T01:18:19.430677+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:54:38.746281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"42'11","reported_seq":40,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.421727+0000","last_change":"2024-04-25T01:18:03.260224+0000","last_active":"2024-04-25T01:18:19.421727+0000","last_peered":"2024-04-25T01:18:19.421727+0000","last_clean":"2024-04-25T01:18:19.421727+0000","last_became_active":"2024-04-25T01:18:03.259886+0000","last_became_peered":"2024-04-25T01:18:03.259886+0000","last_unstale":"2024-04-25T01:18:19.421727+0000","last_undegraded":"2024-04-25T01:18:19.421727+0000","last_fullsized":"2024-04-25T01:18:19.421727+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:24:54.943447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":30,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.422622+0000","last_change":"2024-04-25T01:18:02.265416+0000","last_active":"2024-04-25T01:18:19.422622+0000","last_peered":"2024-04-25T01:18:19.422622+0000","last_clean":"2024-04-25T01:18:19.422622+0000","last_became_active":"2024-04-25T01:18:02.265123+0000","last_became_peered":"2024-04-25T01:18:02.265123+0000","last_unstale":"2024-04-25T01:18:19.422622+0000","last_undegraded":"2024-04-25T01:18:19.422622+0000","last_fullsized":"2024-04-25T01:18:19.422622+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:18.369587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.430059+0000","last_change":"2024-04-25T01:18:19.427690+0000","last_active":"2024-04-25T01:18:19.430059+0000","last_peered":"2024-04-25T01:18:19.430059+0000","last_clean":"2024-04-25T01:18:19.430059+0000","last_became_active":"2024-04-25T01:18:19.427363+0000","last_became_peered":"2024-04-25T01:18:19.427363+0000","last_unstale":"2024-04-25T01:18:19.430059+0000","last_undegraded":"2024-04-25T01:18:19.430059+0000","last_fullsized":"2024-04-25T01:18:19.430059+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:50:13.885015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"42'6","reported_seq":35,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.398240+0000","last_change":"2024-04-25T01:18:03.279931+0000","last_active":"2024-04-25T01:18:19.398240+0000","last_peered":"2024-04-25T01:18:19.398240+0000","last_clean":"2024-04-25T01:18:19.398240+0000","last_became_active":"2024-04-25T01:18:03.279643+0000","last_became_peered":"2024-04-25T01:18:03.279643+0000","last_unstale":"2024-04-25T01:18:19.398240+0000","last_undegraded":"2024-04-25T01:18:19.398240+0000","last_fullsized":"2024-04-25T01:18:19.398240+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:18:05.301920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":31,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.407020+0000","last_change":"2024-04-25T01:18:02.256098+0000","last_active":"2024-04-25T01:18:19.407020+0000","last_peered":"2024-04-25T01:18:19.407020+0000","last_clean":"2024-04-25T01:18:19.407020+0000","last_became_active":"2024-04-25T01:18:02.255609+0000","last_became_peered":"2024-04-25T01:18:02.255609+0000","last_unstale":"2024-04-25T01:18:19.407020+0000","last_undegraded":"2024-04-25T01:18:19.407020+0000","last_fullsized":"2024-04-25T01:18:19.407020+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:46:28.843042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.412953+0000","last_change":"2024-04-25T01:18:19.410836+0000","last_active":"2024-04-25T01:18:19.412953+0000","last_peered":"2024-04-25T01:18:19.412953+0000","last_clean":"2024-04-25T01:18:19.412953+0000","last_became_active":"2024-04-25T01:18:19.410588+0000","last_became_peered":"2024-04-25T01:18:19.410588+0000","last_unstale":"2024-04-25T01:18:19.412953+0000","last_undegraded":"2024-04-25T01:18:19.412953+0000","last_fullsized":"2024-04-25T01:18:19.412953+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:30:55.006132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"42'10","reported_seq":33,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.399420+0000","last_change":"2024-04-25T01:18:10.498339+0000","last_active":"2024-04-25T01:18:19.399420+0000","last_peered":"2024-04-25T01:18:19.399420+0000","last_clean":"2024-04-25T01:18:19.399420+0000","last_became_active":"2024-04-25T01:18:10.497866+0000","last_became_peered":"2024-04-25T01:18:10.497866+0000","last_unstale":"2024-04-25T01:18:19.399420+0000","last_undegraded":"2024-04-25T01:18:19.399420+0000","last_fullsized":"2024-04-25T01:18:19.399420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:57:10.276501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"43'5","reported_seq":36,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.410289+0000","last_change":"2024-04-25T01:18:02.263678+0000","last_active":"2024-04-25T01:18:19.410289+0000","last_peered":"2024-04-25T01:18:19.410289+0000","last_clean":"2024-04-25T01:18:19.410289+0000","last_became_active":"2024-04-25T01:18:02.263328+0000","last_became_peered":"2024-04-25T01:18:02.263328+0000","last_unstale":"2024-04-25T01:18:19.410289+0000","last_undegraded":"2024-04-25T01:18:19.410289+0000","last_fullsized":"2024-04-25T01:18:19.410289+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:30:37.768204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.427942+0000","last_change":"2024-04-25T01:18:19.423579+0000","last_active":"2024-04-25T01:18:19.427942+0000","last_peered":"2024-04-25T01:18:19.427942+0000","last_clean":"2024-04-25T01:18:19.427942+0000","last_became_active":"2024-04-25T01:18:19.423281+0000","last_became_peered":"2024-04-25T01:18:19.423281+0000","last_unstale":"2024-04-25T01:18:19.427942+0000","last_undegraded":"2024-04-25T01:18:19.427942+0000","last_fullsized":"2024-04-25T01:18:19.427942+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:46:46.080969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"42'9","reported_seq":37,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.396763+0000","last_change":"2024-04-25T01:18:03.257743+0000","last_active":"2024-04-25T01:18:19.396763+0000","last_peered":"2024-04-25T01:18:19.396763+0000","last_clean":"2024-04-25T01:18:19.396763+0000","last_became_active":"2024-04-25T01:18:03.257421+0000","last_became_peered":"2024-04-25T01:18:03.257421+0000","last_unstale":"2024-04-25T01:18:19.396763+0000","last_undegraded":"2024-04-25T01:18:19.396763+0000","last_fullsized":"2024-04-25T01:18:19.396763+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:40.084475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.14","version":"43'23","reported_seq":53,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.421843+0000","last_change":"2024-04-25T01:18:02.269681+0000","last_active":"2024-04-25T01:18:19.421843+0000","last_peered":"2024-04-25T01:18:19.421843+0000","last_clean":"2024-04-25T01:18:19.421843+0000","last_became_active":"2024-04-25T01:18:02.268702+0000","last_became_peered":"2024-04-25T01:18:02.268702+0000","last_unstale":"2024-04-25T01:18:19.421843+0000","last_undegraded":"2024-04-25T01:18:19.421843+0000","last_fullsized":"2024-04-25T01:18:19.421843+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:50:13.482420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.416494+0000","last_change":"2024-04-25T01:18:19.411722+0000","last_active":"2024-04-25T01:18:19.416494+0000","last_peered":"2024-04-25T01:18:19.416494+0000","last_clean":"2024-04-25T01:18:19.416494+0000","last_became_active":"2024-04-25T01:18:19.411432+0000","last_became_peered":"2024-04-25T01:18:19.411432+0000","last_unstale":"2024-04-25T01:18:19.416494+0000","last_undegraded":"2024-04-25T01:18:19.416494+0000","last_fullsized":"2024-04-25T01:18:19.416494+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:06:50.650712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"42'6","reported_seq":35,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.422070+0000","last_change":"2024-04-25T01:18:03.269967+0000","last_active":"2024-04-25T01:18:19.422070+0000","last_peered":"2024-04-25T01:18:19.422070+0000","last_clean":"2024-04-25T01:18:19.422070+0000","last_became_active":"2024-04-25T01:18:03.269663+0000","last_became_peered":"2024-04-25T01:18:03.269663+0000","last_unstale":"2024-04-25T01:18:19.422070+0000","last_undegraded":"2024-04-25T01:18:19.422070+0000","last_fullsized":"2024-04-25T01:18:19.422070+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:32:32.815305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"43'5","reported_seq":35,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.421509+0000","last_change":"2024-04-25T01:18:02.272092+0000","last_active":"2024-04-25T01:18:19.421509+0000","last_peered":"2024-04-25T01:18:19.421509+0000","last_clean":"2024-04-25T01:18:19.421509+0000","last_became_active":"2024-04-25T01:18:02.271767+0000","last_became_peered":"2024-04-25T01:18:02.271767+0000","last_unstale":"2024-04-25T01:18:19.421509+0000","last_undegraded":"2024-04-25T01:18:19.421509+0000","last_fullsized":"2024-04-25T01:18:19.421509+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:10.420303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.427794+0000","last_change":"2024-04-25T01:18:19.425131+0000","last_active":"2024-04-25T01:18:19.427794+0000","last_peered":"2024-04-25T01:18:19.427794+0000","last_clean":"2024-04-25T01:18:19.427794+0000","last_became_active":"2024-04-25T01:18:19.424804+0000","last_became_peered":"2024-04-25T01:18:19.424804+0000","last_unstale":"2024-04-25T01:18:19.427794+0000","last_undegraded":"2024-04-25T01:18:19.427794+0000","last_fullsized":"2024-04-25T01:18:19.427794+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:25:27.226499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.11","version":"42'6","reported_seq":35,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.412059+0000","last_change":"2024-04-25T01:18:03.258055+0000","last_active":"2024-04-25T01:18:19.412059+0000","last_peered":"2024-04-25T01:18:19.412059+0000","last_clean":"2024-04-25T01:18:19.412059+0000","last_became_active":"2024-04-25T01:18:03.257517+0000","last_became_peered":"2024-04-25T01:18:03.257517+0000","last_unstale":"2024-04-25T01:18:19.412059+0000","last_undegraded":"2024-04-25T01:18:19.412059+0000","last_fullsized":"2024-04-25T01:18:19.412059+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:47:02.891879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":30,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.412245+0000","last_change":"2024-04-25T01:18:02.258795+0000","last_active":"2024-04-25T01:18:19.412245+0000","last_peered":"2024-04-25T01:18:19.412245+0000","last_clean":"2024-04-25T01:18:19.412245+0000","last_became_active":"2024-04-25T01:18:02.258494+0000","last_became_peered":"2024-04-25T01:18:02.258494+0000","last_unstale":"2024-04-25T01:18:19.412245+0000","last_undegraded":"2024-04-25T01:18:19.412245+0000","last_fullsized":"2024-04-25T01:18:19.412245+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:16:48.693012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.422756+0000","last_change":"2024-04-25T01:18:19.419126+0000","last_active":"2024-04-25T01:18:19.422756+0000","last_peered":"2024-04-25T01:18:19.422756+0000","last_clean":"2024-04-25T01:18:19.422756+0000","last_became_active":"2024-04-25T01:18:19.418836+0000","last_became_peered":"2024-04-25T01:18:19.418836+0000","last_unstale":"2024-04-25T01:18:19.422756+0000","last_undegraded":"2024-04-25T01:18:19.422756+0000","last_fullsized":"2024-04-25T01:18:19.422756+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:36.248433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"42'10","reported_seq":38,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.406704+0000","last_change":"2024-04-25T01:18:03.254569+0000","last_active":"2024-04-25T01:18:19.406704+0000","last_peered":"2024-04-25T01:18:19.406704+0000","last_clean":"2024-04-25T01:18:19.406704+0000","last_became_active":"2024-04-25T01:18:03.254299+0000","last_became_peered":"2024-04-25T01:18:03.254299+0000","last_unstale":"2024-04-25T01:18:19.406704+0000","last_undegraded":"2024-04-25T01:18:19.406704+0000","last_fullsized":"2024-04-25T01:18:19.406704+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:23:03.906624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":31,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.409225+0000","last_change":"2024-04-25T01:18:02.254663+0000","last_active":"2024-04-25T01:18:19.409225+0000","last_peered":"2024-04-25T01:18:19.409225+0000","last_clean":"2024-04-25T01:18:19.409225+0000","last_became_active":"2024-04-25T01:18:02.254405+0000","last_became_peered":"2024-04-25T01:18:02.254405+0000","last_unstale":"2024-04-25T01:18:19.409225+0000","last_undegraded":"2024-04-25T01:18:19.409225+0000","last_fullsized":"2024-04-25T01:18:19.409225+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:00:56.792493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.409394+0000","last_change":"2024-04-25T01:18:19.407532+0000","last_active":"2024-04-25T01:18:19.409394+0000","last_peered":"2024-04-25T01:18:19.409394+0000","last_clean":"2024-04-25T01:18:19.409394+0000","last_became_active":"2024-04-25T01:18:19.406223+0000","last_became_peered":"2024-04-25T01:18:19.406223+0000","last_unstale":"2024-04-25T01:18:19.409394+0000","last_undegraded":"2024-04-25T01:18:19.409394+0000","last_fullsized":"2024-04-25T01:18:19.409394+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:01:14.030421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"42'15","reported_seq":44,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.407855+0000","last_change":"2024-04-25T01:18:03.252241+0000","last_active":"2024-04-25T01:18:19.407855+0000","last_peered":"2024-04-25T01:18:19.407855+0000","last_clean":"2024-04-25T01:18:19.407855+0000","last_became_active":"2024-04-25T01:18:03.251874+0000","last_became_peered":"2024-04-25T01:18:03.251874+0000","last_unstale":"2024-04-25T01:18:19.407855+0000","last_undegraded":"2024-04-25T01:18:19.407855+0000","last_fullsized":"2024-04-25T01:18:19.407855+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:11.435894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"43'15","reported_seq":45,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.400045+0000","last_change":"2024-04-25T01:18:02.258502+0000","last_active":"2024-04-25T01:18:19.400045+0000","last_peered":"2024-04-25T01:18:19.400045+0000","last_clean":"2024-04-25T01:18:19.400045+0000","last_became_active":"2024-04-25T01:18:02.258205+0000","last_became_peered":"2024-04-25T01:18:02.258205+0000","last_unstale":"2024-04-25T01:18:19.400045+0000","last_undegraded":"2024-04-25T01:18:19.400045+0000","last_fullsized":"2024-04-25T01:18:19.400045+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:20:28.341160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.430613+0000","last_change":"2024-04-25T01:18:19.425664+0000","last_active":"2024-04-25T01:18:19.430613+0000","last_peered":"2024-04-25T01:18:19.430613+0000","last_clean":"2024-04-25T01:18:19.430613+0000","last_became_active":"2024-04-25T01:18:19.425355+0000","last_became_peered":"2024-04-25T01:18:19.425355+0000","last_unstale":"2024-04-25T01:18:19.430613+0000","last_undegraded":"2024-04-25T01:18:19.430613+0000","last_fullsized":"2024-04-25T01:18:19.430613+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:10:04.612611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"42'6","reported_seq":35,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.412163+0000","last_change":"2024-04-25T01:18:03.275449+0000","last_active":"2024-04-25T01:18:19.412163+0000","last_peered":"2024-04-25T01:18:19.412163+0000","last_clean":"2024-04-25T01:18:19.412163+0000","last_became_active":"2024-04-25T01:18:03.275006+0000","last_became_peered":"2024-04-25T01:18:03.275006+0000","last_unstale":"2024-04-25T01:18:19.412163+0000","last_undegraded":"2024-04-25T01:18:19.412163+0000","last_fullsized":"2024-04-25T01:18:19.412163+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:48:53.313464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"43'2","reported_seq":31,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.397019+0000","last_change":"2024-04-25T01:18:02.259736+0000","last_active":"2024-04-25T01:18:19.397019+0000","last_peered":"2024-04-25T01:18:19.397019+0000","last_clean":"2024-04-25T01:18:19.397019+0000","last_became_active":"2024-04-25T01:18:02.259127+0000","last_became_peered":"2024-04-25T01:18:02.259127+0000","last_unstale":"2024-04-25T01:18:19.397019+0000","last_undegraded":"2024-04-25T01:18:19.397019+0000","last_fullsized":"2024-04-25T01:18:19.397019+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:58:04.484125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.407277+0000","last_change":"2024-04-25T01:18:19.406596+0000","last_active":"2024-04-25T01:18:19.407277+0000","last_peered":"2024-04-25T01:18:19.407277+0000","last_clean":"2024-04-25T01:18:19.407277+0000","last_became_active":"2024-04-25T01:18:19.406280+0000","last_became_peered":"2024-04-25T01:18:19.406280+0000","last_unstale":"2024-04-25T01:18:19.407277+0000","last_undegraded":"2024-04-25T01:18:19.407277+0000","last_fullsized":"2024-04-25T01:18:19.407277+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:01:14.030421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"42'7","reported_seq":35,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.396232+0000","last_change":"2024-04-25T01:18:03.270127+0000","last_active":"2024-04-25T01:18:19.396232+0000","last_peered":"2024-04-25T01:18:19.396232+0000","last_clean":"2024-04-25T01:18:19.396232+0000","last_became_active":"2024-04-25T01:18:03.267471+0000","last_became_peered":"2024-04-25T01:18:03.267471+0000","last_unstale":"2024-04-25T01:18:19.396232+0000","last_undegraded":"2024-04-25T01:18:19.396232+0000","last_fullsized":"2024-04-25T01:18:19.396232+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:50:36.431482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"43'3","reported_seq":34,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.407279+0000","last_change":"2024-04-25T01:18:02.267703+0000","last_active":"2024-04-25T01:18:19.407279+0000","last_peered":"2024-04-25T01:18:19.407279+0000","last_clean":"2024-04-25T01:18:19.407279+0000","last_became_active":"2024-04-25T01:18:02.267415+0000","last_became_peered":"2024-04-25T01:18:02.267415+0000","last_unstale":"2024-04-25T01:18:19.407279+0000","last_undegraded":"2024-04-25T01:18:19.407279+0000","last_fullsized":"2024-04-25T01:18:19.407279+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:09:47.374684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.431182+0000","last_change":"2024-04-25T01:18:19.429464+0000","last_active":"2024-04-25T01:18:19.431182+0000","last_peered":"2024-04-25T01:18:19.431182+0000","last_clean":"2024-04-25T01:18:19.431182+0000","last_became_active":"2024-04-25T01:18:19.429156+0000","last_became_peered":"2024-04-25T01:18:19.429156+0000","last_unstale":"2024-04-25T01:18:19.431182+0000","last_undegraded":"2024-04-25T01:18:19.431182+0000","last_fullsized":"2024-04-25T01:18:19.431182+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:07.863347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"42'5","reported_seq":33,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.396376+0000","last_change":"2024-04-25T01:18:03.266233+0000","last_active":"2024-04-25T01:18:19.396376+0000","last_peered":"2024-04-25T01:18:19.396376+0000","last_clean":"2024-04-25T01:18:19.396376+0000","last_became_active":"2024-04-25T01:18:03.265929+0000","last_became_peered":"2024-04-25T01:18:03.265929+0000","last_unstale":"2024-04-25T01:18:19.396376+0000","last_undegraded":"2024-04-25T01:18:19.396376+0000","last_fullsized":"2024-04-25T01:18:19.396376+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:30:00.416602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"43'2","reported_seq":32,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.406897+0000","last_change":"2024-04-25T01:18:02.267511+0000","last_active":"2024-04-25T01:18:19.406897+0000","last_peered":"2024-04-25T01:18:19.406897+0000","last_clean":"2024-04-25T01:18:19.406897+0000","last_became_active":"2024-04-25T01:18:02.266732+0000","last_became_peered":"2024-04-25T01:18:02.266732+0000","last_unstale":"2024-04-25T01:18:19.406897+0000","last_undegraded":"2024-04-25T01:18:19.406897+0000","last_fullsized":"2024-04-25T01:18:19.406897+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:58:04.484125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.425463+0000","last_change":"2024-04-25T01:18:19.423488+0000","last_active":"2024-04-25T01:18:19.425463+0000","last_peered":"2024-04-25T01:18:19.425463+0000","last_clean":"2024-04-25T01:18:19.425463+0000","last_became_active":"2024-04-25T01:18:19.423143+0000","last_became_peered":"2024-04-25T01:18:19.423143+0000","last_unstale":"2024-04-25T01:18:19.425463+0000","last_undegraded":"2024-04-25T01:18:19.425463+0000","last_fullsized":"2024-04-25T01:18:19.425463+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:33.361047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"42'9","reported_seq":40,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.401861+0000","last_change":"2024-04-25T01:18:03.255121+0000","last_active":"2024-04-25T01:18:19.401861+0000","last_peered":"2024-04-25T01:18:19.401861+0000","last_clean":"2024-04-25T01:18:19.401861+0000","last_became_active":"2024-04-25T01:18:03.254831+0000","last_became_peered":"2024-04-25T01:18:03.254831+0000","last_unstale":"2024-04-25T01:18:19.401861+0000","last_undegraded":"2024-04-25T01:18:19.401861+0000","last_fullsized":"2024-04-25T01:18:19.401861+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:42:01.659435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":32,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.399392+0000","last_change":"2024-04-25T01:18:02.259323+0000","last_active":"2024-04-25T01:18:19.399392+0000","last_peered":"2024-04-25T01:18:19.399392+0000","last_clean":"2024-04-25T01:18:19.399392+0000","last_became_active":"2024-04-25T01:18:02.258937+0000","last_became_peered":"2024-04-25T01:18:02.258937+0000","last_unstale":"2024-04-25T01:18:19.399392+0000","last_undegraded":"2024-04-25T01:18:19.399392+0000","last_fullsized":"2024-04-25T01:18:19.399392+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:12.725383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.430079+0000","last_change":"2024-04-25T01:18:19.426878+0000","last_active":"2024-04-25T01:18:19.430079+0000","last_peered":"2024-04-25T01:18:19.430079+0000","last_clean":"2024-04-25T01:18:19.430079+0000","last_became_active":"2024-04-25T01:18:19.426297+0000","last_became_peered":"2024-04-25T01:18:19.426297+0000","last_unstale":"2024-04-25T01:18:19.430079+0000","last_undegraded":"2024-04-25T01:18:19.430079+0000","last_fullsized":"2024-04-25T01:18:19.430079+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:47:19.114215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"42'9","reported_seq":39,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.409925+0000","last_change":"2024-04-25T01:18:03.253199+0000","last_active":"2024-04-25T01:18:19.409925+0000","last_peered":"2024-04-25T01:18:19.409925+0000","last_clean":"2024-04-25T01:18:19.409925+0000","last_became_active":"2024-04-25T01:18:03.252912+0000","last_became_peered":"2024-04-25T01:18:03.252912+0000","last_unstale":"2024-04-25T01:18:19.409925+0000","last_undegraded":"2024-04-25T01:18:19.409925+0000","last_fullsized":"2024-04-25T01:18:19.409925+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:26:28.066399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"43'4","reported_seq":36,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.399834+0000","last_change":"2024-04-25T01:18:02.267229+0000","last_active":"2024-04-25T01:18:19.399834+0000","last_peered":"2024-04-25T01:18:19.399834+0000","last_clean":"2024-04-25T01:18:19.399834+0000","last_became_active":"2024-04-25T01:18:02.266605+0000","last_became_peered":"2024-04-25T01:18:02.266605+0000","last_unstale":"2024-04-25T01:18:19.399834+0000","last_undegraded":"2024-04-25T01:18:19.399834+0000","last_fullsized":"2024-04-25T01:18:19.399834+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:34:33.983053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.423610+0000","last_change":"2024-04-25T01:18:19.405615+0000","last_active":"2024-04-25T01:18:19.423610+0000","last_peered":"2024-04-25T01:18:19.423610+0000","last_clean":"2024-04-25T01:18:19.423610+0000","last_became_active":"2024-04-25T01:18:19.404531+0000","last_became_peered":"2024-04-25T01:18:19.404531+0000","last_unstale":"2024-04-25T01:18:19.423610+0000","last_undegraded":"2024-04-25T01:18:19.423610+0000","last_fullsized":"2024-04-25T01:18:19.423610+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:49:09.535800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"42'2","reported_seq":31,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.421200+0000","last_change":"2024-04-25T01:18:03.274242+0000","last_active":"2024-04-25T01:18:19.421200+0000","last_peered":"2024-04-25T01:18:19.421200+0000","last_clean":"2024-04-25T01:18:19.421200+0000","last_became_active":"2024-04-25T01:18:03.273857+0000","last_became_peered":"2024-04-25T01:18:03.273857+0000","last_unstale":"2024-04-25T01:18:19.421200+0000","last_undegraded":"2024-04-25T01:18:19.421200+0000","last_fullsized":"2024-04-25T01:18:19.421200+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:39:30.588445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":30,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.412087+0000","last_change":"2024-04-25T01:18:02.259159+0000","last_active":"2024-04-25T01:18:19.412087+0000","last_peered":"2024-04-25T01:18:19.412087+0000","last_clean":"2024-04-25T01:18:19.412087+0000","last_became_active":"2024-04-25T01:18:02.257389+0000","last_became_peered":"2024-04-25T01:18:02.257389+0000","last_unstale":"2024-04-25T01:18:19.412087+0000","last_undegraded":"2024-04-25T01:18:19.412087+0000","last_fullsized":"2024-04-25T01:18:19.412087+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:43.394014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.423600+0000","last_change":"2024-04-25T01:18:19.420297+0000","last_active":"2024-04-25T01:18:19.423600+0000","last_peered":"2024-04-25T01:18:19.423600+0000","last_clean":"2024-04-25T01:18:19.423600+0000","last_became_active":"2024-04-25T01:18:19.420011+0000","last_became_peered":"2024-04-25T01:18:19.420011+0000","last_unstale":"2024-04-25T01:18:19.423600+0000","last_undegraded":"2024-04-25T01:18:19.423600+0000","last_fullsized":"2024-04-25T01:18:19.423600+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:50:51.812840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"42'5","reported_seq":33,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.406210+0000","last_change":"2024-04-25T01:18:03.274693+0000","last_active":"2024-04-25T01:18:19.406210+0000","last_peered":"2024-04-25T01:18:19.406210+0000","last_clean":"2024-04-25T01:18:19.406210+0000","last_became_active":"2024-04-25T01:18:03.274419+0000","last_became_peered":"2024-04-25T01:18:03.274419+0000","last_unstale":"2024-04-25T01:18:19.406210+0000","last_undegraded":"2024-04-25T01:18:19.406210+0000","last_fullsized":"2024-04-25T01:18:19.406210+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:03:36.196096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":29,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.394177+0000","last_change":"2024-04-25T01:18:02.254033+0000","last_active":"2024-04-25T01:18:19.394177+0000","last_peered":"2024-04-25T01:18:19.394177+0000","last_clean":"2024-04-25T01:18:19.394177+0000","last_became_active":"2024-04-25T01:18:02.253731+0000","last_became_peered":"2024-04-25T01:18:02.253731+0000","last_unstale":"2024-04-25T01:18:19.394177+0000","last_undegraded":"2024-04-25T01:18:19.394177+0000","last_fullsized":"2024-04-25T01:18:19.394177+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:00:15.697767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.428595+0000","last_change":"2024-04-25T01:18:19.426316+0000","last_active":"2024-04-25T01:18:19.428595+0000","last_peered":"2024-04-25T01:18:19.428595+0000","last_clean":"2024-04-25T01:18:19.428595+0000","last_became_active":"2024-04-25T01:18:19.426037+0000","last_became_peered":"2024-04-25T01:18:19.426037+0000","last_unstale":"2024-04-25T01:18:19.428595+0000","last_undegraded":"2024-04-25T01:18:19.428595+0000","last_fullsized":"2024-04-25T01:18:19.428595+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:27.121340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"42'8","reported_seq":37,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.421384+0000","last_change":"2024-04-25T01:18:03.258959+0000","last_active":"2024-04-25T01:18:19.421384+0000","last_peered":"2024-04-25T01:18:19.421384+0000","last_clean":"2024-04-25T01:18:19.421384+0000","last_became_active":"2024-04-25T01:18:03.258462+0000","last_became_peered":"2024-04-25T01:18:03.258462+0000","last_unstale":"2024-04-25T01:18:19.421384+0000","last_undegraded":"2024-04-25T01:18:19.421384+0000","last_fullsized":"2024-04-25T01:18:19.421384+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:52:20.724622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":30,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.411462+0000","last_change":"2024-04-25T01:18:02.266874+0000","last_active":"2024-04-25T01:18:19.411462+0000","last_peered":"2024-04-25T01:18:19.411462+0000","last_clean":"2024-04-25T01:18:19.411462+0000","last_became_active":"2024-04-25T01:18:02.266188+0000","last_became_peered":"2024-04-25T01:18:02.266188+0000","last_unstale":"2024-04-25T01:18:19.411462+0000","last_undegraded":"2024-04-25T01:18:19.411462+0000","last_fullsized":"2024-04-25T01:18:19.411462+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:12:59.564111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.430694+0000","last_change":"2024-04-25T01:18:19.426888+0000","last_active":"2024-04-25T01:18:19.430694+0000","last_peered":"2024-04-25T01:18:19.430694+0000","last_clean":"2024-04-25T01:18:19.430694+0000","last_became_active":"2024-04-25T01:18:19.426641+0000","last_became_peered":"2024-04-25T01:18:19.426641+0000","last_unstale":"2024-04-25T01:18:19.430694+0000","last_undegraded":"2024-04-25T01:18:19.430694+0000","last_fullsized":"2024-04-25T01:18:19.430694+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:22.653702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"42'9","reported_seq":38,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.409356+0000","last_change":"2024-04-25T01:18:03.269115+0000","last_active":"2024-04-25T01:18:19.409356+0000","last_peered":"2024-04-25T01:18:19.409356+0000","last_clean":"2024-04-25T01:18:19.409356+0000","last_became_active":"2024-04-25T01:18:03.268774+0000","last_became_peered":"2024-04-25T01:18:03.268774+0000","last_unstale":"2024-04-25T01:18:19.409356+0000","last_undegraded":"2024-04-25T01:18:19.409356+0000","last_fullsized":"2024-04-25T01:18:19.409356+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:50:14.498011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"43'3","reported_seq":34,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.412560+0000","last_change":"2024-04-25T01:18:02.257314+0000","last_active":"2024-04-25T01:18:19.412560+0000","last_peered":"2024-04-25T01:18:19.412560+0000","last_clean":"2024-04-25T01:18:19.412560+0000","last_became_active":"2024-04-25T01:18:02.256973+0000","last_became_peered":"2024-04-25T01:18:02.256973+0000","last_unstale":"2024-04-25T01:18:19.412560+0000","last_undegraded":"2024-04-25T01:18:19.412560+0000","last_fullsized":"2024-04-25T01:18:19.412560+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:06:24.081982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.429512+0000","last_change":"2024-04-25T01:18:19.427898+0000","last_active":"2024-04-25T01:18:19.429512+0000","last_peered":"2024-04-25T01:18:19.429512+0000","last_clean":"2024-04-25T01:18:19.429512+0000","last_became_active":"2024-04-25T01:18:19.427260+0000","last_became_peered":"2024-04-25T01:18:19.427260+0000","last_unstale":"2024-04-25T01:18:19.429512+0000","last_undegraded":"2024-04-25T01:18:19.429512+0000","last_fullsized":"2024-04-25T01:18:19.429512+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:33:48.493716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"42'5","reported_seq":33,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.408993+0000","last_change":"2024-04-25T01:18:03.276655+0000","last_active":"2024-04-25T01:18:19.408993+0000","last_peered":"2024-04-25T01:18:19.408993+0000","last_clean":"2024-04-25T01:18:19.408993+0000","last_became_active":"2024-04-25T01:18:03.276023+0000","last_became_peered":"2024-04-25T01:18:03.276023+0000","last_unstale":"2024-04-25T01:18:19.408993+0000","last_undegraded":"2024-04-25T01:18:19.408993+0000","last_fullsized":"2024-04-25T01:18:19.408993+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:29:13.129901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"43'4","reported_seq":34,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.421899+0000","last_change":"2024-04-25T01:18:02.265987+0000","last_active":"2024-04-25T01:18:19.421899+0000","last_peered":"2024-04-25T01:18:19.421899+0000","last_clean":"2024-04-25T01:18:19.421899+0000","last_became_active":"2024-04-25T01:18:02.265704+0000","last_became_peered":"2024-04-25T01:18:02.265704+0000","last_unstale":"2024-04-25T01:18:19.421899+0000","last_undegraded":"2024-04-25T01:18:19.421899+0000","last_fullsized":"2024-04-25T01:18:19.421899+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:03:28.616338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.425103+0000","last_change":"2024-04-25T01:18:19.419566+0000","last_active":"2024-04-25T01:18:19.425103+0000","last_peered":"2024-04-25T01:18:19.425103+0000","last_clean":"2024-04-25T01:18:19.425103+0000","last_became_active":"2024-04-25T01:18:19.419251+0000","last_became_peered":"2024-04-25T01:18:19.419251+0000","last_unstale":"2024-04-25T01:18:19.425103+0000","last_undegraded":"2024-04-25T01:18:19.425103+0000","last_fullsized":"2024-04-25T01:18:19.425103+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:47:23.704689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"42'7","reported_seq":38,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.400751+0000","last_change":"2024-04-25T01:18:03.260610+0000","last_active":"2024-04-25T01:18:19.400751+0000","last_peered":"2024-04-25T01:18:19.400751+0000","last_clean":"2024-04-25T01:18:19.400751+0000","last_became_active":"2024-04-25T01:18:03.259487+0000","last_became_peered":"2024-04-25T01:18:03.259487+0000","last_unstale":"2024-04-25T01:18:19.400751+0000","last_undegraded":"2024-04-25T01:18:19.400751+0000","last_fullsized":"2024-04-25T01:18:19.400751+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:47.778367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":30,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.407252+0000","last_change":"2024-04-25T01:18:02.269139+0000","last_active":"2024-04-25T01:18:19.407252+0000","last_peered":"2024-04-25T01:18:19.407252+0000","last_clean":"2024-04-25T01:18:19.407252+0000","last_became_active":"2024-04-25T01:18:02.268747+0000","last_became_peered":"2024-04-25T01:18:02.268747+0000","last_unstale":"2024-04-25T01:18:19.407252+0000","last_undegraded":"2024-04-25T01:18:19.407252+0000","last_fullsized":"2024-04-25T01:18:19.407252+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:49:10.075668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.406280+0000","last_change":"2024-04-25T01:18:19.404519+0000","last_active":"2024-04-25T01:18:19.406280+0000","last_peered":"2024-04-25T01:18:19.406280+0000","last_clean":"2024-04-25T01:18:19.406280+0000","last_became_active":"2024-04-25T01:18:19.404144+0000","last_became_peered":"2024-04-25T01:18:19.404144+0000","last_unstale":"2024-04-25T01:18:19.406280+0000","last_undegraded":"2024-04-25T01:18:19.406280+0000","last_fullsized":"2024-04-25T01:18:19.406280+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:56.306811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"42'8","reported_seq":37,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.407134+0000","last_change":"2024-04-25T01:18:03.254793+0000","last_active":"2024-04-25T01:18:19.407134+0000","last_peered":"2024-04-25T01:18:19.407134+0000","last_clean":"2024-04-25T01:18:19.407134+0000","last_became_active":"2024-04-25T01:18:03.253467+0000","last_became_peered":"2024-04-25T01:18:03.253467+0000","last_unstale":"2024-04-25T01:18:19.407134+0000","last_undegraded":"2024-04-25T01:18:19.407134+0000","last_fullsized":"2024-04-25T01:18:19.407134+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:02:29.348596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":30,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.422640+0000","last_change":"2024-04-25T01:18:02.269410+0000","last_active":"2024-04-25T01:18:19.422640+0000","last_peered":"2024-04-25T01:18:19.422640+0000","last_clean":"2024-04-25T01:18:19.422640+0000","last_became_active":"2024-04-25T01:18:02.268924+0000","last_became_peered":"2024-04-25T01:18:02.268924+0000","last_unstale":"2024-04-25T01:18:19.422640+0000","last_undegraded":"2024-04-25T01:18:19.422640+0000","last_fullsized":"2024-04-25T01:18:19.422640+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:19.010506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.410607+0000","last_change":"2024-04-25T01:18:19.408014+0000","last_active":"2024-04-25T01:18:19.410607+0000","last_peered":"2024-04-25T01:18:19.410607+0000","last_clean":"2024-04-25T01:18:19.410607+0000","last_became_active":"2024-04-25T01:18:19.405706+0000","last_became_peered":"2024-04-25T01:18:19.405706+0000","last_unstale":"2024-04-25T01:18:19.410607+0000","last_undegraded":"2024-04-25T01:18:19.410607+0000","last_fullsized":"2024-04-25T01:18:19.410607+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:06:50.650712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"42'5","reported_seq":34,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.415722+0000","last_change":"2024-04-25T01:18:03.261409+0000","last_active":"2024-04-25T01:18:19.415722+0000","last_peered":"2024-04-25T01:18:19.415722+0000","last_clean":"2024-04-25T01:18:19.415722+0000","last_became_active":"2024-04-25T01:18:03.261126+0000","last_became_peered":"2024-04-25T01:18:03.261126+0000","last_unstale":"2024-04-25T01:18:19.415722+0000","last_undegraded":"2024-04-25T01:18:19.415722+0000","last_fullsized":"2024-04-25T01:18:19.415722+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:49:57.662679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":29,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.406913+0000","last_change":"2024-04-25T01:18:02.267491+0000","last_active":"2024-04-25T01:18:19.406913+0000","last_peered":"2024-04-25T01:18:19.406913+0000","last_clean":"2024-04-25T01:18:19.406913+0000","last_became_active":"2024-04-25T01:18:02.267044+0000","last_became_peered":"2024-04-25T01:18:02.267044+0000","last_unstale":"2024-04-25T01:18:19.406913+0000","last_undegraded":"2024-04-25T01:18:19.406913+0000","last_fullsized":"2024-04-25T01:18:19.406913+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:30:07.096033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.423467+0000","last_change":"2024-04-25T01:18:19.408627+0000","last_active":"2024-04-25T01:18:19.423467+0000","last_peered":"2024-04-25T01:18:19.423467+0000","last_clean":"2024-04-25T01:18:19.423467+0000","last_became_active":"2024-04-25T01:18:19.407856+0000","last_became_peered":"2024-04-25T01:18:19.407856+0000","last_unstale":"2024-04-25T01:18:19.423467+0000","last_undegraded":"2024-04-25T01:18:19.423467+0000","last_fullsized":"2024-04-25T01:18:19.423467+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:23:29.707982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2","version":"42'8","reported_seq":38,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.410409+0000","last_change":"2024-04-25T01:18:03.254489+0000","last_active":"2024-04-25T01:18:19.410409+0000","last_peered":"2024-04-25T01:18:19.410409+0000","last_clean":"2024-04-25T01:18:19.410409+0000","last_became_active":"2024-04-25T01:18:03.254164+0000","last_became_peered":"2024-04-25T01:18:03.254164+0000","last_unstale":"2024-04-25T01:18:19.410409+0000","last_undegraded":"2024-04-25T01:18:19.410409+0000","last_fullsized":"2024-04-25T01:18:19.410409+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:31:29.760406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":31,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.409273+0000","last_change":"2024-04-25T01:18:02.263853+0000","last_active":"2024-04-25T01:18:19.409273+0000","last_peered":"2024-04-25T01:18:19.409273+0000","last_clean":"2024-04-25T01:18:19.409273+0000","last_became_active":"2024-04-25T01:18:02.263501+0000","last_became_peered":"2024-04-25T01:18:02.263501+0000","last_unstale":"2024-04-25T01:18:19.409273+0000","last_undegraded":"2024-04-25T01:18:19.409273+0000","last_fullsized":"2024-04-25T01:18:19.409273+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:06:33.412784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.428367+0000","last_change":"2024-04-25T01:18:19.425151+0000","last_active":"2024-04-25T01:18:19.428367+0000","last_peered":"2024-04-25T01:18:19.428367+0000","last_clean":"2024-04-25T01:18:19.428367+0000","last_became_active":"2024-04-25T01:18:19.424002+0000","last_became_peered":"2024-04-25T01:18:19.424002+0000","last_unstale":"2024-04-25T01:18:19.428367+0000","last_undegraded":"2024-04-25T01:18:19.428367+0000","last_fullsized":"2024-04-25T01:18:19.428367+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:47:50.829942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"42'7","reported_seq":35,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.404294+0000","last_change":"2024-04-25T01:18:03.256275+0000","last_active":"2024-04-25T01:18:19.404294+0000","last_peered":"2024-04-25T01:18:19.404294+0000","last_clean":"2024-04-25T01:18:19.404294+0000","last_became_active":"2024-04-25T01:18:03.255950+0000","last_became_peered":"2024-04-25T01:18:03.255950+0000","last_unstale":"2024-04-25T01:18:19.404294+0000","last_undegraded":"2024-04-25T01:18:19.404294+0000","last_fullsized":"2024-04-25T01:18:19.404294+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:01:09.052002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":30,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.422082+0000","last_change":"2024-04-25T01:18:02.269077+0000","last_active":"2024-04-25T01:18:19.422082+0000","last_peered":"2024-04-25T01:18:19.422082+0000","last_clean":"2024-04-25T01:18:19.422082+0000","last_became_active":"2024-04-25T01:18:02.268795+0000","last_became_peered":"2024-04-25T01:18:02.268795+0000","last_unstale":"2024-04-25T01:18:19.422082+0000","last_undegraded":"2024-04-25T01:18:19.422082+0000","last_fullsized":"2024-04-25T01:18:19.422082+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:16.123119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.423340+0000","last_change":"2024-04-25T01:18:19.410158+0000","last_active":"2024-04-25T01:18:19.423340+0000","last_peered":"2024-04-25T01:18:19.423340+0000","last_clean":"2024-04-25T01:18:19.423340+0000","last_became_active":"2024-04-25T01:18:19.409353+0000","last_became_peered":"2024-04-25T01:18:19.409353+0000","last_unstale":"2024-04-25T01:18:19.423340+0000","last_undegraded":"2024-04-25T01:18:19.423340+0000","last_fullsized":"2024-04-25T01:18:19.423340+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:31.688068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"42'8","reported_seq":38,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.411955+0000","last_change":"2024-04-25T01:18:03.253285+0000","last_active":"2024-04-25T01:18:19.411955+0000","last_peered":"2024-04-25T01:18:19.411955+0000","last_clean":"2024-04-25T01:18:19.411955+0000","last_became_active":"2024-04-25T01:18:03.252973+0000","last_became_peered":"2024-04-25T01:18:03.252973+0000","last_unstale":"2024-04-25T01:18:19.411955+0000","last_undegraded":"2024-04-25T01:18:19.411955+0000","last_fullsized":"2024-04-25T01:18:19.411955+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:58:23.459413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":29,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.404431+0000","last_change":"2024-04-25T01:18:02.267166+0000","last_active":"2024-04-25T01:18:19.404431+0000","last_peered":"2024-04-25T01:18:19.404431+0000","last_clean":"2024-04-25T01:18:19.404431+0000","last_became_active":"2024-04-25T01:18:02.266866+0000","last_became_peered":"2024-04-25T01:18:02.266866+0000","last_unstale":"2024-04-25T01:18:19.404431+0000","last_undegraded":"2024-04-25T01:18:19.404431+0000","last_fullsized":"2024-04-25T01:18:19.404431+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:20:28.341160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":220,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.394026+0000","last_change":"2024-04-25T01:16:17.466569+0000","last_active":"2024-04-25T01:18:19.394026+0000","last_peered":"2024-04-25T01:18:19.394026+0000","last_clean":"2024-04-25T01:18:19.394026+0000","last_became_active":"2024-04-25T01:16:17.465687+0000","last_became_peered":"2024-04-25T01:16:17.465687+0000","last_unstale":"2024-04-25T01:18:19.394026+0000","last_undegraded":"2024-04-25T01:18:19.394026+0000","last_fullsized":"2024-04-25T01:18:19.394026+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:16.440559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:16.440559+0000","last_clean_scrub_stamp":"2024-04-25T01:16:16.440559+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:18:17.493555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":6291456,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12582912,"data_stored":12582912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":504,"ondisk_log_size":504,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":96,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542172,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":257136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542171,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":557056,"data_stored":249362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542172,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36416,"kb_used_data":5232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335424,"statfs":{"total":96636764160,"available":96599474176,"internally_reserved":0,"allocated":5357568,"data_stored":5041808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542171,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":557056,"data_stored":249362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542171,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33720,"kb_used_data":6700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338120,"statfs":{"total":96636764160,"available":96602234880,"internally_reserved":0,"allocated":6860800,"data_stored":6548592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542173,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":257136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542171,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28148,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343692,"statfs":{"total":96636764160,"available":96607940608,"internally_reserved":0,"allocated":1150976,"data_stored":839730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542172,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29620,"kb_used_data":2592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342220,"statfs":{"total":96636764160,"available":96606433280,"internally_reserved":0,"allocated":2654208,"data_stored":2346514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2097152,"data_stored":2097152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6291456,"data_stored":6291456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:18:23.844 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:18:24.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.102+0000 7fcce7000640 1 Processor -- start 2024-04-25T01:18:24.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.102+0000 7fcce7000640 1 -- start start 2024-04-25T01:18:24.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.103+0000 7fcce7000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcce015a980 0x7fcce006d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:24.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.103+0000 7fcce7000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce0071dd0 0x7fcce00cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:24.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.104+0000 7fcce7000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcce00ccb10 con 0x7fcce00d0070 2024-04-25T01:18:24.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.104+0000 7fcce7000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fcce0072230 con 0x7fcce015a980 2024-04-25T01:18:24.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.104+0000 7fcce7000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcce00aede0 con 0x7fcce0071dd0 2024-04-25T01:18:24.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.104+0000 7fcce5c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcce015a980 0x7fcce006d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:24.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.104+0000 7fcce5c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcce015a980 0x7fcce006d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41166/0 (socket says 172.21.15.90:41166) 2024-04-25T01:18:24.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.104+0000 7fcce5c00640 1 -- 172.21.15.90:0/3377571180 learned_addr learned my addr 172.21.15.90:0/3377571180 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:24.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.104+0000 7fcce5200640 1 --2- 172.21.15.90:0/3377571180 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce0071dd0 0x7fcce00cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:24.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.104+0000 7fcce5c00640 1 -- 172.21.15.90:0/3377571180 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce0071dd0 msgr2=0x7fcce00cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:24.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.104+0000 7fcce5c00640 1 --2- 172.21.15.90:0/3377571180 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce0071dd0 0x7fcce00cfb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:24.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.104+0000 7fcce5c00640 1 -- 172.21.15.90:0/3377571180 >> v1:172.21.15.103:6789/0 conn(0x7fcce00d0070 legacy=0x7fcce00b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:24.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.104+0000 7fcce5c00640 1 -- 172.21.15.90:0/3377571180 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcce006ddc0 con 0x7fcce015a980 2024-04-25T01:18:24.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.104+0000 7fcce5200640 1 --2- 172.21.15.90:0/3377571180 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce0071dd0 0x7fcce00cfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:24.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.105+0000 7fcce5c00640 1 --2- 172.21.15.90:0/3377571180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcce015a980 0x7fcce006d770 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fccd4009870 tx=0x7fccd402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d46ddd7aa01b1f4c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:24.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.106+0000 7fccdfe00640 1 -- 172.21.15.90:0/3377571180 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccd4037070 con 0x7fcce015a980 2024-04-25T01:18:24.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.106+0000 7fccdfe00640 1 -- 172.21.15.90:0/3377571180 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccd4032440 con 0x7fcce015a980 2024-04-25T01:18:24.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.106+0000 7fccdfe00640 1 -- 172.21.15.90:0/3377571180 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccd4032810 con 0x7fcce015a980 2024-04-25T01:18:24.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.106+0000 7fcce7000640 1 -- 172.21.15.90:0/3377571180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcce015a980 msgr2=0x7fcce006d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:24.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.106+0000 7fcce7000640 1 --2- 172.21.15.90:0/3377571180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcce015a980 0x7fcce006d770 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fccd4009870 tx=0x7fccd402a180 comp rx=0 tx=0).stop 2024-04-25T01:18:24.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.106+0000 7fcce7000640 1 -- 172.21.15.90:0/3377571180 shutdown_connections 2024-04-25T01:18:24.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.106+0000 7fcce7000640 1 --2- 172.21.15.90:0/3377571180 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce0071dd0 0x7fcce00cfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:24.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.106+0000 7fcce7000640 1 --2- 172.21.15.90:0/3377571180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcce015a980 0x7fcce006d770 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:24.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.106+0000 7fcce7000640 1 -- 172.21.15.90:0/3377571180 >> 172.21.15.90:0/3377571180 conn(0x7fcce006b320 msgr2=0x7fcce006b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:24.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.107+0000 7fcce7000640 1 -- 172.21.15.90:0/3377571180 shutdown_connections 2024-04-25T01:18:24.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.107+0000 7fcce7000640 1 -- 172.21.15.90:0/3377571180 wait complete. 2024-04-25T01:18:24.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.108+0000 7fcce7000640 1 Processor -- start 2024-04-25T01:18:24.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.108+0000 7fcce7000640 1 -- start start 2024-04-25T01:18:24.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.109+0000 7fcce7000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce0071dd0 0x7fcce01e7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:24.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.109+0000 7fcce7000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcce00d0070 0x7fcce0144e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:24.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.109+0000 7fcce7000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcce015a980 0x7fcce01453a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:24.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.109+0000 7fcce7000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcce006e660 con 0x7fcce00d0070 2024-04-25T01:18:24.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.109+0000 7fcce7000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcce006e380 con 0x7fcce015a980 2024-04-25T01:18:24.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.109+0000 7fcce7000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcce015ad60 con 0x7fcce0071dd0 2024-04-25T01:18:24.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.110+0000 7fcce5200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcce00d0070 0x7fcce0144e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:24.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.110+0000 7fcce5200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcce00d0070 0x7fcce0144e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41182/0 (socket says 172.21.15.90:41182) 2024-04-25T01:18:24.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.110+0000 7fcce5200640 1 -- 172.21.15.90:0/1008077459 learned_addr learned my addr 172.21.15.90:0/1008077459 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:24.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.110+0000 7fcce5c00640 1 --2- 172.21.15.90:0/1008077459 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce0071dd0 0x7fcce01e7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:24.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.110+0000 7fcce5200640 1 -- 172.21.15.90:0/1008077459 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce0071dd0 msgr2=0x7fcce01e7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:24.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.110+0000 7fcce6600640 1 --2- 172.21.15.90:0/1008077459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcce015a980 0x7fcce01453a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:24.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.110+0000 7fcce5200640 1 --2- 172.21.15.90:0/1008077459 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce0071dd0 0x7fcce01e7670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:24.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.110+0000 7fcce5200640 1 -- 172.21.15.90:0/1008077459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcce015a980 msgr2=0x7fcce01453a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:24.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.110+0000 7fcce5200640 1 --2- 172.21.15.90:0/1008077459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcce015a980 0x7fcce01453a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:24.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.110+0000 7fcce5200640 1 -- 172.21.15.90:0/1008077459 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcce01f6f50 con 0x7fcce00d0070 2024-04-25T01:18:24.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.110+0000 7fcce6600640 1 --2- 172.21.15.90:0/1008077459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcce015a980 0x7fcce01453a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:24.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.110+0000 7fcce5200640 1 --2- 172.21.15.90:0/1008077459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcce00d0070 0x7fcce0144e60 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fcccc00c820 tx=0x7fcccc00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:24.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.111+0000 7fccdd600640 1 -- 172.21.15.90:0/1008077459 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcccc015070 con 0x7fcce00d0070 2024-04-25T01:18:24.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.111+0000 7fccdd600640 1 -- 172.21.15.90:0/1008077459 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcccc007a30 con 0x7fcce00d0070 2024-04-25T01:18:24.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.111+0000 7fccdd600640 1 -- 172.21.15.90:0/1008077459 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcccc007e00 con 0x7fcce00d0070 2024-04-25T01:18:24.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.111+0000 7fcce7000640 1 -- 172.21.15.90:0/1008077459 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcce01f7230 con 0x7fcce00d0070 2024-04-25T01:18:24.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.111+0000 7fcce7000640 1 -- 172.21.15.90:0/1008077459 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcce01e7bb0 con 0x7fcce00d0070 2024-04-25T01:18:24.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.112+0000 7fcce7000640 1 -- 172.21.15.90:0/1008077459 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcce004f500 con 0x7fcce00d0070 2024-04-25T01:18:24.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.113+0000 7fccdd600640 1 -- 172.21.15.90:0/1008077459 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fcccc018070 con 0x7fcce00d0070 2024-04-25T01:18:24.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.114+0000 7fccdd600640 1 --2- 172.21.15.90:0/1008077459 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fccc006be30 0x7fccc006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:24.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.114+0000 7fccdd600640 1 -- 172.21.15.90:0/1008077459 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7fcccc08b360 con 0x7fcce00d0070 2024-04-25T01:18:24.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.114+0000 7fcce5c00640 1 --2- 172.21.15.90:0/1008077459 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fccc006be30 0x7fccc006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:24.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.115+0000 7fcce5c00640 1 --2- 172.21.15.90:0/1008077459 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fccc006be30 0x7fccc006e2f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fccd4034000 tx=0x7fccd402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:24.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.119+0000 7fccdd600640 1 -- 172.21.15.90:0/1008077459 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fcccc059800 con 0x7fcce00d0070 2024-04-25T01:18:24.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.317+0000 7fcce7000640 1 -- 172.21.15.90:0/1008077459 --> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcce01458e0 con 0x7fccc006be30 2024-04-25T01:18:24.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.342+0000 7fccdd600640 1 -- 172.21.15.90:0/1008077459 <== mgr.4104 v2:172.21.15.90:6832/2198086733 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332509 (secure 0 0 0) 0x7fcce01458e0 con 0x7fccc006be30 2024-04-25T01:18:24.331 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:24.333 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-25T01:18:24.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.349+0000 7fcce7000640 1 -- 172.21.15.90:0/1008077459 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fccc006be30 msgr2=0x7fccc006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:24.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.349+0000 7fcce7000640 1 --2- 172.21.15.90:0/1008077459 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fccc006be30 0x7fccc006e2f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fccd4034000 tx=0x7fccd402e000 comp rx=0 tx=0).stop 2024-04-25T01:18:24.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.349+0000 7fcce7000640 1 -- 172.21.15.90:0/1008077459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcce00d0070 msgr2=0x7fcce0144e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.349+0000 7fcce7000640 1 --2- 172.21.15.90:0/1008077459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcce00d0070 0x7fcce0144e60 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fcccc00c820 tx=0x7fcccc00cc20 comp rx=0 tx=0).stop 2024-04-25T01:18:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.349+0000 7fcce7000640 1 -- 172.21.15.90:0/1008077459 shutdown_connections 2024-04-25T01:18:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.349+0000 7fcce7000640 1 --2- 172.21.15.90:0/1008077459 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fccc006be30 0x7fccc006e2f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.349+0000 7fcce7000640 1 --2- 172.21.15.90:0/1008077459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcce015a980 0x7fcce01453a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.349+0000 7fcce7000640 1 --2- 172.21.15.90:0/1008077459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcce00d0070 0x7fcce0144e60 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.349+0000 7fcce7000640 1 --2- 172.21.15.90:0/1008077459 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcce0071dd0 0x7fcce01e7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.349+0000 7fcce7000640 1 -- 172.21.15.90:0/1008077459 >> 172.21.15.90:0/1008077459 conn(0x7fcce006b320 msgr2=0x7fcce00b2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.350+0000 7fcce7000640 1 -- 172.21.15.90:0/1008077459 shutdown_connections 2024-04-25T01:18:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.350+0000 7fcce7000640 1 -- 172.21.15.90:0/1008077459 wait complete. 2024-04-25T01:18:24.354 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":108,"stamp":"2024-04-25T01:18:23.863850+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6881824,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":117,"num_read_kb":100,"num_write":259,"num_write_kb":8604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":704,"ondisk_log_size":704,"up":258,"acting":258,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":214,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238196,"kb_used_data":17848,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754736524,"statfs":{"total":773094113280,"available":772850200576,"internally_reserved":0,"allocated":18276352,"data_stored":15789640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17557,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":44,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001304"},"pg_stats":[{"pgid":"6.3f","version":"42'8","reported_seq":37,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.422510+0000","last_change":"2024-04-25T01:18:03.253664+0000","last_active":"2024-04-25T01:18:19.422510+0000","last_peered":"2024-04-25T01:18:19.422510+0000","last_clean":"2024-04-25T01:18:19.422510+0000","last_became_active":"2024-04-25T01:18:03.253362+0000","last_became_peered":"2024-04-25T01:18:03.253362+0000","last_unstale":"2024-04-25T01:18:19.422510+0000","last_undegraded":"2024-04-25T01:18:19.422510+0000","last_fullsized":"2024-04-25T01:18:19.422510+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:21:24.721366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3e","version":"42'4","reported_seq":33,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.412764+0000","last_change":"2024-04-25T01:18:03.247432+0000","last_active":"2024-04-25T01:18:19.412764+0000","last_peered":"2024-04-25T01:18:19.412764+0000","last_clean":"2024-04-25T01:18:19.412764+0000","last_became_active":"2024-04-25T01:18:03.247132+0000","last_became_peered":"2024-04-25T01:18:03.247132+0000","last_unstale":"2024-04-25T01:18:19.412764+0000","last_undegraded":"2024-04-25T01:18:19.412764+0000","last_fullsized":"2024-04-25T01:18:19.412764+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:23:13.485645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3d","version":"42'7","reported_seq":36,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.422299+0000","last_change":"2024-04-25T01:18:03.267777+0000","last_active":"2024-04-25T01:18:19.422299+0000","last_peered":"2024-04-25T01:18:19.422299+0000","last_clean":"2024-04-25T01:18:19.422299+0000","last_became_active":"2024-04-25T01:18:03.267444+0000","last_became_peered":"2024-04-25T01:18:03.267444+0000","last_unstale":"2024-04-25T01:18:19.422299+0000","last_undegraded":"2024-04-25T01:18:19.422299+0000","last_fullsized":"2024-04-25T01:18:19.422299+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:06.961610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3c","version":"42'8","reported_seq":37,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.421434+0000","last_change":"2024-04-25T01:18:03.260302+0000","last_active":"2024-04-25T01:18:19.421434+0000","last_peered":"2024-04-25T01:18:19.421434+0000","last_clean":"2024-04-25T01:18:19.421434+0000","last_became_active":"2024-04-25T01:18:03.259967+0000","last_became_peered":"2024-04-25T01:18:03.259967+0000","last_unstale":"2024-04-25T01:18:19.421434+0000","last_undegraded":"2024-04-25T01:18:19.421434+0000","last_fullsized":"2024-04-25T01:18:19.421434+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:57:03.302298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3b","version":"42'4","reported_seq":33,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.407372+0000","last_change":"2024-04-25T01:18:03.257452+0000","last_active":"2024-04-25T01:18:19.407372+0000","last_peered":"2024-04-25T01:18:19.407372+0000","last_clean":"2024-04-25T01:18:19.407372+0000","last_became_active":"2024-04-25T01:18:03.256443+0000","last_became_peered":"2024-04-25T01:18:03.256443+0000","last_unstale":"2024-04-25T01:18:19.407372+0000","last_undegraded":"2024-04-25T01:18:19.407372+0000","last_fullsized":"2024-04-25T01:18:19.407372+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:48:59.966764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3a","version":"42'6","reported_seq":35,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.411910+0000","last_change":"2024-04-25T01:18:03.242630+0000","last_active":"2024-04-25T01:18:19.411910+0000","last_peered":"2024-04-25T01:18:19.411910+0000","last_clean":"2024-04-25T01:18:19.411910+0000","last_became_active":"2024-04-25T01:18:03.242276+0000","last_became_peered":"2024-04-25T01:18:03.242276+0000","last_unstale":"2024-04-25T01:18:19.411910+0000","last_undegraded":"2024-04-25T01:18:19.411910+0000","last_fullsized":"2024-04-25T01:18:19.411910+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:42:01.659435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.39","version":"42'5","reported_seq":33,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.393897+0000","last_change":"2024-04-25T01:18:03.257363+0000","last_active":"2024-04-25T01:18:19.393897+0000","last_peered":"2024-04-25T01:18:19.393897+0000","last_clean":"2024-04-25T01:18:19.393897+0000","last_became_active":"2024-04-25T01:18:03.256685+0000","last_became_peered":"2024-04-25T01:18:03.256685+0000","last_unstale":"2024-04-25T01:18:19.393897+0000","last_undegraded":"2024-04-25T01:18:19.393897+0000","last_fullsized":"2024-04-25T01:18:19.393897+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:19.882784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.38","version":"42'9","reported_seq":38,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.421336+0000","last_change":"2024-04-25T01:18:03.274858+0000","last_active":"2024-04-25T01:18:19.421336+0000","last_peered":"2024-04-25T01:18:19.421336+0000","last_clean":"2024-04-25T01:18:19.421336+0000","last_became_active":"2024-04-25T01:18:03.273775+0000","last_became_peered":"2024-04-25T01:18:03.273775+0000","last_unstale":"2024-04-25T01:18:19.421336+0000","last_undegraded":"2024-04-25T01:18:19.421336+0000","last_fullsized":"2024-04-25T01:18:19.421336+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:20:51.212419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.37","version":"42'11","reported_seq":40,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.414081+0000","last_change":"2024-04-25T01:18:03.265545+0000","last_active":"2024-04-25T01:18:19.414081+0000","last_peered":"2024-04-25T01:18:19.414081+0000","last_clean":"2024-04-25T01:18:19.414081+0000","last_became_active":"2024-04-25T01:18:03.265251+0000","last_became_peered":"2024-04-25T01:18:03.265251+0000","last_unstale":"2024-04-25T01:18:19.414081+0000","last_undegraded":"2024-04-25T01:18:19.414081+0000","last_fullsized":"2024-04-25T01:18:19.414081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:13.833482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.36","version":"42'15","reported_seq":43,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.396223+0000","last_change":"2024-04-25T01:18:03.267089+0000","last_active":"2024-04-25T01:18:19.396223+0000","last_peered":"2024-04-25T01:18:19.396223+0000","last_clean":"2024-04-25T01:18:19.396223+0000","last_became_active":"2024-04-25T01:18:03.266799+0000","last_became_peered":"2024-04-25T01:18:03.266799+0000","last_unstale":"2024-04-25T01:18:19.396223+0000","last_undegraded":"2024-04-25T01:18:19.396223+0000","last_fullsized":"2024-04-25T01:18:19.396223+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:57:10.276501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.35","version":"42'6","reported_seq":34,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.408260+0000","last_change":"2024-04-25T01:18:03.276878+0000","last_active":"2024-04-25T01:18:19.408260+0000","last_peered":"2024-04-25T01:18:19.408260+0000","last_clean":"2024-04-25T01:18:19.408260+0000","last_became_active":"2024-04-25T01:18:03.276396+0000","last_became_peered":"2024-04-25T01:18:03.276396+0000","last_unstale":"2024-04-25T01:18:19.408260+0000","last_undegraded":"2024-04-25T01:18:19.408260+0000","last_fullsized":"2024-04-25T01:18:19.408260+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:39.462581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.34","version":"42'7","reported_seq":36,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.408902+0000","last_change":"2024-04-25T01:18:03.249998+0000","last_active":"2024-04-25T01:18:19.408902+0000","last_peered":"2024-04-25T01:18:19.408902+0000","last_clean":"2024-04-25T01:18:19.408902+0000","last_became_active":"2024-04-25T01:18:03.249714+0000","last_became_peered":"2024-04-25T01:18:03.249714+0000","last_unstale":"2024-04-25T01:18:19.408902+0000","last_undegraded":"2024-04-25T01:18:19.408902+0000","last_fullsized":"2024-04-25T01:18:19.408902+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:19.385178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.33","version":"42'10","reported_seq":38,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.393975+0000","last_change":"2024-04-25T01:18:03.256824+0000","last_active":"2024-04-25T01:18:19.393975+0000","last_peered":"2024-04-25T01:18:19.393975+0000","last_clean":"2024-04-25T01:18:19.393975+0000","last_became_active":"2024-04-25T01:18:03.256477+0000","last_became_peered":"2024-04-25T01:18:03.256477+0000","last_unstale":"2024-04-25T01:18:19.393975+0000","last_undegraded":"2024-04-25T01:18:19.393975+0000","last_fullsized":"2024-04-25T01:18:19.393975+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:19.060283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.32","version":"42'8","reported_seq":37,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.398241+0000","last_change":"2024-04-25T01:18:03.265116+0000","last_active":"2024-04-25T01:18:19.398241+0000","last_peered":"2024-04-25T01:18:19.398241+0000","last_clean":"2024-04-25T01:18:19.398241+0000","last_became_active":"2024-04-25T01:18:03.264863+0000","last_became_peered":"2024-04-25T01:18:03.264863+0000","last_unstale":"2024-04-25T01:18:19.398241+0000","last_undegraded":"2024-04-25T01:18:19.398241+0000","last_fullsized":"2024-04-25T01:18:19.398241+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:35:34.926511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.31","version":"42'7","reported_seq":37,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.409005+0000","last_change":"2024-04-25T01:18:03.258504+0000","last_active":"2024-04-25T01:18:19.409005+0000","last_peered":"2024-04-25T01:18:19.409005+0000","last_clean":"2024-04-25T01:18:19.409005+0000","last_became_active":"2024-04-25T01:18:03.258242+0000","last_became_peered":"2024-04-25T01:18:03.258242+0000","last_unstale":"2024-04-25T01:18:19.409005+0000","last_undegraded":"2024-04-25T01:18:19.409005+0000","last_fullsized":"2024-04-25T01:18:19.409005+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:33:32.271379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.30","version":"42'9","reported_seq":37,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.393559+0000","last_change":"2024-04-25T01:18:03.261774+0000","last_active":"2024-04-25T01:18:19.393559+0000","last_peered":"2024-04-25T01:18:19.393559+0000","last_clean":"2024-04-25T01:18:19.393559+0000","last_became_active":"2024-04-25T01:18:03.261428+0000","last_became_peered":"2024-04-25T01:18:03.261428+0000","last_unstale":"2024-04-25T01:18:19.393559+0000","last_undegraded":"2024-04-25T01:18:19.393559+0000","last_fullsized":"2024-04-25T01:18:19.393559+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:55:31.639411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2f","version":"42'9","reported_seq":39,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.412342+0000","last_change":"2024-04-25T01:18:03.273618+0000","last_active":"2024-04-25T01:18:19.412342+0000","last_peered":"2024-04-25T01:18:19.412342+0000","last_clean":"2024-04-25T01:18:19.412342+0000","last_became_active":"2024-04-25T01:18:03.273161+0000","last_became_peered":"2024-04-25T01:18:03.273161+0000","last_unstale":"2024-04-25T01:18:19.412342+0000","last_undegraded":"2024-04-25T01:18:19.412342+0000","last_fullsized":"2024-04-25T01:18:19.412342+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:42:40.677650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2097152,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":2048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2e","version":"42'10","reported_seq":40,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.406922+0000","last_change":"2024-04-25T01:18:03.265079+0000","last_active":"2024-04-25T01:18:19.406922+0000","last_peered":"2024-04-25T01:18:19.406922+0000","last_clean":"2024-04-25T01:18:19.406922+0000","last_became_active":"2024-04-25T01:18:03.264708+0000","last_became_peered":"2024-04-25T01:18:03.264708+0000","last_unstale":"2024-04-25T01:18:19.406922+0000","last_undegraded":"2024-04-25T01:18:19.406922+0000","last_fullsized":"2024-04-25T01:18:19.406922+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:47:34.607606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2d","version":"42'7","reported_seq":38,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.400124+0000","last_change":"2024-04-25T01:18:03.257519+0000","last_active":"2024-04-25T01:18:19.400124+0000","last_peered":"2024-04-25T01:18:19.400124+0000","last_clean":"2024-04-25T01:18:19.400124+0000","last_became_active":"2024-04-25T01:18:03.257225+0000","last_became_peered":"2024-04-25T01:18:03.257225+0000","last_unstale":"2024-04-25T01:18:19.400124+0000","last_undegraded":"2024-04-25T01:18:19.400124+0000","last_fullsized":"2024-04-25T01:18:19.400124+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:12:04.693125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2c","version":"42'8","reported_seq":36,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.404274+0000","last_change":"2024-04-25T01:18:03.267873+0000","last_active":"2024-04-25T01:18:19.404274+0000","last_peered":"2024-04-25T01:18:19.404274+0000","last_clean":"2024-04-25T01:18:19.404274+0000","last_became_active":"2024-04-25T01:18:03.267601+0000","last_became_peered":"2024-04-25T01:18:03.267601+0000","last_unstale":"2024-04-25T01:18:19.404274+0000","last_undegraded":"2024-04-25T01:18:19.404274+0000","last_fullsized":"2024-04-25T01:18:19.404274+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:38:56.106596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2b","version":"42'11","reported_seq":39,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.404116+0000","last_change":"2024-04-25T01:18:03.256260+0000","last_active":"2024-04-25T01:18:19.404116+0000","last_peered":"2024-04-25T01:18:19.404116+0000","last_clean":"2024-04-25T01:18:19.404116+0000","last_became_active":"2024-04-25T01:18:03.255488+0000","last_became_peered":"2024-04-25T01:18:03.255488+0000","last_unstale":"2024-04-25T01:18:19.404116+0000","last_undegraded":"2024-04-25T01:18:19.404116+0000","last_fullsized":"2024-04-25T01:18:19.404116+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:06:25.097573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2a","version":"42'9","reported_seq":37,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.405642+0000","last_change":"2024-04-25T01:18:03.262912+0000","last_active":"2024-04-25T01:18:19.405642+0000","last_peered":"2024-04-25T01:18:19.405642+0000","last_clean":"2024-04-25T01:18:19.405642+0000","last_became_active":"2024-04-25T01:18:03.262598+0000","last_became_peered":"2024-04-25T01:18:03.262598+0000","last_unstale":"2024-04-25T01:18:19.405642+0000","last_undegraded":"2024-04-25T01:18:19.405642+0000","last_fullsized":"2024-04-25T01:18:19.405642+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:41:12.326642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.29","version":"42'8","reported_seq":38,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.409114+0000","last_change":"2024-04-25T01:18:03.259084+0000","last_active":"2024-04-25T01:18:19.409114+0000","last_peered":"2024-04-25T01:18:19.409114+0000","last_clean":"2024-04-25T01:18:19.409114+0000","last_became_active":"2024-04-25T01:18:03.258651+0000","last_became_peered":"2024-04-25T01:18:03.258651+0000","last_unstale":"2024-04-25T01:18:19.409114+0000","last_undegraded":"2024-04-25T01:18:19.409114+0000","last_fullsized":"2024-04-25T01:18:19.409114+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:08:28.501544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.28","version":"42'6","reported_seq":35,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.421564+0000","last_change":"2024-04-25T01:18:03.270002+0000","last_active":"2024-04-25T01:18:19.421564+0000","last_peered":"2024-04-25T01:18:19.421564+0000","last_clean":"2024-04-25T01:18:19.421564+0000","last_became_active":"2024-04-25T01:18:03.269711+0000","last_became_peered":"2024-04-25T01:18:03.269711+0000","last_unstale":"2024-04-25T01:18:19.421564+0000","last_undegraded":"2024-04-25T01:18:19.421564+0000","last_fullsized":"2024-04-25T01:18:19.421564+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:20:54.301682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.27","version":"42'2","reported_seq":31,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.399760+0000","last_change":"2024-04-25T01:18:03.265266+0000","last_active":"2024-04-25T01:18:19.399760+0000","last_peered":"2024-04-25T01:18:19.399760+0000","last_clean":"2024-04-25T01:18:19.399760+0000","last_became_active":"2024-04-25T01:18:03.264973+0000","last_became_peered":"2024-04-25T01:18:03.264973+0000","last_unstale":"2024-04-25T01:18:19.399760+0000","last_undegraded":"2024-04-25T01:18:19.399760+0000","last_fullsized":"2024-04-25T01:18:19.399760+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:30:00.416602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.26","version":"42'15","reported_seq":44,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.421901+0000","last_change":"2024-04-25T01:18:03.263008+0000","last_active":"2024-04-25T01:18:19.421901+0000","last_peered":"2024-04-25T01:18:19.421901+0000","last_clean":"2024-04-25T01:18:19.421901+0000","last_became_active":"2024-04-25T01:18:03.262081+0000","last_became_peered":"2024-04-25T01:18:03.262081+0000","last_unstale":"2024-04-25T01:18:19.421901+0000","last_undegraded":"2024-04-25T01:18:19.421901+0000","last_fullsized":"2024-04-25T01:18:19.421901+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:32:52.505647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.25","version":"42'6","reported_seq":34,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.408617+0000","last_change":"2024-04-25T01:18:03.263526+0000","last_active":"2024-04-25T01:18:19.408617+0000","last_peered":"2024-04-25T01:18:19.408617+0000","last_clean":"2024-04-25T01:18:19.408617+0000","last_became_active":"2024-04-25T01:18:03.258646+0000","last_became_peered":"2024-04-25T01:18:03.258646+0000","last_unstale":"2024-04-25T01:18:19.408617+0000","last_undegraded":"2024-04-25T01:18:19.408617+0000","last_fullsized":"2024-04-25T01:18:19.408617+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:34:33.950222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.24","version":"42'4","reported_seq":32,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.394633+0000","last_change":"2024-04-25T01:18:03.267937+0000","last_active":"2024-04-25T01:18:19.394633+0000","last_peered":"2024-04-25T01:18:19.394633+0000","last_clean":"2024-04-25T01:18:19.394633+0000","last_became_active":"2024-04-25T01:18:03.266925+0000","last_became_peered":"2024-04-25T01:18:03.266925+0000","last_unstale":"2024-04-25T01:18:19.394633+0000","last_undegraded":"2024-04-25T01:18:19.394633+0000","last_fullsized":"2024-04-25T01:18:19.394633+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:18:05.301920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.23","version":"42'7","reported_seq":37,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.409268+0000","last_change":"2024-04-25T01:18:03.253210+0000","last_active":"2024-04-25T01:18:19.409268+0000","last_peered":"2024-04-25T01:18:19.409268+0000","last_clean":"2024-04-25T01:18:19.409268+0000","last_became_active":"2024-04-25T01:18:03.252930+0000","last_became_peered":"2024-04-25T01:18:03.252930+0000","last_unstale":"2024-04-25T01:18:19.409268+0000","last_undegraded":"2024-04-25T01:18:19.409268+0000","last_fullsized":"2024-04-25T01:18:19.409268+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:56:17.738368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.22","version":"42'8","reported_seq":38,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.410148+0000","last_change":"2024-04-25T01:18:03.255049+0000","last_active":"2024-04-25T01:18:19.410148+0000","last_peered":"2024-04-25T01:18:19.410148+0000","last_clean":"2024-04-25T01:18:19.410148+0000","last_became_active":"2024-04-25T01:18:03.254795+0000","last_became_peered":"2024-04-25T01:18:03.254795+0000","last_unstale":"2024-04-25T01:18:19.410148+0000","last_undegraded":"2024-04-25T01:18:19.410148+0000","last_fullsized":"2024-04-25T01:18:19.410148+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:31:38.615550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.21","version":"42'12","reported_seq":40,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.394480+0000","last_change":"2024-04-25T01:18:03.257602+0000","last_active":"2024-04-25T01:18:19.394480+0000","last_peered":"2024-04-25T01:18:19.394480+0000","last_clean":"2024-04-25T01:18:19.394480+0000","last_became_active":"2024-04-25T01:18:03.257303+0000","last_became_peered":"2024-04-25T01:18:03.257303+0000","last_unstale":"2024-04-25T01:18:19.394480+0000","last_undegraded":"2024-04-25T01:18:19.394480+0000","last_fullsized":"2024-04-25T01:18:19.394480+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:35:34.926511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.20","version":"42'9","reported_seq":32,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.399253+0000","last_change":"2024-04-25T01:18:10.498531+0000","last_active":"2024-04-25T01:18:19.399253+0000","last_peered":"2024-04-25T01:18:19.399253+0000","last_clean":"2024-04-25T01:18:19.399253+0000","last_became_active":"2024-04-25T01:18:10.497858+0000","last_became_peered":"2024-04-25T01:18:10.497858+0000","last_unstale":"2024-04-25T01:18:19.399253+0000","last_undegraded":"2024-04-25T01:18:19.399253+0000","last_fullsized":"2024-04-25T01:18:19.399253+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:50:36.431482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.425409+0000","last_change":"2024-04-25T01:18:19.416518+0000","last_active":"2024-04-25T01:18:19.425409+0000","last_peered":"2024-04-25T01:18:19.425409+0000","last_clean":"2024-04-25T01:18:19.425409+0000","last_became_active":"2024-04-25T01:18:19.415975+0000","last_became_peered":"2024-04-25T01:18:19.415975+0000","last_unstale":"2024-04-25T01:18:19.425409+0000","last_undegraded":"2024-04-25T01:18:19.425409+0000","last_fullsized":"2024-04-25T01:18:19.425409+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:55:53.445775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"42'9","reported_seq":39,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.411290+0000","last_change":"2024-04-25T01:18:03.264199+0000","last_active":"2024-04-25T01:18:19.411290+0000","last_peered":"2024-04-25T01:18:19.411290+0000","last_clean":"2024-04-25T01:18:19.411290+0000","last_became_active":"2024-04-25T01:18:03.260827+0000","last_became_peered":"2024-04-25T01:18:03.260827+0000","last_unstale":"2024-04-25T01:18:19.411290+0000","last_undegraded":"2024-04-25T01:18:19.411290+0000","last_fullsized":"2024-04-25T01:18:19.411290+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:39:15.865514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"43'8","reported_seq":38,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.400042+0000","last_change":"2024-04-25T01:18:02.263556+0000","last_active":"2024-04-25T01:18:19.400042+0000","last_peered":"2024-04-25T01:18:19.400042+0000","last_clean":"2024-04-25T01:18:19.400042+0000","last_became_active":"2024-04-25T01:18:02.263123+0000","last_became_peered":"2024-04-25T01:18:02.263123+0000","last_unstale":"2024-04-25T01:18:19.400042+0000","last_undegraded":"2024-04-25T01:18:19.400042+0000","last_fullsized":"2024-04-25T01:18:19.400042+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:09:21.858773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.430342+0000","last_change":"2024-04-25T01:18:19.426375+0000","last_active":"2024-04-25T01:18:19.430342+0000","last_peered":"2024-04-25T01:18:19.430342+0000","last_clean":"2024-04-25T01:18:19.430342+0000","last_became_active":"2024-04-25T01:18:19.425993+0000","last_became_peered":"2024-04-25T01:18:19.425993+0000","last_unstale":"2024-04-25T01:18:19.430342+0000","last_undegraded":"2024-04-25T01:18:19.430342+0000","last_fullsized":"2024-04-25T01:18:19.430342+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:30:55.006132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"42'15","reported_seq":44,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.412649+0000","last_change":"2024-04-25T01:18:03.269144+0000","last_active":"2024-04-25T01:18:19.412649+0000","last_peered":"2024-04-25T01:18:19.412649+0000","last_clean":"2024-04-25T01:18:19.412649+0000","last_became_active":"2024-04-25T01:18:03.268851+0000","last_became_peered":"2024-04-25T01:18:03.268851+0000","last_unstale":"2024-04-25T01:18:19.412649+0000","last_undegraded":"2024-04-25T01:18:19.412649+0000","last_fullsized":"2024-04-25T01:18:19.412649+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:55:37.223439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"39'1","reported_seq":31,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.413037+0000","last_change":"2024-04-25T01:18:02.255987+0000","last_active":"2024-04-25T01:18:19.413037+0000","last_peered":"2024-04-25T01:18:19.413037+0000","last_clean":"2024-04-25T01:18:19.413037+0000","last_became_active":"2024-04-25T01:18:02.255540+0000","last_became_peered":"2024-04-25T01:18:02.255540+0000","last_unstale":"2024-04-25T01:18:19.413037+0000","last_undegraded":"2024-04-25T01:18:19.413037+0000","last_fullsized":"2024-04-25T01:18:19.413037+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:38:12.422806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.422350+0000","last_change":"2024-04-25T01:18:19.418961+0000","last_active":"2024-04-25T01:18:19.422350+0000","last_peered":"2024-04-25T01:18:19.422350+0000","last_clean":"2024-04-25T01:18:19.422350+0000","last_became_active":"2024-04-25T01:18:19.418629+0000","last_became_peered":"2024-04-25T01:18:19.418629+0000","last_unstale":"2024-04-25T01:18:19.422350+0000","last_undegraded":"2024-04-25T01:18:19.422350+0000","last_fullsized":"2024-04-25T01:18:19.422350+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:54:12.857479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"42'5","reported_seq":35,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.407177+0000","last_change":"2024-04-25T01:18:03.260381+0000","last_active":"2024-04-25T01:18:19.407177+0000","last_peered":"2024-04-25T01:18:19.407177+0000","last_clean":"2024-04-25T01:18:19.407177+0000","last_became_active":"2024-04-25T01:18:03.260114+0000","last_became_peered":"2024-04-25T01:18:03.260114+0000","last_unstale":"2024-04-25T01:18:19.407177+0000","last_undegraded":"2024-04-25T01:18:19.407177+0000","last_fullsized":"2024-04-25T01:18:19.407177+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:10.899004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"43'3","reported_seq":33,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.422765+0000","last_change":"2024-04-25T01:18:02.270338+0000","last_active":"2024-04-25T01:18:19.422765+0000","last_peered":"2024-04-25T01:18:19.422765+0000","last_clean":"2024-04-25T01:18:19.422765+0000","last_became_active":"2024-04-25T01:18:02.270063+0000","last_became_peered":"2024-04-25T01:18:02.270063+0000","last_unstale":"2024-04-25T01:18:19.422765+0000","last_undegraded":"2024-04-25T01:18:19.422765+0000","last_fullsized":"2024-04-25T01:18:19.422765+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:25:09.988571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.431621+0000","last_change":"2024-04-25T01:18:19.430171+0000","last_active":"2024-04-25T01:18:19.431621+0000","last_peered":"2024-04-25T01:18:19.431621+0000","last_clean":"2024-04-25T01:18:19.431621+0000","last_became_active":"2024-04-25T01:18:19.429361+0000","last_became_peered":"2024-04-25T01:18:19.429361+0000","last_unstale":"2024-04-25T01:18:19.431621+0000","last_undegraded":"2024-04-25T01:18:19.431621+0000","last_fullsized":"2024-04-25T01:18:19.431621+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:55:47.861748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1f","version":"42'6","reported_seq":35,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.413019+0000","last_change":"2024-04-25T01:18:03.264063+0000","last_active":"2024-04-25T01:18:19.413019+0000","last_peered":"2024-04-25T01:18:19.413019+0000","last_clean":"2024-04-25T01:18:19.413019+0000","last_became_active":"2024-04-25T01:18:03.263708+0000","last_became_peered":"2024-04-25T01:18:03.263708+0000","last_unstale":"2024-04-25T01:18:19.413019+0000","last_undegraded":"2024-04-25T01:18:19.413019+0000","last_fullsized":"2024-04-25T01:18:19.413019+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:15.465731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"43'3","reported_seq":33,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.399076+0000","last_change":"2024-04-25T01:18:02.263250+0000","last_active":"2024-04-25T01:18:19.399076+0000","last_peered":"2024-04-25T01:18:19.399076+0000","last_clean":"2024-04-25T01:18:19.399076+0000","last_became_active":"2024-04-25T01:18:02.262763+0000","last_became_peered":"2024-04-25T01:18:02.262763+0000","last_unstale":"2024-04-25T01:18:19.399076+0000","last_undegraded":"2024-04-25T01:18:19.399076+0000","last_fullsized":"2024-04-25T01:18:19.399076+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:22:34.107130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.425141+0000","last_change":"2024-04-25T01:18:19.420413+0000","last_active":"2024-04-25T01:18:19.425141+0000","last_peered":"2024-04-25T01:18:19.425141+0000","last_clean":"2024-04-25T01:18:19.425141+0000","last_became_active":"2024-04-25T01:18:19.420145+0000","last_became_peered":"2024-04-25T01:18:19.420145+0000","last_unstale":"2024-04-25T01:18:19.425141+0000","last_undegraded":"2024-04-25T01:18:19.425141+0000","last_fullsized":"2024-04-25T01:18:19.425141+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:00:19.438096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"42'8","reported_seq":37,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.421842+0000","last_change":"2024-04-25T01:18:03.250303+0000","last_active":"2024-04-25T01:18:19.421842+0000","last_peered":"2024-04-25T01:18:19.421842+0000","last_clean":"2024-04-25T01:18:19.421842+0000","last_became_active":"2024-04-25T01:18:03.249995+0000","last_became_peered":"2024-04-25T01:18:03.249995+0000","last_unstale":"2024-04-25T01:18:19.421842+0000","last_undegraded":"2024-04-25T01:18:19.421842+0000","last_fullsized":"2024-04-25T01:18:19.421842+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:11.366143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"39'1","reported_seq":31,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.401035+0000","last_change":"2024-04-25T01:18:02.270642+0000","last_active":"2024-04-25T01:18:19.401035+0000","last_peered":"2024-04-25T01:18:19.401035+0000","last_clean":"2024-04-25T01:18:19.401035+0000","last_became_active":"2024-04-25T01:18:02.270383+0000","last_became_peered":"2024-04-25T01:18:02.270383+0000","last_unstale":"2024-04-25T01:18:19.401035+0000","last_undegraded":"2024-04-25T01:18:19.401035+0000","last_fullsized":"2024-04-25T01:18:19.401035+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:20:17.417113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.428101+0000","last_change":"2024-04-25T01:18:19.424356+0000","last_active":"2024-04-25T01:18:19.428101+0000","last_peered":"2024-04-25T01:18:19.428101+0000","last_clean":"2024-04-25T01:18:19.428101+0000","last_became_active":"2024-04-25T01:18:19.423905+0000","last_became_peered":"2024-04-25T01:18:19.423905+0000","last_unstale":"2024-04-25T01:18:19.428101+0000","last_undegraded":"2024-04-25T01:18:19.428101+0000","last_fullsized":"2024-04-25T01:18:19.428101+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:45:25.608799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"42'10","reported_seq":38,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.404680+0000","last_change":"2024-04-25T01:18:03.262229+0000","last_active":"2024-04-25T01:18:19.404680+0000","last_peered":"2024-04-25T01:18:19.404680+0000","last_clean":"2024-04-25T01:18:19.404680+0000","last_became_active":"2024-04-25T01:18:03.256356+0000","last_became_peered":"2024-04-25T01:18:03.256356+0000","last_unstale":"2024-04-25T01:18:19.404680+0000","last_undegraded":"2024-04-25T01:18:19.404680+0000","last_fullsized":"2024-04-25T01:18:19.404680+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:10:58.007657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"39'1","reported_seq":30,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.394763+0000","last_change":"2024-04-25T01:18:02.267951+0000","last_active":"2024-04-25T01:18:19.394763+0000","last_peered":"2024-04-25T01:18:19.394763+0000","last_clean":"2024-04-25T01:18:19.394763+0000","last_became_active":"2024-04-25T01:18:02.267086+0000","last_became_peered":"2024-04-25T01:18:02.267086+0000","last_unstale":"2024-04-25T01:18:19.394763+0000","last_undegraded":"2024-04-25T01:18:19.394763+0000","last_fullsized":"2024-04-25T01:18:19.394763+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:03:28.616338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.429647+0000","last_change":"2024-04-25T01:18:19.428103+0000","last_active":"2024-04-25T01:18:19.429647+0000","last_peered":"2024-04-25T01:18:19.429647+0000","last_clean":"2024-04-25T01:18:19.429647+0000","last_became_active":"2024-04-25T01:18:19.427765+0000","last_became_peered":"2024-04-25T01:18:19.427765+0000","last_unstale":"2024-04-25T01:18:19.429647+0000","last_undegraded":"2024-04-25T01:18:19.429647+0000","last_fullsized":"2024-04-25T01:18:19.429647+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:08:44.723881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"42'7","reported_seq":37,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.410279+0000","last_change":"2024-04-25T01:18:03.265044+0000","last_active":"2024-04-25T01:18:19.410279+0000","last_peered":"2024-04-25T01:18:19.410279+0000","last_clean":"2024-04-25T01:18:19.410279+0000","last_became_active":"2024-04-25T01:18:03.264645+0000","last_became_peered":"2024-04-25T01:18:03.264645+0000","last_unstale":"2024-04-25T01:18:19.410279+0000","last_undegraded":"2024-04-25T01:18:19.410279+0000","last_fullsized":"2024-04-25T01:18:19.410279+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:41:51.721827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":30,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.421355+0000","last_change":"2024-04-25T01:18:02.262809+0000","last_active":"2024-04-25T01:18:19.421355+0000","last_peered":"2024-04-25T01:18:19.421355+0000","last_clean":"2024-04-25T01:18:19.421355+0000","last_became_active":"2024-04-25T01:18:02.261620+0000","last_became_peered":"2024-04-25T01:18:02.261620+0000","last_unstale":"2024-04-25T01:18:19.421355+0000","last_undegraded":"2024-04-25T01:18:19.421355+0000","last_fullsized":"2024-04-25T01:18:19.421355+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:12.817891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.419684+0000","last_change":"2024-04-25T01:18:19.414293+0000","last_active":"2024-04-25T01:18:19.419684+0000","last_peered":"2024-04-25T01:18:19.419684+0000","last_clean":"2024-04-25T01:18:19.419684+0000","last_became_active":"2024-04-25T01:18:19.414004+0000","last_became_peered":"2024-04-25T01:18:19.414004+0000","last_unstale":"2024-04-25T01:18:19.419684+0000","last_undegraded":"2024-04-25T01:18:19.419684+0000","last_fullsized":"2024-04-25T01:18:19.419684+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:15:26.510336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"42'8","reported_seq":37,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.411498+0000","last_change":"2024-04-25T01:18:03.261875+0000","last_active":"2024-04-25T01:18:19.411498+0000","last_peered":"2024-04-25T01:18:19.411498+0000","last_clean":"2024-04-25T01:18:19.411498+0000","last_became_active":"2024-04-25T01:18:03.261413+0000","last_became_peered":"2024-04-25T01:18:03.261413+0000","last_unstale":"2024-04-25T01:18:19.411498+0000","last_undegraded":"2024-04-25T01:18:19.411498+0000","last_fullsized":"2024-04-25T01:18:19.411498+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:47.778367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":29,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.393687+0000","last_change":"2024-04-25T01:18:02.257390+0000","last_active":"2024-04-25T01:18:19.393687+0000","last_peered":"2024-04-25T01:18:19.393687+0000","last_clean":"2024-04-25T01:18:19.393687+0000","last_became_active":"2024-04-25T01:18:02.257118+0000","last_became_peered":"2024-04-25T01:18:02.257118+0000","last_unstale":"2024-04-25T01:18:19.393687+0000","last_undegraded":"2024-04-25T01:18:19.393687+0000","last_fullsized":"2024-04-25T01:18:19.393687+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:49:10.075668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.430677+0000","last_change":"2024-04-25T01:18:19.426279+0000","last_active":"2024-04-25T01:18:19.430677+0000","last_peered":"2024-04-25T01:18:19.430677+0000","last_clean":"2024-04-25T01:18:19.430677+0000","last_became_active":"2024-04-25T01:18:19.425962+0000","last_became_peered":"2024-04-25T01:18:19.425962+0000","last_unstale":"2024-04-25T01:18:19.430677+0000","last_undegraded":"2024-04-25T01:18:19.430677+0000","last_fullsized":"2024-04-25T01:18:19.430677+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:54:38.746281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"42'11","reported_seq":40,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.421727+0000","last_change":"2024-04-25T01:18:03.260224+0000","last_active":"2024-04-25T01:18:19.421727+0000","last_peered":"2024-04-25T01:18:19.421727+0000","last_clean":"2024-04-25T01:18:19.421727+0000","last_became_active":"2024-04-25T01:18:03.259886+0000","last_became_peered":"2024-04-25T01:18:03.259886+0000","last_unstale":"2024-04-25T01:18:19.421727+0000","last_undegraded":"2024-04-25T01:18:19.421727+0000","last_fullsized":"2024-04-25T01:18:19.421727+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:24:54.943447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":30,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.422622+0000","last_change":"2024-04-25T01:18:02.265416+0000","last_active":"2024-04-25T01:18:19.422622+0000","last_peered":"2024-04-25T01:18:19.422622+0000","last_clean":"2024-04-25T01:18:19.422622+0000","last_became_active":"2024-04-25T01:18:02.265123+0000","last_became_peered":"2024-04-25T01:18:02.265123+0000","last_unstale":"2024-04-25T01:18:19.422622+0000","last_undegraded":"2024-04-25T01:18:19.422622+0000","last_fullsized":"2024-04-25T01:18:19.422622+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:18.369587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.430059+0000","last_change":"2024-04-25T01:18:19.427690+0000","last_active":"2024-04-25T01:18:19.430059+0000","last_peered":"2024-04-25T01:18:19.430059+0000","last_clean":"2024-04-25T01:18:19.430059+0000","last_became_active":"2024-04-25T01:18:19.427363+0000","last_became_peered":"2024-04-25T01:18:19.427363+0000","last_unstale":"2024-04-25T01:18:19.430059+0000","last_undegraded":"2024-04-25T01:18:19.430059+0000","last_fullsized":"2024-04-25T01:18:19.430059+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:50:13.885015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"42'6","reported_seq":35,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.398240+0000","last_change":"2024-04-25T01:18:03.279931+0000","last_active":"2024-04-25T01:18:19.398240+0000","last_peered":"2024-04-25T01:18:19.398240+0000","last_clean":"2024-04-25T01:18:19.398240+0000","last_became_active":"2024-04-25T01:18:03.279643+0000","last_became_peered":"2024-04-25T01:18:03.279643+0000","last_unstale":"2024-04-25T01:18:19.398240+0000","last_undegraded":"2024-04-25T01:18:19.398240+0000","last_fullsized":"2024-04-25T01:18:19.398240+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:18:05.301920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":31,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.407020+0000","last_change":"2024-04-25T01:18:02.256098+0000","last_active":"2024-04-25T01:18:19.407020+0000","last_peered":"2024-04-25T01:18:19.407020+0000","last_clean":"2024-04-25T01:18:19.407020+0000","last_became_active":"2024-04-25T01:18:02.255609+0000","last_became_peered":"2024-04-25T01:18:02.255609+0000","last_unstale":"2024-04-25T01:18:19.407020+0000","last_undegraded":"2024-04-25T01:18:19.407020+0000","last_fullsized":"2024-04-25T01:18:19.407020+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:46:28.843042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.412953+0000","last_change":"2024-04-25T01:18:19.410836+0000","last_active":"2024-04-25T01:18:19.412953+0000","last_peered":"2024-04-25T01:18:19.412953+0000","last_clean":"2024-04-25T01:18:19.412953+0000","last_became_active":"2024-04-25T01:18:19.410588+0000","last_became_peered":"2024-04-25T01:18:19.410588+0000","last_unstale":"2024-04-25T01:18:19.412953+0000","last_undegraded":"2024-04-25T01:18:19.412953+0000","last_fullsized":"2024-04-25T01:18:19.412953+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:30:55.006132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"42'10","reported_seq":33,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.399420+0000","last_change":"2024-04-25T01:18:10.498339+0000","last_active":"2024-04-25T01:18:19.399420+0000","last_peered":"2024-04-25T01:18:19.399420+0000","last_clean":"2024-04-25T01:18:19.399420+0000","last_became_active":"2024-04-25T01:18:10.497866+0000","last_became_peered":"2024-04-25T01:18:10.497866+0000","last_unstale":"2024-04-25T01:18:19.399420+0000","last_undegraded":"2024-04-25T01:18:19.399420+0000","last_fullsized":"2024-04-25T01:18:19.399420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:57:10.276501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"43'5","reported_seq":36,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.410289+0000","last_change":"2024-04-25T01:18:02.263678+0000","last_active":"2024-04-25T01:18:19.410289+0000","last_peered":"2024-04-25T01:18:19.410289+0000","last_clean":"2024-04-25T01:18:19.410289+0000","last_became_active":"2024-04-25T01:18:02.263328+0000","last_became_peered":"2024-04-25T01:18:02.263328+0000","last_unstale":"2024-04-25T01:18:19.410289+0000","last_undegraded":"2024-04-25T01:18:19.410289+0000","last_fullsized":"2024-04-25T01:18:19.410289+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:30:37.768204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.427942+0000","last_change":"2024-04-25T01:18:19.423579+0000","last_active":"2024-04-25T01:18:19.427942+0000","last_peered":"2024-04-25T01:18:19.427942+0000","last_clean":"2024-04-25T01:18:19.427942+0000","last_became_active":"2024-04-25T01:18:19.423281+0000","last_became_peered":"2024-04-25T01:18:19.423281+0000","last_unstale":"2024-04-25T01:18:19.427942+0000","last_undegraded":"2024-04-25T01:18:19.427942+0000","last_fullsized":"2024-04-25T01:18:19.427942+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:46:46.080969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"42'9","reported_seq":37,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.396763+0000","last_change":"2024-04-25T01:18:03.257743+0000","last_active":"2024-04-25T01:18:19.396763+0000","last_peered":"2024-04-25T01:18:19.396763+0000","last_clean":"2024-04-25T01:18:19.396763+0000","last_became_active":"2024-04-25T01:18:03.257421+0000","last_became_peered":"2024-04-25T01:18:03.257421+0000","last_unstale":"2024-04-25T01:18:19.396763+0000","last_undegraded":"2024-04-25T01:18:19.396763+0000","last_fullsized":"2024-04-25T01:18:19.396763+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:40.084475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.14","version":"43'23","reported_seq":53,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.421843+0000","last_change":"2024-04-25T01:18:02.269681+0000","last_active":"2024-04-25T01:18:19.421843+0000","last_peered":"2024-04-25T01:18:19.421843+0000","last_clean":"2024-04-25T01:18:19.421843+0000","last_became_active":"2024-04-25T01:18:02.268702+0000","last_became_peered":"2024-04-25T01:18:02.268702+0000","last_unstale":"2024-04-25T01:18:19.421843+0000","last_undegraded":"2024-04-25T01:18:19.421843+0000","last_fullsized":"2024-04-25T01:18:19.421843+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:50:13.482420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.416494+0000","last_change":"2024-04-25T01:18:19.411722+0000","last_active":"2024-04-25T01:18:19.416494+0000","last_peered":"2024-04-25T01:18:19.416494+0000","last_clean":"2024-04-25T01:18:19.416494+0000","last_became_active":"2024-04-25T01:18:19.411432+0000","last_became_peered":"2024-04-25T01:18:19.411432+0000","last_unstale":"2024-04-25T01:18:19.416494+0000","last_undegraded":"2024-04-25T01:18:19.416494+0000","last_fullsized":"2024-04-25T01:18:19.416494+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:06:50.650712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"42'6","reported_seq":35,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.422070+0000","last_change":"2024-04-25T01:18:03.269967+0000","last_active":"2024-04-25T01:18:19.422070+0000","last_peered":"2024-04-25T01:18:19.422070+0000","last_clean":"2024-04-25T01:18:19.422070+0000","last_became_active":"2024-04-25T01:18:03.269663+0000","last_became_peered":"2024-04-25T01:18:03.269663+0000","last_unstale":"2024-04-25T01:18:19.422070+0000","last_undegraded":"2024-04-25T01:18:19.422070+0000","last_fullsized":"2024-04-25T01:18:19.422070+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:32:32.815305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"43'5","reported_seq":35,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.421509+0000","last_change":"2024-04-25T01:18:02.272092+0000","last_active":"2024-04-25T01:18:19.421509+0000","last_peered":"2024-04-25T01:18:19.421509+0000","last_clean":"2024-04-25T01:18:19.421509+0000","last_became_active":"2024-04-25T01:18:02.271767+0000","last_became_peered":"2024-04-25T01:18:02.271767+0000","last_unstale":"2024-04-25T01:18:19.421509+0000","last_undegraded":"2024-04-25T01:18:19.421509+0000","last_fullsized":"2024-04-25T01:18:19.421509+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:10.420303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.427794+0000","last_change":"2024-04-25T01:18:19.425131+0000","last_active":"2024-04-25T01:18:19.427794+0000","last_peered":"2024-04-25T01:18:19.427794+0000","last_clean":"2024-04-25T01:18:19.427794+0000","last_became_active":"2024-04-25T01:18:19.424804+0000","last_became_peered":"2024-04-25T01:18:19.424804+0000","last_unstale":"2024-04-25T01:18:19.427794+0000","last_undegraded":"2024-04-25T01:18:19.427794+0000","last_fullsized":"2024-04-25T01:18:19.427794+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:25:27.226499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.11","version":"42'6","reported_seq":35,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.412059+0000","last_change":"2024-04-25T01:18:03.258055+0000","last_active":"2024-04-25T01:18:19.412059+0000","last_peered":"2024-04-25T01:18:19.412059+0000","last_clean":"2024-04-25T01:18:19.412059+0000","last_became_active":"2024-04-25T01:18:03.257517+0000","last_became_peered":"2024-04-25T01:18:03.257517+0000","last_unstale":"2024-04-25T01:18:19.412059+0000","last_undegraded":"2024-04-25T01:18:19.412059+0000","last_fullsized":"2024-04-25T01:18:19.412059+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:47:02.891879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":30,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.412245+0000","last_change":"2024-04-25T01:18:02.258795+0000","last_active":"2024-04-25T01:18:19.412245+0000","last_peered":"2024-04-25T01:18:19.412245+0000","last_clean":"2024-04-25T01:18:19.412245+0000","last_became_active":"2024-04-25T01:18:02.258494+0000","last_became_peered":"2024-04-25T01:18:02.258494+0000","last_unstale":"2024-04-25T01:18:19.412245+0000","last_undegraded":"2024-04-25T01:18:19.412245+0000","last_fullsized":"2024-04-25T01:18:19.412245+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:16:48.693012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.422756+0000","last_change":"2024-04-25T01:18:19.419126+0000","last_active":"2024-04-25T01:18:19.422756+0000","last_peered":"2024-04-25T01:18:19.422756+0000","last_clean":"2024-04-25T01:18:19.422756+0000","last_became_active":"2024-04-25T01:18:19.418836+0000","last_became_peered":"2024-04-25T01:18:19.418836+0000","last_unstale":"2024-04-25T01:18:19.422756+0000","last_undegraded":"2024-04-25T01:18:19.422756+0000","last_fullsized":"2024-04-25T01:18:19.422756+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:36.248433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"42'10","reported_seq":38,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.406704+0000","last_change":"2024-04-25T01:18:03.254569+0000","last_active":"2024-04-25T01:18:19.406704+0000","last_peered":"2024-04-25T01:18:19.406704+0000","last_clean":"2024-04-25T01:18:19.406704+0000","last_became_active":"2024-04-25T01:18:03.254299+0000","last_became_peered":"2024-04-25T01:18:03.254299+0000","last_unstale":"2024-04-25T01:18:19.406704+0000","last_undegraded":"2024-04-25T01:18:19.406704+0000","last_fullsized":"2024-04-25T01:18:19.406704+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:23:03.906624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":31,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.409225+0000","last_change":"2024-04-25T01:18:02.254663+0000","last_active":"2024-04-25T01:18:19.409225+0000","last_peered":"2024-04-25T01:18:19.409225+0000","last_clean":"2024-04-25T01:18:19.409225+0000","last_became_active":"2024-04-25T01:18:02.254405+0000","last_became_peered":"2024-04-25T01:18:02.254405+0000","last_unstale":"2024-04-25T01:18:19.409225+0000","last_undegraded":"2024-04-25T01:18:19.409225+0000","last_fullsized":"2024-04-25T01:18:19.409225+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:00:56.792493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.409394+0000","last_change":"2024-04-25T01:18:19.407532+0000","last_active":"2024-04-25T01:18:19.409394+0000","last_peered":"2024-04-25T01:18:19.409394+0000","last_clean":"2024-04-25T01:18:19.409394+0000","last_became_active":"2024-04-25T01:18:19.406223+0000","last_became_peered":"2024-04-25T01:18:19.406223+0000","last_unstale":"2024-04-25T01:18:19.409394+0000","last_undegraded":"2024-04-25T01:18:19.409394+0000","last_fullsized":"2024-04-25T01:18:19.409394+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:01:14.030421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"42'15","reported_seq":44,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.407855+0000","last_change":"2024-04-25T01:18:03.252241+0000","last_active":"2024-04-25T01:18:19.407855+0000","last_peered":"2024-04-25T01:18:19.407855+0000","last_clean":"2024-04-25T01:18:19.407855+0000","last_became_active":"2024-04-25T01:18:03.251874+0000","last_became_peered":"2024-04-25T01:18:03.251874+0000","last_unstale":"2024-04-25T01:18:19.407855+0000","last_undegraded":"2024-04-25T01:18:19.407855+0000","last_fullsized":"2024-04-25T01:18:19.407855+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:11.435894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"43'15","reported_seq":45,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.400045+0000","last_change":"2024-04-25T01:18:02.258502+0000","last_active":"2024-04-25T01:18:19.400045+0000","last_peered":"2024-04-25T01:18:19.400045+0000","last_clean":"2024-04-25T01:18:19.400045+0000","last_became_active":"2024-04-25T01:18:02.258205+0000","last_became_peered":"2024-04-25T01:18:02.258205+0000","last_unstale":"2024-04-25T01:18:19.400045+0000","last_undegraded":"2024-04-25T01:18:19.400045+0000","last_fullsized":"2024-04-25T01:18:19.400045+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:20:28.341160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.430613+0000","last_change":"2024-04-25T01:18:19.425664+0000","last_active":"2024-04-25T01:18:19.430613+0000","last_peered":"2024-04-25T01:18:19.430613+0000","last_clean":"2024-04-25T01:18:19.430613+0000","last_became_active":"2024-04-25T01:18:19.425355+0000","last_became_peered":"2024-04-25T01:18:19.425355+0000","last_unstale":"2024-04-25T01:18:19.430613+0000","last_undegraded":"2024-04-25T01:18:19.430613+0000","last_fullsized":"2024-04-25T01:18:19.430613+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:10:04.612611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"42'6","reported_seq":35,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.412163+0000","last_change":"2024-04-25T01:18:03.275449+0000","last_active":"2024-04-25T01:18:19.412163+0000","last_peered":"2024-04-25T01:18:19.412163+0000","last_clean":"2024-04-25T01:18:19.412163+0000","last_became_active":"2024-04-25T01:18:03.275006+0000","last_became_peered":"2024-04-25T01:18:03.275006+0000","last_unstale":"2024-04-25T01:18:19.412163+0000","last_undegraded":"2024-04-25T01:18:19.412163+0000","last_fullsized":"2024-04-25T01:18:19.412163+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:48:53.313464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"43'2","reported_seq":31,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.397019+0000","last_change":"2024-04-25T01:18:02.259736+0000","last_active":"2024-04-25T01:18:19.397019+0000","last_peered":"2024-04-25T01:18:19.397019+0000","last_clean":"2024-04-25T01:18:19.397019+0000","last_became_active":"2024-04-25T01:18:02.259127+0000","last_became_peered":"2024-04-25T01:18:02.259127+0000","last_unstale":"2024-04-25T01:18:19.397019+0000","last_undegraded":"2024-04-25T01:18:19.397019+0000","last_fullsized":"2024-04-25T01:18:19.397019+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:58:04.484125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.407277+0000","last_change":"2024-04-25T01:18:19.406596+0000","last_active":"2024-04-25T01:18:19.407277+0000","last_peered":"2024-04-25T01:18:19.407277+0000","last_clean":"2024-04-25T01:18:19.407277+0000","last_became_active":"2024-04-25T01:18:19.406280+0000","last_became_peered":"2024-04-25T01:18:19.406280+0000","last_unstale":"2024-04-25T01:18:19.407277+0000","last_undegraded":"2024-04-25T01:18:19.407277+0000","last_fullsized":"2024-04-25T01:18:19.407277+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:01:14.030421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"42'7","reported_seq":35,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.396232+0000","last_change":"2024-04-25T01:18:03.270127+0000","last_active":"2024-04-25T01:18:19.396232+0000","last_peered":"2024-04-25T01:18:19.396232+0000","last_clean":"2024-04-25T01:18:19.396232+0000","last_became_active":"2024-04-25T01:18:03.267471+0000","last_became_peered":"2024-04-25T01:18:03.267471+0000","last_unstale":"2024-04-25T01:18:19.396232+0000","last_undegraded":"2024-04-25T01:18:19.396232+0000","last_fullsized":"2024-04-25T01:18:19.396232+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:50:36.431482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"43'3","reported_seq":34,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.407279+0000","last_change":"2024-04-25T01:18:02.267703+0000","last_active":"2024-04-25T01:18:19.407279+0000","last_peered":"2024-04-25T01:18:19.407279+0000","last_clean":"2024-04-25T01:18:19.407279+0000","last_became_active":"2024-04-25T01:18:02.267415+0000","last_became_peered":"2024-04-25T01:18:02.267415+0000","last_unstale":"2024-04-25T01:18:19.407279+0000","last_undegraded":"2024-04-25T01:18:19.407279+0000","last_fullsized":"2024-04-25T01:18:19.407279+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:09:47.374684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.431182+0000","last_change":"2024-04-25T01:18:19.429464+0000","last_active":"2024-04-25T01:18:19.431182+0000","last_peered":"2024-04-25T01:18:19.431182+0000","last_clean":"2024-04-25T01:18:19.431182+0000","last_became_active":"2024-04-25T01:18:19.429156+0000","last_became_peered":"2024-04-25T01:18:19.429156+0000","last_unstale":"2024-04-25T01:18:19.431182+0000","last_undegraded":"2024-04-25T01:18:19.431182+0000","last_fullsized":"2024-04-25T01:18:19.431182+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:07.863347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"42'5","reported_seq":33,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.396376+0000","last_change":"2024-04-25T01:18:03.266233+0000","last_active":"2024-04-25T01:18:19.396376+0000","last_peered":"2024-04-25T01:18:19.396376+0000","last_clean":"2024-04-25T01:18:19.396376+0000","last_became_active":"2024-04-25T01:18:03.265929+0000","last_became_peered":"2024-04-25T01:18:03.265929+0000","last_unstale":"2024-04-25T01:18:19.396376+0000","last_undegraded":"2024-04-25T01:18:19.396376+0000","last_fullsized":"2024-04-25T01:18:19.396376+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:30:00.416602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"43'2","reported_seq":32,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.406897+0000","last_change":"2024-04-25T01:18:02.267511+0000","last_active":"2024-04-25T01:18:19.406897+0000","last_peered":"2024-04-25T01:18:19.406897+0000","last_clean":"2024-04-25T01:18:19.406897+0000","last_became_active":"2024-04-25T01:18:02.266732+0000","last_became_peered":"2024-04-25T01:18:02.266732+0000","last_unstale":"2024-04-25T01:18:19.406897+0000","last_undegraded":"2024-04-25T01:18:19.406897+0000","last_fullsized":"2024-04-25T01:18:19.406897+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:58:04.484125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.425463+0000","last_change":"2024-04-25T01:18:19.423488+0000","last_active":"2024-04-25T01:18:19.425463+0000","last_peered":"2024-04-25T01:18:19.425463+0000","last_clean":"2024-04-25T01:18:19.425463+0000","last_became_active":"2024-04-25T01:18:19.423143+0000","last_became_peered":"2024-04-25T01:18:19.423143+0000","last_unstale":"2024-04-25T01:18:19.425463+0000","last_undegraded":"2024-04-25T01:18:19.425463+0000","last_fullsized":"2024-04-25T01:18:19.425463+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:33.361047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"42'9","reported_seq":40,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.401861+0000","last_change":"2024-04-25T01:18:03.255121+0000","last_active":"2024-04-25T01:18:19.401861+0000","last_peered":"2024-04-25T01:18:19.401861+0000","last_clean":"2024-04-25T01:18:19.401861+0000","last_became_active":"2024-04-25T01:18:03.254831+0000","last_became_peered":"2024-04-25T01:18:03.254831+0000","last_unstale":"2024-04-25T01:18:19.401861+0000","last_undegraded":"2024-04-25T01:18:19.401861+0000","last_fullsized":"2024-04-25T01:18:19.401861+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:42:01.659435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":32,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.399392+0000","last_change":"2024-04-25T01:18:02.259323+0000","last_active":"2024-04-25T01:18:19.399392+0000","last_peered":"2024-04-25T01:18:19.399392+0000","last_clean":"2024-04-25T01:18:19.399392+0000","last_became_active":"2024-04-25T01:18:02.258937+0000","last_became_peered":"2024-04-25T01:18:02.258937+0000","last_unstale":"2024-04-25T01:18:19.399392+0000","last_undegraded":"2024-04-25T01:18:19.399392+0000","last_fullsized":"2024-04-25T01:18:19.399392+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:12.725383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.430079+0000","last_change":"2024-04-25T01:18:19.426878+0000","last_active":"2024-04-25T01:18:19.430079+0000","last_peered":"2024-04-25T01:18:19.430079+0000","last_clean":"2024-04-25T01:18:19.430079+0000","last_became_active":"2024-04-25T01:18:19.426297+0000","last_became_peered":"2024-04-25T01:18:19.426297+0000","last_unstale":"2024-04-25T01:18:19.430079+0000","last_undegraded":"2024-04-25T01:18:19.430079+0000","last_fullsized":"2024-04-25T01:18:19.430079+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:47:19.114215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"42'9","reported_seq":39,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.409925+0000","last_change":"2024-04-25T01:18:03.253199+0000","last_active":"2024-04-25T01:18:19.409925+0000","last_peered":"2024-04-25T01:18:19.409925+0000","last_clean":"2024-04-25T01:18:19.409925+0000","last_became_active":"2024-04-25T01:18:03.252912+0000","last_became_peered":"2024-04-25T01:18:03.252912+0000","last_unstale":"2024-04-25T01:18:19.409925+0000","last_undegraded":"2024-04-25T01:18:19.409925+0000","last_fullsized":"2024-04-25T01:18:19.409925+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:26:28.066399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"43'4","reported_seq":36,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.399834+0000","last_change":"2024-04-25T01:18:02.267229+0000","last_active":"2024-04-25T01:18:19.399834+0000","last_peered":"2024-04-25T01:18:19.399834+0000","last_clean":"2024-04-25T01:18:19.399834+0000","last_became_active":"2024-04-25T01:18:02.266605+0000","last_became_peered":"2024-04-25T01:18:02.266605+0000","last_unstale":"2024-04-25T01:18:19.399834+0000","last_undegraded":"2024-04-25T01:18:19.399834+0000","last_fullsized":"2024-04-25T01:18:19.399834+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:34:33.983053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.423610+0000","last_change":"2024-04-25T01:18:19.405615+0000","last_active":"2024-04-25T01:18:19.423610+0000","last_peered":"2024-04-25T01:18:19.423610+0000","last_clean":"2024-04-25T01:18:19.423610+0000","last_became_active":"2024-04-25T01:18:19.404531+0000","last_became_peered":"2024-04-25T01:18:19.404531+0000","last_unstale":"2024-04-25T01:18:19.423610+0000","last_undegraded":"2024-04-25T01:18:19.423610+0000","last_fullsized":"2024-04-25T01:18:19.423610+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:49:09.535800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"42'2","reported_seq":31,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.421200+0000","last_change":"2024-04-25T01:18:03.274242+0000","last_active":"2024-04-25T01:18:19.421200+0000","last_peered":"2024-04-25T01:18:19.421200+0000","last_clean":"2024-04-25T01:18:19.421200+0000","last_became_active":"2024-04-25T01:18:03.273857+0000","last_became_peered":"2024-04-25T01:18:03.273857+0000","last_unstale":"2024-04-25T01:18:19.421200+0000","last_undegraded":"2024-04-25T01:18:19.421200+0000","last_fullsized":"2024-04-25T01:18:19.421200+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:39:30.588445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":30,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.412087+0000","last_change":"2024-04-25T01:18:02.259159+0000","last_active":"2024-04-25T01:18:19.412087+0000","last_peered":"2024-04-25T01:18:19.412087+0000","last_clean":"2024-04-25T01:18:19.412087+0000","last_became_active":"2024-04-25T01:18:02.257389+0000","last_became_peered":"2024-04-25T01:18:02.257389+0000","last_unstale":"2024-04-25T01:18:19.412087+0000","last_undegraded":"2024-04-25T01:18:19.412087+0000","last_fullsized":"2024-04-25T01:18:19.412087+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:43.394014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.423600+0000","last_change":"2024-04-25T01:18:19.420297+0000","last_active":"2024-04-25T01:18:19.423600+0000","last_peered":"2024-04-25T01:18:19.423600+0000","last_clean":"2024-04-25T01:18:19.423600+0000","last_became_active":"2024-04-25T01:18:19.420011+0000","last_became_peered":"2024-04-25T01:18:19.420011+0000","last_unstale":"2024-04-25T01:18:19.423600+0000","last_undegraded":"2024-04-25T01:18:19.423600+0000","last_fullsized":"2024-04-25T01:18:19.423600+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:50:51.812840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"42'5","reported_seq":33,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.406210+0000","last_change":"2024-04-25T01:18:03.274693+0000","last_active":"2024-04-25T01:18:19.406210+0000","last_peered":"2024-04-25T01:18:19.406210+0000","last_clean":"2024-04-25T01:18:19.406210+0000","last_became_active":"2024-04-25T01:18:03.274419+0000","last_became_peered":"2024-04-25T01:18:03.274419+0000","last_unstale":"2024-04-25T01:18:19.406210+0000","last_undegraded":"2024-04-25T01:18:19.406210+0000","last_fullsized":"2024-04-25T01:18:19.406210+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:03:36.196096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":29,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.394177+0000","last_change":"2024-04-25T01:18:02.254033+0000","last_active":"2024-04-25T01:18:19.394177+0000","last_peered":"2024-04-25T01:18:19.394177+0000","last_clean":"2024-04-25T01:18:19.394177+0000","last_became_active":"2024-04-25T01:18:02.253731+0000","last_became_peered":"2024-04-25T01:18:02.253731+0000","last_unstale":"2024-04-25T01:18:19.394177+0000","last_undegraded":"2024-04-25T01:18:19.394177+0000","last_fullsized":"2024-04-25T01:18:19.394177+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:00:15.697767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.428595+0000","last_change":"2024-04-25T01:18:19.426316+0000","last_active":"2024-04-25T01:18:19.428595+0000","last_peered":"2024-04-25T01:18:19.428595+0000","last_clean":"2024-04-25T01:18:19.428595+0000","last_became_active":"2024-04-25T01:18:19.426037+0000","last_became_peered":"2024-04-25T01:18:19.426037+0000","last_unstale":"2024-04-25T01:18:19.428595+0000","last_undegraded":"2024-04-25T01:18:19.428595+0000","last_fullsized":"2024-04-25T01:18:19.428595+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:27.121340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"42'8","reported_seq":37,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.421384+0000","last_change":"2024-04-25T01:18:03.258959+0000","last_active":"2024-04-25T01:18:19.421384+0000","last_peered":"2024-04-25T01:18:19.421384+0000","last_clean":"2024-04-25T01:18:19.421384+0000","last_became_active":"2024-04-25T01:18:03.258462+0000","last_became_peered":"2024-04-25T01:18:03.258462+0000","last_unstale":"2024-04-25T01:18:19.421384+0000","last_undegraded":"2024-04-25T01:18:19.421384+0000","last_fullsized":"2024-04-25T01:18:19.421384+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:52:20.724622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":30,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.411462+0000","last_change":"2024-04-25T01:18:02.266874+0000","last_active":"2024-04-25T01:18:19.411462+0000","last_peered":"2024-04-25T01:18:19.411462+0000","last_clean":"2024-04-25T01:18:19.411462+0000","last_became_active":"2024-04-25T01:18:02.266188+0000","last_became_peered":"2024-04-25T01:18:02.266188+0000","last_unstale":"2024-04-25T01:18:19.411462+0000","last_undegraded":"2024-04-25T01:18:19.411462+0000","last_fullsized":"2024-04-25T01:18:19.411462+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:12:59.564111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.430694+0000","last_change":"2024-04-25T01:18:19.426888+0000","last_active":"2024-04-25T01:18:19.430694+0000","last_peered":"2024-04-25T01:18:19.430694+0000","last_clean":"2024-04-25T01:18:19.430694+0000","last_became_active":"2024-04-25T01:18:19.426641+0000","last_became_peered":"2024-04-25T01:18:19.426641+0000","last_unstale":"2024-04-25T01:18:19.430694+0000","last_undegraded":"2024-04-25T01:18:19.430694+0000","last_fullsized":"2024-04-25T01:18:19.430694+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:22.653702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"42'9","reported_seq":38,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.409356+0000","last_change":"2024-04-25T01:18:03.269115+0000","last_active":"2024-04-25T01:18:19.409356+0000","last_peered":"2024-04-25T01:18:19.409356+0000","last_clean":"2024-04-25T01:18:19.409356+0000","last_became_active":"2024-04-25T01:18:03.268774+0000","last_became_peered":"2024-04-25T01:18:03.268774+0000","last_unstale":"2024-04-25T01:18:19.409356+0000","last_undegraded":"2024-04-25T01:18:19.409356+0000","last_fullsized":"2024-04-25T01:18:19.409356+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:50:14.498011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"43'3","reported_seq":34,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.412560+0000","last_change":"2024-04-25T01:18:02.257314+0000","last_active":"2024-04-25T01:18:19.412560+0000","last_peered":"2024-04-25T01:18:19.412560+0000","last_clean":"2024-04-25T01:18:19.412560+0000","last_became_active":"2024-04-25T01:18:02.256973+0000","last_became_peered":"2024-04-25T01:18:02.256973+0000","last_unstale":"2024-04-25T01:18:19.412560+0000","last_undegraded":"2024-04-25T01:18:19.412560+0000","last_fullsized":"2024-04-25T01:18:19.412560+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:06:24.081982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.429512+0000","last_change":"2024-04-25T01:18:19.427898+0000","last_active":"2024-04-25T01:18:19.429512+0000","last_peered":"2024-04-25T01:18:19.429512+0000","last_clean":"2024-04-25T01:18:19.429512+0000","last_became_active":"2024-04-25T01:18:19.427260+0000","last_became_peered":"2024-04-25T01:18:19.427260+0000","last_unstale":"2024-04-25T01:18:19.429512+0000","last_undegraded":"2024-04-25T01:18:19.429512+0000","last_fullsized":"2024-04-25T01:18:19.429512+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:33:48.493716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"42'5","reported_seq":33,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.408993+0000","last_change":"2024-04-25T01:18:03.276655+0000","last_active":"2024-04-25T01:18:19.408993+0000","last_peered":"2024-04-25T01:18:19.408993+0000","last_clean":"2024-04-25T01:18:19.408993+0000","last_became_active":"2024-04-25T01:18:03.276023+0000","last_became_peered":"2024-04-25T01:18:03.276023+0000","last_unstale":"2024-04-25T01:18:19.408993+0000","last_undegraded":"2024-04-25T01:18:19.408993+0000","last_fullsized":"2024-04-25T01:18:19.408993+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:29:13.129901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"43'4","reported_seq":34,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.421899+0000","last_change":"2024-04-25T01:18:02.265987+0000","last_active":"2024-04-25T01:18:19.421899+0000","last_peered":"2024-04-25T01:18:19.421899+0000","last_clean":"2024-04-25T01:18:19.421899+0000","last_became_active":"2024-04-25T01:18:02.265704+0000","last_became_peered":"2024-04-25T01:18:02.265704+0000","last_unstale":"2024-04-25T01:18:19.421899+0000","last_undegraded":"2024-04-25T01:18:19.421899+0000","last_fullsized":"2024-04-25T01:18:19.421899+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:03:28.616338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.425103+0000","last_change":"2024-04-25T01:18:19.419566+0000","last_active":"2024-04-25T01:18:19.425103+0000","last_peered":"2024-04-25T01:18:19.425103+0000","last_clean":"2024-04-25T01:18:19.425103+0000","last_became_active":"2024-04-25T01:18:19.419251+0000","last_became_peered":"2024-04-25T01:18:19.419251+0000","last_unstale":"2024-04-25T01:18:19.425103+0000","last_undegraded":"2024-04-25T01:18:19.425103+0000","last_fullsized":"2024-04-25T01:18:19.425103+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:47:23.704689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"42'7","reported_seq":38,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.400751+0000","last_change":"2024-04-25T01:18:03.260610+0000","last_active":"2024-04-25T01:18:19.400751+0000","last_peered":"2024-04-25T01:18:19.400751+0000","last_clean":"2024-04-25T01:18:19.400751+0000","last_became_active":"2024-04-25T01:18:03.259487+0000","last_became_peered":"2024-04-25T01:18:03.259487+0000","last_unstale":"2024-04-25T01:18:19.400751+0000","last_undegraded":"2024-04-25T01:18:19.400751+0000","last_fullsized":"2024-04-25T01:18:19.400751+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:47.778367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":30,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.407252+0000","last_change":"2024-04-25T01:18:02.269139+0000","last_active":"2024-04-25T01:18:19.407252+0000","last_peered":"2024-04-25T01:18:19.407252+0000","last_clean":"2024-04-25T01:18:19.407252+0000","last_became_active":"2024-04-25T01:18:02.268747+0000","last_became_peered":"2024-04-25T01:18:02.268747+0000","last_unstale":"2024-04-25T01:18:19.407252+0000","last_undegraded":"2024-04-25T01:18:19.407252+0000","last_fullsized":"2024-04-25T01:18:19.407252+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:49:10.075668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.406280+0000","last_change":"2024-04-25T01:18:19.404519+0000","last_active":"2024-04-25T01:18:19.406280+0000","last_peered":"2024-04-25T01:18:19.406280+0000","last_clean":"2024-04-25T01:18:19.406280+0000","last_became_active":"2024-04-25T01:18:19.404144+0000","last_became_peered":"2024-04-25T01:18:19.404144+0000","last_unstale":"2024-04-25T01:18:19.406280+0000","last_undegraded":"2024-04-25T01:18:19.406280+0000","last_fullsized":"2024-04-25T01:18:19.406280+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:56.306811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"42'8","reported_seq":37,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.407134+0000","last_change":"2024-04-25T01:18:03.254793+0000","last_active":"2024-04-25T01:18:19.407134+0000","last_peered":"2024-04-25T01:18:19.407134+0000","last_clean":"2024-04-25T01:18:19.407134+0000","last_became_active":"2024-04-25T01:18:03.253467+0000","last_became_peered":"2024-04-25T01:18:03.253467+0000","last_unstale":"2024-04-25T01:18:19.407134+0000","last_undegraded":"2024-04-25T01:18:19.407134+0000","last_fullsized":"2024-04-25T01:18:19.407134+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:02:29.348596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":30,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.422640+0000","last_change":"2024-04-25T01:18:02.269410+0000","last_active":"2024-04-25T01:18:19.422640+0000","last_peered":"2024-04-25T01:18:19.422640+0000","last_clean":"2024-04-25T01:18:19.422640+0000","last_became_active":"2024-04-25T01:18:02.268924+0000","last_became_peered":"2024-04-25T01:18:02.268924+0000","last_unstale":"2024-04-25T01:18:19.422640+0000","last_undegraded":"2024-04-25T01:18:19.422640+0000","last_fullsized":"2024-04-25T01:18:19.422640+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:19.010506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.410607+0000","last_change":"2024-04-25T01:18:19.408014+0000","last_active":"2024-04-25T01:18:19.410607+0000","last_peered":"2024-04-25T01:18:19.410607+0000","last_clean":"2024-04-25T01:18:19.410607+0000","last_became_active":"2024-04-25T01:18:19.405706+0000","last_became_peered":"2024-04-25T01:18:19.405706+0000","last_unstale":"2024-04-25T01:18:19.410607+0000","last_undegraded":"2024-04-25T01:18:19.410607+0000","last_fullsized":"2024-04-25T01:18:19.410607+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:06:50.650712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"42'5","reported_seq":34,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.415722+0000","last_change":"2024-04-25T01:18:03.261409+0000","last_active":"2024-04-25T01:18:19.415722+0000","last_peered":"2024-04-25T01:18:19.415722+0000","last_clean":"2024-04-25T01:18:19.415722+0000","last_became_active":"2024-04-25T01:18:03.261126+0000","last_became_peered":"2024-04-25T01:18:03.261126+0000","last_unstale":"2024-04-25T01:18:19.415722+0000","last_undegraded":"2024-04-25T01:18:19.415722+0000","last_fullsized":"2024-04-25T01:18:19.415722+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:49:57.662679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":29,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.406913+0000","last_change":"2024-04-25T01:18:02.267491+0000","last_active":"2024-04-25T01:18:19.406913+0000","last_peered":"2024-04-25T01:18:19.406913+0000","last_clean":"2024-04-25T01:18:19.406913+0000","last_became_active":"2024-04-25T01:18:02.267044+0000","last_became_peered":"2024-04-25T01:18:02.267044+0000","last_unstale":"2024-04-25T01:18:19.406913+0000","last_undegraded":"2024-04-25T01:18:19.406913+0000","last_fullsized":"2024-04-25T01:18:19.406913+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:30:07.096033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.423467+0000","last_change":"2024-04-25T01:18:19.408627+0000","last_active":"2024-04-25T01:18:19.423467+0000","last_peered":"2024-04-25T01:18:19.423467+0000","last_clean":"2024-04-25T01:18:19.423467+0000","last_became_active":"2024-04-25T01:18:19.407856+0000","last_became_peered":"2024-04-25T01:18:19.407856+0000","last_unstale":"2024-04-25T01:18:19.423467+0000","last_undegraded":"2024-04-25T01:18:19.423467+0000","last_fullsized":"2024-04-25T01:18:19.423467+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:23:29.707982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2","version":"42'8","reported_seq":38,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.410409+0000","last_change":"2024-04-25T01:18:03.254489+0000","last_active":"2024-04-25T01:18:19.410409+0000","last_peered":"2024-04-25T01:18:19.410409+0000","last_clean":"2024-04-25T01:18:19.410409+0000","last_became_active":"2024-04-25T01:18:03.254164+0000","last_became_peered":"2024-04-25T01:18:03.254164+0000","last_unstale":"2024-04-25T01:18:19.410409+0000","last_undegraded":"2024-04-25T01:18:19.410409+0000","last_fullsized":"2024-04-25T01:18:19.410409+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:31:29.760406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":31,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.409273+0000","last_change":"2024-04-25T01:18:02.263853+0000","last_active":"2024-04-25T01:18:19.409273+0000","last_peered":"2024-04-25T01:18:19.409273+0000","last_clean":"2024-04-25T01:18:19.409273+0000","last_became_active":"2024-04-25T01:18:02.263501+0000","last_became_peered":"2024-04-25T01:18:02.263501+0000","last_unstale":"2024-04-25T01:18:19.409273+0000","last_undegraded":"2024-04-25T01:18:19.409273+0000","last_fullsized":"2024-04-25T01:18:19.409273+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:06:33.412784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.428367+0000","last_change":"2024-04-25T01:18:19.425151+0000","last_active":"2024-04-25T01:18:19.428367+0000","last_peered":"2024-04-25T01:18:19.428367+0000","last_clean":"2024-04-25T01:18:19.428367+0000","last_became_active":"2024-04-25T01:18:19.424002+0000","last_became_peered":"2024-04-25T01:18:19.424002+0000","last_unstale":"2024-04-25T01:18:19.428367+0000","last_undegraded":"2024-04-25T01:18:19.428367+0000","last_fullsized":"2024-04-25T01:18:19.428367+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:47:50.829942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"42'7","reported_seq":35,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.404294+0000","last_change":"2024-04-25T01:18:03.256275+0000","last_active":"2024-04-25T01:18:19.404294+0000","last_peered":"2024-04-25T01:18:19.404294+0000","last_clean":"2024-04-25T01:18:19.404294+0000","last_became_active":"2024-04-25T01:18:03.255950+0000","last_became_peered":"2024-04-25T01:18:03.255950+0000","last_unstale":"2024-04-25T01:18:19.404294+0000","last_undegraded":"2024-04-25T01:18:19.404294+0000","last_fullsized":"2024-04-25T01:18:19.404294+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:01:09.052002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":30,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.422082+0000","last_change":"2024-04-25T01:18:02.269077+0000","last_active":"2024-04-25T01:18:19.422082+0000","last_peered":"2024-04-25T01:18:19.422082+0000","last_clean":"2024-04-25T01:18:19.422082+0000","last_became_active":"2024-04-25T01:18:02.268795+0000","last_became_peered":"2024-04-25T01:18:02.268795+0000","last_unstale":"2024-04-25T01:18:19.422082+0000","last_undegraded":"2024-04-25T01:18:19.422082+0000","last_fullsized":"2024-04-25T01:18:19.422082+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:16.123119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.423340+0000","last_change":"2024-04-25T01:18:19.410158+0000","last_active":"2024-04-25T01:18:19.423340+0000","last_peered":"2024-04-25T01:18:19.423340+0000","last_clean":"2024-04-25T01:18:19.423340+0000","last_became_active":"2024-04-25T01:18:19.409353+0000","last_became_peered":"2024-04-25T01:18:19.409353+0000","last_unstale":"2024-04-25T01:18:19.423340+0000","last_undegraded":"2024-04-25T01:18:19.423340+0000","last_fullsized":"2024-04-25T01:18:19.423340+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:18.434973+0000","last_clean_scrub_stamp":"2024-04-25T01:18:18.434973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:31.688068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"42'8","reported_seq":38,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.411955+0000","last_change":"2024-04-25T01:18:03.253285+0000","last_active":"2024-04-25T01:18:19.411955+0000","last_peered":"2024-04-25T01:18:19.411955+0000","last_clean":"2024-04-25T01:18:19.411955+0000","last_became_active":"2024-04-25T01:18:03.252973+0000","last_became_peered":"2024-04-25T01:18:03.252973+0000","last_unstale":"2024-04-25T01:18:19.411955+0000","last_undegraded":"2024-04-25T01:18:19.411955+0000","last_fullsized":"2024-04-25T01:18:19.411955+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:02.212637+0000","last_clean_scrub_stamp":"2024-04-25T01:18:02.212637+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:58:23.459413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":29,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.404431+0000","last_change":"2024-04-25T01:18:02.267166+0000","last_active":"2024-04-25T01:18:19.404431+0000","last_peered":"2024-04-25T01:18:19.404431+0000","last_clean":"2024-04-25T01:18:19.404431+0000","last_became_active":"2024-04-25T01:18:02.266866+0000","last_became_peered":"2024-04-25T01:18:02.266866+0000","last_unstale":"2024-04-25T01:18:19.404431+0000","last_undegraded":"2024-04-25T01:18:19.404431+0000","last_fullsized":"2024-04-25T01:18:19.404431+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:18:01.197046+0000","last_clean_scrub_stamp":"2024-04-25T01:18:01.197046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:20:28.341160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":220,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-25T01:18:19.394026+0000","last_change":"2024-04-25T01:16:17.466569+0000","last_active":"2024-04-25T01:18:19.394026+0000","last_peered":"2024-04-25T01:18:19.394026+0000","last_clean":"2024-04-25T01:18:19.394026+0000","last_became_active":"2024-04-25T01:16:17.465687+0000","last_became_peered":"2024-04-25T01:16:17.465687+0000","last_unstale":"2024-04-25T01:18:19.394026+0000","last_undegraded":"2024-04-25T01:18:19.394026+0000","last_fullsized":"2024-04-25T01:18:19.394026+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:16.440559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:16.440559+0000","last_clean_scrub_stamp":"2024-04-25T01:16:16.440559+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:18:17.493555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":6291456,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12582912,"data_stored":12582912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":504,"ondisk_log_size":504,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":96,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542172,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":257136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542171,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":557056,"data_stored":249362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542172,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36416,"kb_used_data":5232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335424,"statfs":{"total":96636764160,"available":96599474176,"internally_reserved":0,"allocated":5357568,"data_stored":5041808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542171,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":557056,"data_stored":249362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542171,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33720,"kb_used_data":6700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338120,"statfs":{"total":96636764160,"available":96602234880,"internally_reserved":0,"allocated":6860800,"data_stored":6548592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542173,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":257136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542171,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28148,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343692,"statfs":{"total":96636764160,"available":96607940608,"internally_reserved":0,"allocated":1150976,"data_stored":839730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542172,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29620,"kb_used_data":2592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342220,"statfs":{"total":96636764160,"available":96606433280,"internally_reserved":0,"allocated":2654208,"data_stored":2346514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2097152,"data_stored":2097152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6291456,"data_stored":6291456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:18:24.360 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T01:18:24.360 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -s 2024-04-25T01:18:24.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.612+0000 7f61ce400640 1 Processor -- start 2024-04-25T01:18:24.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.612+0000 7f61ce400640 1 -- start start 2024-04-25T01:18:24.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.613+0000 7f61ce400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f61c80d2820 0x7f61c80d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:24.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.613+0000 7f61ce400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f61c80d1c70 0x7f61c80b3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:24.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.613+0000 7f61ce400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61c80d1190 con 0x7f61c815df80 2024-04-25T01:18:24.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.613+0000 7f61ce400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f61c80d1750 con 0x7f61c80d2820 2024-04-25T01:18:24.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.613+0000 7f61ce400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f61c80b3710 con 0x7f61c80d1c70 2024-04-25T01:18:24.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.613+0000 7f61c7e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f61c80d2820 0x7f61c80d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:24.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.614+0000 7f61cd000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f61c815df80 0x7f61c80d2110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53450/0 (socket says 172.21.15.90:53450) 2024-04-25T01:18:24.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.614+0000 7f61cd000640 1 -- 172.21.15.90:0/142198509 learned_addr learned my addr 172.21.15.90:0/142198509 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:24.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.614+0000 7f61cda00640 1 --2- 172.21.15.90:0/142198509 >> v2:172.21.15.103:3300/0 conn(0x7f61c80d1c70 0x7f61c80b3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:24.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.614+0000 7f61c7e00640 1 -- 172.21.15.90:0/142198509 >> v2:172.21.15.103:3300/0 conn(0x7f61c80d1c70 msgr2=0x7f61c80b3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:24.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.614+0000 7f61c7e00640 1 --2- 172.21.15.90:0/142198509 >> v2:172.21.15.103:3300/0 conn(0x7f61c80d1c70 0x7f61c80b3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:24.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.614+0000 7f61c7e00640 1 -- 172.21.15.90:0/142198509 >> v1:172.21.15.90:6789/0 conn(0x7f61c815df80 legacy=0x7f61c80d2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:24.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.614+0000 7f61c7e00640 1 -- 172.21.15.90:0/142198509 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61c80d1470 con 0x7f61c80d2820 2024-04-25T01:18:24.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.614+0000 7f61c7400640 1 -- 172.21.15.90:0/142198509 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3390416241 0 0) 0x7f61c80d1190 con 0x7f61c815df80 2024-04-25T01:18:24.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.615+0000 7f61c7e00640 1 --2- 172.21.15.90:0/142198509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61c80d2820 0x7f61c80d4920 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f61c80d34f0 tx=0x7f61b402a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9befd336cc6a9e65 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:24.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.616+0000 7f61c7400640 1 -- 172.21.15.90:0/142198509 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b4037070 con 0x7f61c80d2820 2024-04-25T01:18:24.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.616+0000 7f61c7400640 1 -- 172.21.15.90:0/142198509 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61b402add0 con 0x7f61c80d2820 2024-04-25T01:18:24.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.616+0000 7f61c7400640 1 -- 172.21.15.90:0/142198509 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b4032660 con 0x7f61c80d2820 2024-04-25T01:18:24.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.616+0000 7f61ce400640 1 -- 172.21.15.90:0/142198509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61c80d2820 msgr2=0x7f61c80d4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:24.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.616+0000 7f61ce400640 1 --2- 172.21.15.90:0/142198509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61c80d2820 0x7f61c80d4920 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f61c80d34f0 tx=0x7f61b402a080 comp rx=0 tx=0).stop 2024-04-25T01:18:24.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.617+0000 7f61ce400640 1 -- 172.21.15.90:0/142198509 shutdown_connections 2024-04-25T01:18:24.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.617+0000 7f61ce400640 1 --2- 172.21.15.90:0/142198509 >> v2:172.21.15.103:3300/0 conn(0x7f61c80d1c70 0x7f61c80b3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:24.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.617+0000 7f61ce400640 1 --2- 172.21.15.90:0/142198509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61c80d2820 0x7f61c80d4920 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:24.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.617+0000 7f61ce400640 1 -- 172.21.15.90:0/142198509 >> 172.21.15.90:0/142198509 conn(0x7f61c80d3e00 msgr2=0x7f61c80d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:24.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.617+0000 7f61ce400640 1 -- 172.21.15.90:0/142198509 shutdown_connections 2024-04-25T01:18:24.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.617+0000 7f61ce400640 1 -- 172.21.15.90:0/142198509 wait complete. 2024-04-25T01:18:24.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.618+0000 7f61ce400640 1 Processor -- start 2024-04-25T01:18:24.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.619+0000 7f61ce400640 1 -- start start 2024-04-25T01:18:24.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.619+0000 7f61ce400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61c80d1c70 0x7f61c815b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:24.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.619+0000 7f61ce400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f61c80d2820 0x7f61c80ccd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:24.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.619+0000 7f61ce400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61c815df80 0x7f61c80b8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:24.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.619+0000 7f61ce400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f61c80d0e40 con 0x7f61c815df80 2024-04-25T01:18:24.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.619+0000 7f61ce400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f61c80b33a0 con 0x7f61c80d1c70 2024-04-25T01:18:24.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.619+0000 7f61ce400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f61c815e360 con 0x7f61c80d2820 2024-04-25T01:18:24.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.620+0000 7f61cda00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61c815df80 0x7f61c80b8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:24.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.620+0000 7f61cda00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61c815df80 0x7f61c80b8100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41208/0 (socket says 172.21.15.90:41208) 2024-04-25T01:18:24.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.620+0000 7f61cda00640 1 -- 172.21.15.90:0/3023980041 learned_addr learned my addr 172.21.15.90:0/3023980041 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.620+0000 7f61cd000640 1 --2- 172.21.15.90:0/3023980041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61c80d1c70 0x7f61c815b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.620+0000 7f61cda00640 1 -- 172.21.15.90:0/3023980041 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f61c80d2820 msgr2=0x7f61c80ccd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.620+0000 7f61c7e00640 1 --2- 172.21.15.90:0/3023980041 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f61c80d2820 0x7f61c80ccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.620+0000 7f61cda00640 1 --2- 172.21.15.90:0/3023980041 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f61c80d2820 0x7f61c80ccd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.620+0000 7f61cda00640 1 -- 172.21.15.90:0/3023980041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61c80d1c70 msgr2=0x7f61c815b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.620+0000 7f61cda00640 1 --2- 172.21.15.90:0/3023980041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61c80d1c70 0x7f61c815b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.620+0000 7f61cda00640 1 -- 172.21.15.90:0/3023980041 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61c80cbcc0 con 0x7f61c815df80 2024-04-25T01:18:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.620+0000 7f61cd000640 1 --2- 172.21.15.90:0/3023980041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61c80d1c70 0x7f61c815b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.621+0000 7f61cda00640 1 --2- 172.21.15.90:0/3023980041 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61c815df80 0x7f61c80b8100 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f61c0007e40 tx=0x7f61c000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.621+0000 7f61c4c00640 1 -- 172.21.15.90:0/3023980041 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c0014620 con 0x7f61c815df80 2024-04-25T01:18:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.621+0000 7f61c4c00640 1 -- 172.21.15.90:0/3023980041 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61c0019070 con 0x7f61c815df80 2024-04-25T01:18:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.621+0000 7f61ce400640 1 -- 172.21.15.90:0/3023980041 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61c80cbfa0 con 0x7f61c815df80 2024-04-25T01:18:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.621+0000 7f61c4c00640 1 -- 172.21.15.90:0/3023980041 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c0014a20 con 0x7f61c815df80 2024-04-25T01:18:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.621+0000 7f61ce400640 1 -- 172.21.15.90:0/3023980041 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61c80cc560 con 0x7f61c815df80 2024-04-25T01:18:24.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.622+0000 7f61ce400640 1 -- 172.21.15.90:0/3023980041 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61c815ecc0 con 0x7f61c815df80 2024-04-25T01:18:24.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.623+0000 7f61c4c00640 1 -- 172.21.15.90:0/3023980041 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f61c0014bc0 con 0x7f61c815df80 2024-04-25T01:18:24.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.624+0000 7f61c4c00640 1 --2- 172.21.15.90:0/3023980041 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f61a806be30 0x7f61a806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:24.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.624+0000 7f61c4c00640 1 -- 172.21.15.90:0/3023980041 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f61c008bac0 con 0x7f61c815df80 2024-04-25T01:18:24.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.624+0000 7f61cd000640 1 --2- 172.21.15.90:0/3023980041 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f61a806be30 0x7f61a806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:24.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.625+0000 7f61cd000640 1 --2- 172.21.15.90:0/3023980041 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f61a806be30 0x7f61a806e2f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f61bc000f60 tx=0x7f61bc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:24.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.629+0000 7f61c4c00640 1 -- 172.21.15.90:0/3023980041 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f61c815ecc0 con 0x7f61c815df80 2024-04-25T01:18:24.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.899+0000 7f61ce400640 1 -- 172.21.15.90:0/3023980041 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f61c80b4870 con 0x7f61c815df80 2024-04-25T01:18:24.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.900+0000 7f61c4c00640 1 -- 172.21.15.90:0/3023980041 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+611 (secure 0 0 0) 0x7f61c005a120 con 0x7f61c815df80 2024-04-25T01:18:24.883 INFO:teuthology.orchestra.run.smithi090.stdout: cluster: 2024-04-25T01:18:24.883 INFO:teuthology.orchestra.run.smithi090.stdout: id: e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21 2024-04-25T01:18:24.883 INFO:teuthology.orchestra.run.smithi090.stdout: health: HEALTH_ERR 2024-04-25T01:18:24.883 INFO:teuthology.orchestra.run.smithi090.stdout: 2 filesystems are degraded 2024-04-25T01:18:24.883 INFO:teuthology.orchestra.run.smithi090.stdout: 2 filesystems are offline 2024-04-25T01:18:24.883 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:24.884 INFO:teuthology.orchestra.run.smithi090.stdout: services: 2024-04-25T01:18:24.884 INFO:teuthology.orchestra.run.smithi090.stdout: mon: 3 daemons, quorum a,b,c (age 2m) 2024-04-25T01:18:24.884 INFO:teuthology.orchestra.run.smithi090.stdout: mgr: y(active, since 2m), standbys: x 2024-04-25T01:18:24.884 INFO:teuthology.orchestra.run.smithi090.stdout: mds: 0/2 daemons up (2 failed), 4 standby 2024-04-25T01:18:24.884 INFO:teuthology.orchestra.run.smithi090.stdout: osd: 8 osds: 8 up (since 2m), 8 in (since 2m) 2024-04-25T01:18:24.884 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:24.884 INFO:teuthology.orchestra.run.smithi090.stdout: data: 2024-04-25T01:18:24.884 INFO:teuthology.orchestra.run.smithi090.stdout: volumes: 0/2 healthy, 2 failed 2024-04-25T01:18:24.884 INFO:teuthology.orchestra.run.smithi090.stdout: pools: 4 pools, 129 pgs 2024-04-25T01:18:24.884 INFO:teuthology.orchestra.run.smithi090.stdout: objects: 4 objects, 6.6 MiB 2024-04-25T01:18:24.884 INFO:teuthology.orchestra.run.smithi090.stdout: usage: 233 MiB used, 720 GiB / 720 GiB avail 2024-04-25T01:18:24.884 INFO:teuthology.orchestra.run.smithi090.stdout: pgs: 129 active+clean 2024-04-25T01:18:24.884 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:24.884 INFO:teuthology.orchestra.run.smithi090.stdout: io: 2024-04-25T01:18:24.884 INFO:teuthology.orchestra.run.smithi090.stdout: client: 0 B/s rd, 1.7 KiB/s wr, 0 op/s rd, 7 op/s wr 2024-04-25T01:18:24.885 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:24.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.902+0000 7f61ce400640 1 -- 172.21.15.90:0/3023980041 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f61a806be30 msgr2=0x7f61a806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:24.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.902+0000 7f61ce400640 1 --2- 172.21.15.90:0/3023980041 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f61a806be30 0x7f61a806e2f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f61bc000f60 tx=0x7f61bc009040 comp rx=0 tx=0).stop 2024-04-25T01:18:24.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.902+0000 7f61ce400640 1 -- 172.21.15.90:0/3023980041 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61c815df80 msgr2=0x7f61c80b8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:24.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.902+0000 7f61ce400640 1 --2- 172.21.15.90:0/3023980041 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61c815df80 0x7f61c80b8100 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f61c0007e40 tx=0x7f61c000e4b0 comp rx=0 tx=0).stop 2024-04-25T01:18:24.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.903+0000 7f61ce400640 1 -- 172.21.15.90:0/3023980041 shutdown_connections 2024-04-25T01:18:24.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.903+0000 7f61ce400640 1 --2- 172.21.15.90:0/3023980041 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f61a806be30 0x7f61a806e2f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:24.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.903+0000 7f61ce400640 1 --2- 172.21.15.90:0/3023980041 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61c815df80 0x7f61c80b8100 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:24.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.903+0000 7f61ce400640 1 --2- 172.21.15.90:0/3023980041 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f61c80d2820 0x7f61c80ccd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:24.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.903+0000 7f61ce400640 1 --2- 172.21.15.90:0/3023980041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61c80d1c70 0x7f61c815b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:24.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.903+0000 7f61ce400640 1 -- 172.21.15.90:0/3023980041 >> 172.21.15.90:0/3023980041 conn(0x7f61c80d3e00 msgr2=0x7f61c815d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:24.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.903+0000 7f61ce400640 1 -- 172.21.15.90:0/3023980041 shutdown_connections 2024-04-25T01:18:24.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:24.903+0000 7f61ce400640 1 -- 172.21.15.90:0/3023980041 wait complete. 2024-04-25T01:18:24.907 DEBUG:teuthology.orchestra.run.smithi090:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 init --force-init --filesystem recovery_fs --alternate-pool recovery_fs_metadata 2024-04-25T01:18:25.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.025+0000 7f003fd22740 1 -- start start 2024-04-25T01:18:25.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.025+0000 7f003fd22740 1 Processor -- start 2024-04-25T01:18:25.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.026+0000 7f003fd22740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c9ea9ac00 0x555c9eb3e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.026+0000 7f003fd22740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555c9ea9b800 0x555c9eb3eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.026+0000 7f003fd22740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c9ea9bc00 0x555c9eb3f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.026+0000 7f003fd22740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x555c9f87a180 con 0x555c9ea9ac00 2024-04-25T01:18:25.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.026+0000 7f003fd22740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555c9f87a300 con 0x555c9ea9bc00 2024-04-25T01:18:25.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.026+0000 7f003fd22740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x555c9f87a480 con 0x555c9ea9b800 2024-04-25T01:18:25.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.027+0000 7f003d600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c9ea9ac00 0x555c9eb3e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.027+0000 7f003d600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c9ea9ac00 0x555c9eb3e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47530/0 (socket says 172.21.15.90:47530) 2024-04-25T01:18:25.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.027+0000 7f003d600640 1 -- 172.21.15.90:0/818757857 learned_addr learned my addr 172.21.15.90:0/818757857 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:25.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.027+0000 7f003cc00640 1 --2- 172.21.15.90:0/818757857 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555c9ea9b800 0x555c9eb3eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.027+0000 7f003d600640 1 -- 172.21.15.90:0/818757857 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555c9ea9b800 msgr2=0x555c9eb3eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:25.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.027+0000 7f003d600640 1 --2- 172.21.15.90:0/818757857 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555c9ea9b800 0x555c9eb3eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.027+0000 7f003d600640 1 -- 172.21.15.90:0/818757857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c9ea9bc00 msgr2=0x555c9eb3f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:25.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.027+0000 7f003d600640 1 --2- 172.21.15.90:0/818757857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c9ea9bc00 0x555c9eb3f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.027+0000 7f003d600640 1 -- 172.21.15.90:0/818757857 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c9eac7680 con 0x555c9ea9ac00 2024-04-25T01:18:25.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.027+0000 7f003d600640 1 --2- 172.21.15.90:0/818757857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c9ea9ac00 0x555c9eb3e580 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x555c9f8af200 tx=0x555c9f832b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5da8303c238676b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.028+0000 7f003a400640 1 -- 172.21.15.90:0/818757857 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x555c9eac09c0 con 0x555c9ea9ac00 2024-04-25T01:18:25.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.028+0000 7f003a400640 1 -- 172.21.15.90:0/818757857 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x555c9eae4e00 con 0x555c9ea9ac00 2024-04-25T01:18:25.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.028+0000 7f003a400640 1 -- 172.21.15.90:0/818757857 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x555c9eac0680 con 0x555c9ea9ac00 2024-04-25T01:18:25.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.028+0000 7f003fd22740 1 -- 172.21.15.90:0/818757857 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c9eac7860 con 0x555c9ea9ac00 2024-04-25T01:18:25.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.028+0000 7f003fd22740 1 -- 172.21.15.90:0/818757857 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) -- 0x555c9eac72c0 con 0x555c9ea9ac00 2024-04-25T01:18:25.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.028+0000 7f003fd22740 4 waiting for MDS map... 2024-04-25T01:18:25.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.028+0000 7f003a400640 1 -- 172.21.15.90:0/818757857 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x555c9f968200 con 0x555c9ea9ac00 2024-04-25T01:18:25.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.028+0000 7f003a400640 1 -- 172.21.15.90:0/818757857 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 23) ==== 4135+0+0 (secure 0 0 0) 0x555c9eaf3180 con 0x555c9ea9ac00 2024-04-25T01:18:25.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.028+0000 7f003fd22740 4 Got MDS map 23 2024-04-25T01:18:25.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.028+0000 7f003fd22740 4 datascan.main: Using metadata pool output 2024-04-25T01:18:25.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.028+0000 7f003fd22740 4 datascan.main: connecting to RADOS... 2024-04-25T01:18:25.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.029+0000 7f003fd22740 1 Processor -- start 2024-04-25T01:18:25.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.029+0000 7f003fd22740 1 -- start start 2024-04-25T01:18:25.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.030+0000 7f003fd22740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c9f977000 0x555c9eb3f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.030+0000 7f003fd22740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c9f977400 0x555c9eb3fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.030+0000 7f003fd22740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555c9f977800 0x555c9eb40100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.030+0000 7f003fd22740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x555c9f87a600 con 0x555c9f977400 2024-04-25T01:18:25.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.030+0000 7f003fd22740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555c9f87a780 con 0x555c9f977000 2024-04-25T01:18:25.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.030+0000 7f003fd22740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x555c9f87a900 con 0x555c9f977800 2024-04-25T01:18:25.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.030+0000 7f003d600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c9f977400 0x555c9eb3fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.030+0000 7f003d600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c9f977400 0x555c9eb3fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47546/0 (socket says 172.21.15.90:47546) 2024-04-25T01:18:25.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.030+0000 7f003d600640 1 -- 172.21.15.90:0/153295919 learned_addr learned my addr 172.21.15.90:0/153295919 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:25.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.030+0000 7f003d600640 1 -- 172.21.15.90:0/153295919 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555c9f977800 msgr2=0x555c9eb40100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:25.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.031+0000 7f003d600640 1 --2- 172.21.15.90:0/153295919 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555c9f977800 0x555c9eb40100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.031+0000 7f003d600640 1 -- 172.21.15.90:0/153295919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c9f977000 msgr2=0x555c9eb3f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:25.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.031+0000 7f003d600640 1 --2- 172.21.15.90:0/153295919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c9f977000 0x555c9eb3f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.031+0000 7f003d600640 1 -- 172.21.15.90:0/153295919 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c9eac74a0 con 0x555c9f977400 2024-04-25T01:18:25.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.031+0000 7f003d600640 1 --2- 172.21.15.90:0/153295919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c9f977400 0x555c9eb3fb80 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x555c9f870d80 tx=0x555c9eb48d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6d624cacebcfa5b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.031+0000 7f0037c00640 1 -- 172.21.15.90:0/153295919 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x555c9eac0b60 con 0x555c9f977400 2024-04-25T01:18:25.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.031+0000 7f0037c00640 1 -- 172.21.15.90:0/153295919 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x555c9eae4c40 con 0x555c9f977400 2024-04-25T01:18:25.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.031+0000 7f0037c00640 1 -- 172.21.15.90:0/153295919 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x555c9eac0820 con 0x555c9f977400 2024-04-25T01:18:25.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.031+0000 7f003fd22740 1 -- 172.21.15.90:0/153295919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c9f977400 msgr2=0x555c9eb3fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.031+0000 7f003fd22740 1 --2- 172.21.15.90:0/153295919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c9f977400 0x555c9eb3fb80 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x555c9f870d80 tx=0x555c9eb48d80 comp rx=0 tx=0).stop 2024-04-25T01:18:25.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.031+0000 7f003fd22740 1 -- 172.21.15.90:0/153295919 shutdown_connections 2024-04-25T01:18:25.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.031+0000 7f003fd22740 1 --2- 172.21.15.90:0/153295919 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555c9f977800 0x555c9eb40100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.032+0000 7f003fd22740 1 --2- 172.21.15.90:0/153295919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c9f977400 0x555c9eb3fb80 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.032+0000 7f003fd22740 1 --2- 172.21.15.90:0/153295919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c9f977000 0x555c9eb3f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.032+0000 7f003fd22740 1 -- 172.21.15.90:0/153295919 >> 172.21.15.90:0/153295919 conn(0x555c9f976c00 msgr2=0x555c9f83b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:25.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.032+0000 7f003fd22740 1 -- 172.21.15.90:0/153295919 shutdown_connections 2024-04-25T01:18:25.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.032+0000 7f003fd22740 1 -- 172.21.15.90:0/153295919 wait complete. 2024-04-25T01:18:25.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.032+0000 7f003fd22740 1 Processor -- start 2024-04-25T01:18:25.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.032+0000 7f003fd22740 1 -- start start 2024-04-25T01:18:25.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.033+0000 7f003fd22740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c9f977000 0x555c9eb41180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.033+0000 7f003fd22740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555c9f977400 0x555c9eb40c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.033+0000 7f003fd22740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c9f977800 0x555c9eb40680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.033+0000 7f003fd22740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x555c9f87bc80 con 0x555c9f977800 2024-04-25T01:18:25.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.033+0000 7f003fd22740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555c9f87bb00 con 0x555c9f977000 2024-04-25T01:18:25.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.033+0000 7f003fd22740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x555c9f87b980 con 0x555c9f977400 2024-04-25T01:18:25.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.033+0000 7f003cc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c9f977800 0x555c9eb40680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.034+0000 7f003cc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c9f977800 0x555c9eb40680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47556/0 (socket says 172.21.15.90:47556) 2024-04-25T01:18:25.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.034+0000 7f003cc00640 1 -- 172.21.15.90:0/1098595506 learned_addr learned my addr 172.21.15.90:0/1098595506 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:25.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.034+0000 7f003e000640 1 --2- 172.21.15.90:0/1098595506 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c9f977000 0x555c9eb41180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.034+0000 7f003d600640 1 --2- 172.21.15.90:0/1098595506 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555c9f977400 0x555c9eb40c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.034+0000 7f003cc00640 1 -- 172.21.15.90:0/1098595506 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555c9f977400 msgr2=0x555c9eb40c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.034+0000 7f003cc00640 1 --2- 172.21.15.90:0/1098595506 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555c9f977400 0x555c9eb40c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.034+0000 7f003cc00640 1 -- 172.21.15.90:0/1098595506 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c9f977000 msgr2=0x555c9eb41180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.034+0000 7f003cc00640 1 --2- 172.21.15.90:0/1098595506 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c9f977000 0x555c9eb41180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.034+0000 7f003cc00640 1 -- 172.21.15.90:0/1098595506 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c9eac70e0 con 0x555c9f977800 2024-04-25T01:18:25.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.034+0000 7f003d600640 1 --2- 172.21.15.90:0/1098595506 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555c9f977400 0x555c9eb40c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:25.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.034+0000 7f003cc00640 1 --2- 172.21.15.90:0/1098595506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c9f977800 0x555c9eb40680 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x555c9f871b60 tx=0x555c9f979920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.034+0000 7f0035e00640 1 -- 172.21.15.90:0/1098595506 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x555c9eac0b60 con 0x555c9f977800 2024-04-25T01:18:25.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.035+0000 7f0035e00640 1 -- 172.21.15.90:0/1098595506 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x555c9eae4a80 con 0x555c9f977800 2024-04-25T01:18:25.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.035+0000 7f0035e00640 1 -- 172.21.15.90:0/1098595506 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x555c9eac0820 con 0x555c9f977800 2024-04-25T01:18:25.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.035+0000 7f003fd22740 1 -- 172.21.15.90:0/1098595506 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x555c9eac7a40 con 0x555c9f977800 2024-04-25T01:18:25.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.035+0000 7f003fd22740 1 -- 172.21.15.90:0/1098595506 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c9eac63c0 con 0x555c9f977800 2024-04-25T01:18:25.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.035+0000 7f003fd22740 4 datascan.init: forcing metadata pool 'recovery_fs_metadata' 2024-04-25T01:18:25.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.036+0000 7f0035e00640 1 -- 172.21.15.90:0/1098595506 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x555c9f9b8900 con 0x555c9f977800 2024-04-25T01:18:25.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.037+0000 7f0035e00640 1 --2- 172.21.15.90:0/1098595506 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x555c9f9d0c00 0x555c9eb41700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.037+0000 7f0035e00640 1 -- 172.21.15.90:0/1098595506 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x555c9f9c4e00 con 0x555c9f977800 2024-04-25T01:18:25.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.037+0000 7f003fd22740 1 --2- 172.21.15.90:0/1098595506 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x555c9f9d1000 0x555c9eb40100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.037+0000 7f003fd22740 1 -- 172.21.15.90:0/1098595506 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- osd_op(unknown.0.0:1 7.14 7:287034c4:::1.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50) -- 0x555c9f9d1400 con 0x555c9f9d1000 2024-04-25T01:18:25.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.037+0000 7f003e000640 1 --2- 172.21.15.90:0/1098595506 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x555c9f9d0c00 0x555c9eb41700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.038+0000 7f003d600640 1 --2- 172.21.15.90:0/1098595506 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x555c9f9d1000 0x555c9eb40100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.038+0000 7f003e000640 1 --2- 172.21.15.90:0/1098595506 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x555c9f9d0c00 0x555c9eb41700 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x555c9f834ff0 tx=0x555c9ed90de0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.039+0000 7f003d600640 1 --2- 172.21.15.90:0/1098595506 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x555c9f9d1000 0x555c9eb40100 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.040+0000 7f003d600640 1 -- 172.21.15.90:0/1098595506 <== osd.5 v2:172.21.15.103:6801/1348846168 1 ==== osd_op_reply(1 1.00000000.inode [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 160+0+0 (crc 0 0 0) 0x555c9f82a000 con 0x555c9f9d1000 2024-04-25T01:18:25.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.040+0000 7f003fd22740 1 -- 172.21.15.90:0/1098595506 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- osd_op(unknown.0.0:2 7.14 7:287034c4:::1.00000000.inode:head [writefull 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) -- 0x555c9f9d1800 con 0x555c9f9d1000 2024-04-25T01:18:25.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.044+0000 7f003d600640 1 -- 172.21.15.90:0/1098595506 <== osd.5 v2:172.21.15.103:6801/1348846168 2 ==== osd_op_reply(2 1.00000000.inode [writefull 0~582] v50'1 uv1 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x555c9f82a000 con 0x555c9f9d1000 2024-04-25T01:18:25.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.044+0000 7f003fd22740 1 --2- 172.21.15.90:0/1098595506 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x555c9f9d1c00 0x555c9eb3fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.044+0000 7f003fd22740 1 -- 172.21.15.90:0/1098595506 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:3 7.1f 7:fe07bba1:::100.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50) -- 0x555c9ee16c00 con 0x555c9f9d1c00 2024-04-25T01:18:25.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.044+0000 7f003e000640 1 --2- 172.21.15.90:0/1098595506 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x555c9f9d1c00 0x555c9eb3fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.044+0000 7f003e000640 1 --2- 172.21.15.90:0/1098595506 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x555c9f9d1c00 0x555c9eb3fb80 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.045+0000 7f003e000640 1 -- 172.21.15.90:0/1098595506 <== osd.0 v2:172.21.15.90:6816/2625481430 1 ==== osd_op_reply(3 100.00000000.inode [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 162+0+0 (crc 0 0 0) 0x555c9f82a500 con 0x555c9f9d1c00 2024-04-25T01:18:25.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.045+0000 7f003fd22740 1 -- 172.21.15.90:0/1098595506 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:4 7.1f 7:fe07bba1:::100.00000000.inode:head [writefull 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) -- 0x555c9ee17000 con 0x555c9f9d1c00 2024-04-25T01:18:25.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.049+0000 7f003e000640 1 -- 172.21.15.90:0/1098595506 <== osd.0 v2:172.21.15.90:6816/2625481430 2 ==== osd_op_reply(4 100.00000000.inode [writefull 0~582] v50'1 uv1 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x555c9f82a500 con 0x555c9f9d1c00 2024-04-25T01:18:25.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.049+0000 7f003fd22740 1 --2- 172.21.15.90:0/1098595506 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x555c9ee17400 0x555c9eb3f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.049+0000 7f003fd22740 1 -- 172.21.15.90:0/1098595506 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:5 7.13 7:cd5a64a3:::100.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50) -- 0x555c9ee17800 con 0x555c9ee17400 2024-04-25T01:18:25.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.049+0000 7f003d600640 1 --2- 172.21.15.90:0/1098595506 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x555c9ee17400 0x555c9eb3f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.050+0000 7f003d600640 1 --2- 172.21.15.90:0/1098595506 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x555c9ee17400 0x555c9eb3f600 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.051+0000 7f003d600640 1 -- 172.21.15.90:0/1098595506 <== osd.7 v2:172.21.15.103:6800/1145679154 1 ==== osd_op_reply(5 100.00000000 [omap-get-header] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x555c9f82a000 con 0x555c9ee17400 2024-04-25T01:18:25.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.051+0000 7f003fd22740 10 datascan.find_or_create_dirfrag: read_version = 0 2024-04-25T01:18:25.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.051+0000 7f003fd22740 1 -- 172.21.15.90:0/1098595506 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:6 7.13 7:cd5a64a3:::100.00000000:head [create,omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) -- 0x555c9ee17c00 con 0x555c9ee17400 2024-04-25T01:18:25.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.054+0000 7f003d600640 1 -- 172.21.15.90:0/1098595506 <== osd.7 v2:172.21.15.103:6800/1145679154 2 ==== osd_op_reply(6 100.00000000 [create,omap-set-header] v50'1 uv1 ondisk = 0) ==== 198+0+0 (crc 0 0 0) 0x555c9f82a000 con 0x555c9ee17400 2024-04-25T01:18:25.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.054+0000 7f003fd22740 10 datascan.find_or_create_dirfrag: Created dirfrag: 0x0x100 2024-04-25T01:18:25.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.054+0000 7f003fd22740 1 -- 172.21.15.90:0/1098595506 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x555c9f9d1c00 msgr2=0x555c9eb3fb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.054+0000 7f003fd22740 1 --2- 172.21.15.90:0/1098595506 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x555c9f9d1c00 0x555c9eb3fb80 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.054+0000 7f003fd22740 1 -- 172.21.15.90:0/1098595506 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x555c9f9d1000 msgr2=0x555c9eb40100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.054+0000 7f003fd22740 1 --2- 172.21.15.90:0/1098595506 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x555c9f9d1000 0x555c9eb40100 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.054+0000 7f003fd22740 1 -- 172.21.15.90:0/1098595506 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x555c9ee17400 msgr2=0x555c9eb3f600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.054+0000 7f003fd22740 1 --2- 172.21.15.90:0/1098595506 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x555c9ee17400 0x555c9eb3f600 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.055+0000 7f003cc00640 1 -- 172.21.15.90:0/1098595506 reap_dead start 2024-04-25T01:18:25.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.055+0000 7f003fd22740 1 -- 172.21.15.90:0/1098595506 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x555c9f9d0c00 msgr2=0x555c9eb41700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.055+0000 7f003fd22740 1 --2- 172.21.15.90:0/1098595506 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x555c9f9d0c00 0x555c9eb41700 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x555c9f834ff0 tx=0x555c9ed90de0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.055+0000 7f003fd22740 1 -- 172.21.15.90:0/1098595506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c9f977800 msgr2=0x555c9eb40680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.055+0000 7f003fd22740 1 --2- 172.21.15.90:0/1098595506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c9f977800 0x555c9eb40680 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x555c9f871b60 tx=0x555c9f979920 comp rx=0 tx=0).stop 2024-04-25T01:18:25.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.055+0000 7f003fd22740 1 -- 172.21.15.90:0/1098595506 shutdown_connections 2024-04-25T01:18:25.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.055+0000 7f003fd22740 1 --2- 172.21.15.90:0/1098595506 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x555c9f9d0c00 0x555c9eb41700 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.055+0000 7f003fd22740 1 --2- 172.21.15.90:0/1098595506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c9f977800 0x555c9eb40680 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.055+0000 7f003fd22740 1 -- 172.21.15.90:0/1098595506 >> 172.21.15.90:0/1098595506 conn(0x555c9f976c00 msgr2=0x555c9f83b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:25.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.055+0000 7f003fd22740 1 -- 172.21.15.90:0/1098595506 shutdown_connections 2024-04-25T01:18:25.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.055+0000 7f003fd22740 1 -- 172.21.15.90:0/1098595506 wait complete. 2024-04-25T01:18:25.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.055+0000 7f003fd22740 1 -- 172.21.15.90:0/818757857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c9ea9ac00 msgr2=0x555c9eb3e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.055+0000 7f003fd22740 1 --2- 172.21.15.90:0/818757857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c9ea9ac00 0x555c9eb3e580 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x555c9f8af200 tx=0x555c9f832b40 comp rx=0 tx=0).stop 2024-04-25T01:18:25.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.056+0000 7f003fd22740 1 -- 172.21.15.90:0/818757857 shutdown_connections 2024-04-25T01:18:25.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.056+0000 7f003fd22740 1 --2- 172.21.15.90:0/818757857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c9ea9bc00 0x555c9eb3f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.056+0000 7f003fd22740 1 --2- 172.21.15.90:0/818757857 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x555c9ea9b800 0x555c9eb3eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.056+0000 7f003fd22740 1 --2- 172.21.15.90:0/818757857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c9ea9ac00 0x555c9eb3e580 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.056+0000 7f003fd22740 1 -- 172.21.15.90:0/818757857 >> 172.21.15.90:0/818757857 conn(0x555c9ea9b000 msgr2=0x555c9f83a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:25.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.056+0000 7f003fd22740 1 -- 172.21.15.90:0/818757857 shutdown_connections 2024-04-25T01:18:25.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.056+0000 7f003fd22740 1 -- 172.21.15.90:0/818757857 wait complete. 2024-04-25T01:18:25.046 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'init', '--force-init', '--filesystem', 'recovery_fs', '--alternate-pool', 'recovery_fs_metadata'] in time 0:00:00.139742, result: 2024-04-25T01:18:25.046 DEBUG:teuthology.orchestra.run.smithi090:> cephfs-table-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 recovery_fs:0 reset session 2024-04-25T01:18:25.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.160+0000 7f1edc11b280 1 -- start start 2024-04-25T01:18:25.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.160+0000 7f1edc11b280 1 Processor -- start 2024-04-25T01:18:25.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.161+0000 7f1edc11b280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558205760c00 0x5582057cc580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.161+0000 7f1edc11b280 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558205761800 0x5582057ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.161+0000 7f1edc11b280 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558205761c00 0x5582057cd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.161+0000 7f1edc11b280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x558206616180 con 0x558205760c00 2024-04-25T01:18:25.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.161+0000 7f1edc11b280 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558206616300 con 0x558205761800 2024-04-25T01:18:25.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.161+0000 7f1edc11b280 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x558206616480 con 0x558205761c00 2024-04-25T01:18:25.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.161+0000 7f1ed9600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558205760c00 0x5582057cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.162+0000 7f1ed9600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558205760c00 0x5582057cc580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47576/0 (socket says 172.21.15.90:47576) 2024-04-25T01:18:25.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.162+0000 7f1ed9600640 1 -- 172.21.15.90:0/1274846577 learned_addr learned my addr 172.21.15.90:0/1274846577 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:25.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.162+0000 7f1ed8c00640 1 --2- 172.21.15.90:0/1274846577 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558205761800 0x5582057ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.162+0000 7f1eda000640 1 --2- 172.21.15.90:0/1274846577 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558205761c00 0x5582057cd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.162+0000 7f1ed9600640 1 -- 172.21.15.90:0/1274846577 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558205761c00 msgr2=0x5582057cd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:25.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.162+0000 7f1ed9600640 1 --2- 172.21.15.90:0/1274846577 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558205761c00 0x5582057cd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.162+0000 7f1ed9600640 1 -- 172.21.15.90:0/1274846577 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558205761800 msgr2=0x5582057ccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:25.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.162+0000 7f1ed9600640 1 --2- 172.21.15.90:0/1274846577 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558205761800 0x5582057ccb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.162+0000 7f1ed9600640 1 -- 172.21.15.90:0/1274846577 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558205794780 con 0x558205760c00 2024-04-25T01:18:25.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.162+0000 7f1ed9600640 1 --2- 172.21.15.90:0/1274846577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558205760c00 0x5582057cc580 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x5582065791d0 tx=0x5582064f4180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=458f49586df4cb08 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.163+0000 7f1ed6400640 1 -- 172.21.15.90:0/1274846577 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x558205786680 con 0x558205760c00 2024-04-25T01:18:25.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.163+0000 7f1ed6400640 1 -- 172.21.15.90:0/1274846577 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55820578ca80 con 0x558205760c00 2024-04-25T01:18:25.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.163+0000 7f1ed6400640 1 -- 172.21.15.90:0/1274846577 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x558205786b60 con 0x558205760c00 2024-04-25T01:18:25.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.163+0000 7f1edc11b280 1 -- 172.21.15.90:0/1274846577 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582057954a0 con 0x558205760c00 2024-04-25T01:18:25.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.163+0000 7f1edc11b280 1 -- 172.21.15.90:0/1274846577 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) -- 0x558205794000 con 0x558205760c00 2024-04-25T01:18:25.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.163+0000 7f1edc11b280 4 waiting for MDS map... 2024-04-25T01:18:25.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.163+0000 7f1ed6400640 1 -- 172.21.15.90:0/1274846577 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x558206630600 con 0x558205760c00 2024-04-25T01:18:25.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.163+0000 7f1ed6400640 1 -- 172.21.15.90:0/1274846577 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 23) ==== 4135+0+0 (secure 0 0 0) 0x5582057bb180 con 0x558205760c00 2024-04-25T01:18:25.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.163+0000 7f1edc11b280 4 Got MDS map 23 2024-04-25T01:18:25.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.163+0000 7f1edc11b280 10 main: main 2024-04-25T01:18:25.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.163+0000 7f1edc11b280 4 main: connecting to RADOS... 2024-04-25T01:18:25.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.164+0000 7f1edc11b280 1 Processor -- start 2024-04-25T01:18:25.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.164+0000 7f1edc11b280 1 -- start start 2024-04-25T01:18:25.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.165+0000 7f1edc11b280 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55820662dc00 0x5582057cd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.165+0000 7f1edc11b280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558206686000 0x5582057cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.165+0000 7f1edc11b280 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558206686400 0x5582057ce100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.165+0000 7f1edc11b280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x558206616600 con 0x558206686000 2024-04-25T01:18:25.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.165+0000 7f1edc11b280 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558206616780 con 0x558206686400 2024-04-25T01:18:25.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.165+0000 7f1edc11b280 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x558206616900 con 0x55820662dc00 2024-04-25T01:18:25.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.165+0000 7f1ed9600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558206686000 0x5582057cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.165+0000 7f1ed9600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558206686000 0x5582057cdb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47582/0 (socket says 172.21.15.90:47582) 2024-04-25T01:18:25.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.165+0000 7f1ed9600640 1 -- 172.21.15.90:0/3882038219 learned_addr learned my addr 172.21.15.90:0/3882038219 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:25.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.166+0000 7f1eda000640 1 --2- 172.21.15.90:0/3882038219 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55820662dc00 0x5582057cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.166+0000 7f1ed9600640 1 -- 172.21.15.90:0/3882038219 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55820662dc00 msgr2=0x5582057cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:25.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.166+0000 7f1ed9600640 1 --2- 172.21.15.90:0/3882038219 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55820662dc00 0x5582057cd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.166+0000 7f1ed9600640 1 -- 172.21.15.90:0/3882038219 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558206686400 msgr2=0x5582057ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:25.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.166+0000 7f1ed9600640 1 --2- 172.21.15.90:0/3882038219 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558206686400 0x5582057ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.166+0000 7f1ed9600640 1 -- 172.21.15.90:0/3882038219 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582057943c0 con 0x558206686000 2024-04-25T01:18:25.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.166+0000 7f1eda000640 1 --2- 172.21.15.90:0/3882038219 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55820662dc00 0x5582057cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:25.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.166+0000 7f1ed9600640 1 --2- 172.21.15.90:0/3882038219 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558206686000 0x5582057cdb80 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x55820663c210 tx=0x5582057dd020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aebbf672b1749505 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.166+0000 7f1ed3c00640 1 -- 172.21.15.90:0/3882038219 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x558205786820 con 0x558206686000 2024-04-25T01:18:25.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.166+0000 7f1ed3c00640 1 -- 172.21.15.90:0/3882038219 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55820578cfc0 con 0x558206686000 2024-04-25T01:18:25.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.166+0000 7f1ed3c00640 1 -- 172.21.15.90:0/3882038219 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x558205786680 con 0x558206686000 2024-04-25T01:18:25.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.167+0000 7f1edc11b280 1 -- 172.21.15.90:0/3882038219 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558206686000 msgr2=0x5582057cdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.167+0000 7f1edc11b280 1 --2- 172.21.15.90:0/3882038219 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558206686000 0x5582057cdb80 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x55820663c210 tx=0x5582057dd020 comp rx=0 tx=0).stop 2024-04-25T01:18:25.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.167+0000 7f1edc11b280 1 -- 172.21.15.90:0/3882038219 shutdown_connections 2024-04-25T01:18:25.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.167+0000 7f1edc11b280 1 --2- 172.21.15.90:0/3882038219 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558206686400 0x5582057ce100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.167+0000 7f1edc11b280 1 --2- 172.21.15.90:0/3882038219 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558206686000 0x5582057cdb80 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.167+0000 7f1edc11b280 1 --2- 172.21.15.90:0/3882038219 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55820662dc00 0x5582057cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.167+0000 7f1edc11b280 1 -- 172.21.15.90:0/3882038219 >> 172.21.15.90:0/3882038219 conn(0x55820662d800 msgr2=0x5582064df000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:25.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.167+0000 7f1edc11b280 1 -- 172.21.15.90:0/3882038219 shutdown_connections 2024-04-25T01:18:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.167+0000 7f1edc11b280 1 -- 172.21.15.90:0/3882038219 wait complete. 2024-04-25T01:18:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.168+0000 7f1edc11b280 1 Processor -- start 2024-04-25T01:18:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.168+0000 7f1edc11b280 1 -- start start 2024-04-25T01:18:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.168+0000 7f1edc11b280 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55820662d800 0x5582057cf180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.169+0000 7f1edc11b280 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558206687000 0x5582057cec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.169+0000 7f1edc11b280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558206686c00 0x5582057ce680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.169+0000 7f1edc11b280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x558206617c80 con 0x558206686c00 2024-04-25T01:18:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.169+0000 7f1edc11b280 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558206616a80 con 0x558206687000 2024-04-25T01:18:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.169+0000 7f1edc11b280 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x558206617b00 con 0x55820662d800 2024-04-25T01:18:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.169+0000 7f1ed8c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558206686c00 0x5582057ce680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.169+0000 7f1ed8c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558206686c00 0x5582057ce680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47592/0 (socket says 172.21.15.90:47592) 2024-04-25T01:18:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.169+0000 7f1ed8c00640 1 -- 172.21.15.90:0/2091889591 learned_addr learned my addr 172.21.15.90:0/2091889591 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.169+0000 7f1eda000640 1 --2- 172.21.15.90:0/2091889591 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55820662d800 0x5582057cf180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.169+0000 7f1ed9600640 1 --2- 172.21.15.90:0/2091889591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558206687000 0x5582057cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.169+0000 7f1ed8c00640 1 -- 172.21.15.90:0/2091889591 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55820662d800 msgr2=0x5582057cf180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.169+0000 7f1ed8c00640 1 --2- 172.21.15.90:0/2091889591 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55820662d800 0x5582057cf180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.169+0000 7f1ed8c00640 1 -- 172.21.15.90:0/2091889591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558206687000 msgr2=0x5582057cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.169+0000 7f1ed8c00640 1 --2- 172.21.15.90:0/2091889591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558206687000 0x5582057cec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.169+0000 7f1ed8c00640 1 -- 172.21.15.90:0/2091889591 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558205794780 con 0x558206686c00 2024-04-25T01:18:25.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.169+0000 7f1ed9600640 1 --2- 172.21.15.90:0/2091889591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558206687000 0x5582057cec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:25.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.170+0000 7f1ed8c00640 1 --2- 172.21.15.90:0/2091889591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558206686c00 0x5582057ce680 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x55820663d3e0 tx=0x55820663fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.170+0000 7f1ed1e00640 1 -- 172.21.15.90:0/2091889591 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x558205786820 con 0x558206686c00 2024-04-25T01:18:25.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.170+0000 7f1ed1e00640 1 -- 172.21.15.90:0/2091889591 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55820578ce00 con 0x558206686c00 2024-04-25T01:18:25.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.170+0000 7f1ed1e00640 1 -- 172.21.15.90:0/2091889591 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x558205786680 con 0x558206686c00 2024-04-25T01:18:25.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.170+0000 7f1edc11b280 1 -- 172.21.15.90:0/2091889591 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5582057954a0 con 0x558206686c00 2024-04-25T01:18:25.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.170+0000 7f1edc11b280 1 -- 172.21.15.90:0/2091889591 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558205794960 con 0x558206686c00 2024-04-25T01:18:25.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.170+0000 7f1edc11b280 4 main: resolving pool 7 2024-04-25T01:18:25.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.172+0000 7f1ed1e00640 1 -- 172.21.15.90:0/2091889591 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x55820667a900 con 0x558206686c00 2024-04-25T01:18:25.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.172+0000 7f1ed1e00640 1 --2- 172.21.15.90:0/2091889591 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x558206698c00 0x5582057cf700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.172+0000 7f1ed1e00640 1 -- 172.21.15.90:0/2091889591 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x5582066a0a00 con 0x558206686c00 2024-04-25T01:18:25.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.173+0000 7f1edc11b280 4 main: creating IoCtx.. 2024-04-25T01:18:25.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.173+0000 7f1edc11b280 1 --2- 172.21.15.90:0/2091889591 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x558206686800 0x5582057ce100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.173+0000 7f1edc11b280 1 -- 172.21.15.90:0/2091889591 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:1 7.b 7:d0630e4c:::mds0_sessionmap:head [omap-clear,omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) -- 0x558206686400 con 0x558206686800 2024-04-25T01:18:25.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.173+0000 7f1eda000640 1 --2- 172.21.15.90:0/2091889591 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x558206698c00 0x5582057cf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.173+0000 7f1eda000640 1 --2- 172.21.15.90:0/2091889591 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x558206698c00 0x5582057cf700 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x5582057f19e0 tx=0x5582064f2780 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.173+0000 7f1ed9600640 1 --2- 172.21.15.90:0/2091889591 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x558206686800 0x5582057ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.174+0000 7f1ed9600640 1 --2- 172.21.15.90:0/2091889591 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x558206686800 0x5582057ce100 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.161 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T01:18:25.161 INFO:teuthology.orchestra.run.smithi090.stdout: "0": { 2024-04-25T01:18:25.161 INFO:teuthology.orchestra.run.smithi090.stdout: "data": {}, 2024-04-25T01:18:25.161 INFO:teuthology.orchestra.run.smithi090.stdout: "result": 0 2024-04-25T01:18:25.161 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:18:25.161 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T01:18:25.161 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:25.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.177+0000 7f1ed9600640 1 -- 172.21.15.90:0/2091889591 <== osd.6 v2:172.21.15.103:6816/2008388906 1 ==== osd_op_reply(1 mds0_sessionmap [omap-clear,omap-set-header] v50'1 uv1 ondisk = 0) ==== 201+0+0 (crc 0 0 0) 0x5582064e2000 con 0x558206686800 2024-04-25T01:18:25.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.177+0000 7f1edc11b280 1 -- 172.21.15.90:0/2091889591 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x558206686800 msgr2=0x5582057ce100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.177+0000 7f1edc11b280 1 --2- 172.21.15.90:0/2091889591 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x558206686800 0x5582057ce100 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.177+0000 7f1edc11b280 1 -- 172.21.15.90:0/2091889591 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x558206698c00 msgr2=0x5582057cf700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.177+0000 7f1edc11b280 1 --2- 172.21.15.90:0/2091889591 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x558206698c00 0x5582057cf700 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x5582057f19e0 tx=0x5582064f2780 comp rx=0 tx=0).stop 2024-04-25T01:18:25.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.177+0000 7f1edc11b280 1 -- 172.21.15.90:0/2091889591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558206686c00 msgr2=0x5582057ce680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.177+0000 7f1edc11b280 1 --2- 172.21.15.90:0/2091889591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558206686c00 0x5582057ce680 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x55820663d3e0 tx=0x55820663fa40 comp rx=0 tx=0).stop 2024-04-25T01:18:25.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.177+0000 7f1ed8c00640 1 -- 172.21.15.90:0/2091889591 reap_dead start 2024-04-25T01:18:25.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.177+0000 7f1edc11b280 1 -- 172.21.15.90:0/2091889591 shutdown_connections 2024-04-25T01:18:25.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.177+0000 7f1edc11b280 1 -- 172.21.15.90:0/2091889591 >> 172.21.15.90:0/2091889591 conn(0x55820662dc00 msgr2=0x5582064df000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:25.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.177+0000 7f1edc11b280 1 -- 172.21.15.90:0/2091889591 shutdown_connections 2024-04-25T01:18:25.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.177+0000 7f1edc11b280 1 -- 172.21.15.90:0/2091889591 wait complete. 2024-04-25T01:18:25.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.178+0000 7f1edc11b280 1 -- 172.21.15.90:0/1274846577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558205760c00 msgr2=0x5582057cc580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.178+0000 7f1edc11b280 1 --2- 172.21.15.90:0/1274846577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558205760c00 0x5582057cc580 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x5582065791d0 tx=0x5582064f4180 comp rx=0 tx=0).stop 2024-04-25T01:18:25.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.178+0000 7f1edc11b280 1 -- 172.21.15.90:0/1274846577 shutdown_connections 2024-04-25T01:18:25.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.178+0000 7f1edc11b280 1 --2- 172.21.15.90:0/1274846577 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x558205761c00 0x5582057cd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.178+0000 7f1edc11b280 1 --2- 172.21.15.90:0/1274846577 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558205761800 0x5582057ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.178+0000 7f1edc11b280 1 --2- 172.21.15.90:0/1274846577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558205760c00 0x5582057cc580 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.178+0000 7f1edc11b280 1 -- 172.21.15.90:0/1274846577 >> 172.21.15.90:0/1274846577 conn(0x558205761000 msgr2=0x5582064de800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:25.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.178+0000 7f1edc11b280 1 -- 172.21.15.90:0/1274846577 shutdown_connections 2024-04-25T01:18:25.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.178+0000 7f1edc11b280 1 -- 172.21.15.90:0/1274846577 wait complete. 2024-04-25T01:18:25.167 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-table-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'recovery_fs:0', 'reset', 'session'] in time 0:00:00.120510, result: { "0": { "data": {}, "result": 0 } } 2024-04-25T01:18:25.167 DEBUG:teuthology.orchestra.run.smithi090:> cephfs-table-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 recovery_fs:0 reset snap 2024-04-25T01:18:25.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.268+0000 7f13ed398280 1 -- start start 2024-04-25T01:18:25.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.268+0000 7f13ed398280 1 Processor -- start 2024-04-25T01:18:25.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.269+0000 7f13ed398280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56476c928c00 0x56476c994580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.269+0000 7f13ed398280 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56476c929800 0x56476c994b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.269+0000 7f13ed398280 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56476c929c00 0x56476c995080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.269+0000 7f13ed398280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56476d70e600 con 0x56476c928c00 2024-04-25T01:18:25.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.269+0000 7f13ed398280 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56476d70e480 con 0x56476c929800 2024-04-25T01:18:25.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.269+0000 7f13ed398280 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x56476d70e300 con 0x56476c929c00 2024-04-25T01:18:25.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.269+0000 7f13eb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56476c928c00 0x56476c994580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.269+0000 7f13eb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56476c928c00 0x56476c994580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47594/0 (socket says 172.21.15.90:47594) 2024-04-25T01:18:25.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.269+0000 7f13eb400640 1 -- 172.21.15.90:0/2396806224 learned_addr learned my addr 172.21.15.90:0/2396806224 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:25.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.270+0000 7f13eaa00640 1 --2- 172.21.15.90:0/2396806224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56476c929800 0x56476c994b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.270+0000 7f13ebe00640 1 --2- 172.21.15.90:0/2396806224 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56476c929c00 0x56476c995080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.270+0000 7f13eb400640 1 -- 172.21.15.90:0/2396806224 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56476c929c00 msgr2=0x56476c995080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:25.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.270+0000 7f13eb400640 1 --2- 172.21.15.90:0/2396806224 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56476c929c00 0x56476c995080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.270+0000 7f13eb400640 1 -- 172.21.15.90:0/2396806224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56476c929800 msgr2=0x56476c994b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:25.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.270+0000 7f13eb400640 1 --2- 172.21.15.90:0/2396806224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56476c929800 0x56476c994b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.270+0000 7f13eb400640 1 -- 172.21.15.90:0/2396806224 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56476c95c780 con 0x56476c928c00 2024-04-25T01:18:25.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.270+0000 7f13eb400640 1 --2- 172.21.15.90:0/2396806224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56476c928c00 0x56476c994580 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x56476d7451a0 tx=0x56476d6bc180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9919ecc985abf755 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.270+0000 7f13e8200640 1 -- 172.21.15.90:0/2396806224 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x56476c94e680 con 0x56476c928c00 2024-04-25T01:18:25.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.270+0000 7f13e8200640 1 -- 172.21.15.90:0/2396806224 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x56476c954fc0 con 0x56476c928c00 2024-04-25T01:18:25.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.270+0000 7f13e8200640 1 -- 172.21.15.90:0/2396806224 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x56476c94eb60 con 0x56476c928c00 2024-04-25T01:18:25.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.271+0000 7f13ed398280 1 -- 172.21.15.90:0/2396806224 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56476c95d4a0 con 0x56476c928c00 2024-04-25T01:18:25.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.271+0000 7f13ed398280 1 -- 172.21.15.90:0/2396806224 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) -- 0x56476c95c000 con 0x56476c928c00 2024-04-25T01:18:25.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.271+0000 7f13ed398280 4 waiting for MDS map... 2024-04-25T01:18:25.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.271+0000 7f13e8200640 1 -- 172.21.15.90:0/2396806224 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x56476d7f8600 con 0x56476c928c00 2024-04-25T01:18:25.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.271+0000 7f13e8200640 1 -- 172.21.15.90:0/2396806224 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 23) ==== 4135+0+0 (secure 0 0 0) 0x56476c983180 con 0x56476c928c00 2024-04-25T01:18:25.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.271+0000 7f13ed398280 4 Got MDS map 23 2024-04-25T01:18:25.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.271+0000 7f13ed398280 10 main: main 2024-04-25T01:18:25.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.271+0000 7f13ed398280 4 main: connecting to RADOS... 2024-04-25T01:18:25.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.272+0000 7f13ed398280 1 Processor -- start 2024-04-25T01:18:25.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.272+0000 7f13ed398280 1 -- start start 2024-04-25T01:18:25.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.273+0000 7f13ed398280 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56476d7e5c00 0x56476c995600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.273+0000 7f13ed398280 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56476d84e000 0x56476c995b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.273+0000 7f13ed398280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56476d84e400 0x56476c996100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.273+0000 7f13ed398280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56476d70e180 con 0x56476d84e400 2024-04-25T01:18:25.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.273+0000 7f13ed398280 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56476d70e900 con 0x56476d7e5c00 2024-04-25T01:18:25.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.273+0000 7f13ed398280 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x56476c98a900 con 0x56476d84e000 2024-04-25T01:18:25.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.273+0000 7f13eaa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56476d84e400 0x56476c996100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.273+0000 7f13eaa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56476d84e400 0x56476c996100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47604/0 (socket says 172.21.15.90:47604) 2024-04-25T01:18:25.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.273+0000 7f13eaa00640 1 -- 172.21.15.90:0/3416537071 learned_addr learned my addr 172.21.15.90:0/3416537071 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:25.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.273+0000 7f13ebe00640 1 --2- 172.21.15.90:0/3416537071 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56476d7e5c00 0x56476c995600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.274+0000 7f13eaa00640 1 -- 172.21.15.90:0/3416537071 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56476d84e000 msgr2=0x56476c995b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:25.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.274+0000 7f13eb400640 1 --2- 172.21.15.90:0/3416537071 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56476d84e000 0x56476c995b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.274+0000 7f13eaa00640 1 --2- 172.21.15.90:0/3416537071 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56476d84e000 0x56476c995b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.274+0000 7f13eaa00640 1 -- 172.21.15.90:0/3416537071 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56476d7e5c00 msgr2=0x56476c995600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:25.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.274+0000 7f13eaa00640 1 --2- 172.21.15.90:0/3416537071 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56476d7e5c00 0x56476c995600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.274+0000 7f13eaa00640 1 -- 172.21.15.90:0/3416537071 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56476c95c3c0 con 0x56476d84e400 2024-04-25T01:18:25.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.274+0000 7f13ebe00640 1 --2- 172.21.15.90:0/3416537071 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56476d7e5c00 0x56476c995600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:25.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.274+0000 7f13eaa00640 1 --2- 172.21.15.90:0/3416537071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56476d84e400 0x56476c996100 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x56476d8041b0 tx=0x56476d6bbf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d4f345e20b2f58a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.274+0000 7f13e5a00640 1 -- 172.21.15.90:0/3416537071 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x56476c94e9c0 con 0x56476d84e400 2024-04-25T01:18:25.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.274+0000 7f13e5a00640 1 -- 172.21.15.90:0/3416537071 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x56476c954e00 con 0x56476d84e400 2024-04-25T01:18:25.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.274+0000 7f13e5a00640 1 -- 172.21.15.90:0/3416537071 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x56476c94e680 con 0x56476d84e400 2024-04-25T01:18:25.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.274+0000 7f13ed398280 1 -- 172.21.15.90:0/3416537071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56476d84e400 msgr2=0x56476c996100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.274+0000 7f13ed398280 1 --2- 172.21.15.90:0/3416537071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56476d84e400 0x56476c996100 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x56476d8041b0 tx=0x56476d6bbf80 comp rx=0 tx=0).stop 2024-04-25T01:18:25.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.275+0000 7f13ed398280 1 -- 172.21.15.90:0/3416537071 shutdown_connections 2024-04-25T01:18:25.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.275+0000 7f13ed398280 1 --2- 172.21.15.90:0/3416537071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56476d84e400 0x56476c996100 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.275+0000 7f13ed398280 1 --2- 172.21.15.90:0/3416537071 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56476d84e000 0x56476c995b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.275+0000 7f13ed398280 1 --2- 172.21.15.90:0/3416537071 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56476d7e5c00 0x56476c995600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.275+0000 7f13ed398280 1 -- 172.21.15.90:0/3416537071 >> 172.21.15.90:0/3416537071 conn(0x56476d7e5800 msgr2=0x56476d6a7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:25.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.275+0000 7f13ed398280 1 -- 172.21.15.90:0/3416537071 shutdown_connections 2024-04-25T01:18:25.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.275+0000 7f13ed398280 1 -- 172.21.15.90:0/3416537071 wait complete. 2024-04-25T01:18:25.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.275+0000 7f13ed398280 1 Processor -- start 2024-04-25T01:18:25.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.276+0000 7f13ed398280 1 -- start start 2024-04-25T01:18:25.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.276+0000 7f13ed398280 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56476d7e5800 0x56476c997180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.276+0000 7f13ed398280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56476d71c000 0x56476c996c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.276+0000 7f13ed398280 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56476d71c400 0x56476c996680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.276+0000 7f13ed398280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56476d70fc80 con 0x56476d71c000 2024-04-25T01:18:25.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.276+0000 7f13ed398280 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56476d70e900 con 0x56476d7e5800 2024-04-25T01:18:25.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.276+0000 7f13ed398280 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x56476d70fb00 con 0x56476d71c400 2024-04-25T01:18:25.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.276+0000 7f13eb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56476d71c000 0x56476c996c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.277+0000 7f13eb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56476d71c000 0x56476c996c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47612/0 (socket says 172.21.15.90:47612) 2024-04-25T01:18:25.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.277+0000 7f13eb400640 1 -- 172.21.15.90:0/2747463813 learned_addr learned my addr 172.21.15.90:0/2747463813 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:25.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.277+0000 7f13eaa00640 1 --2- 172.21.15.90:0/2747463813 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56476d71c400 0x56476c996680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.277+0000 7f13ebe00640 1 --2- 172.21.15.90:0/2747463813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56476d7e5800 0x56476c997180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.277+0000 7f13eb400640 1 -- 172.21.15.90:0/2747463813 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56476d71c400 msgr2=0x56476c996680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.277+0000 7f13eb400640 1 --2- 172.21.15.90:0/2747463813 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56476d71c400 0x56476c996680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.277+0000 7f13eb400640 1 -- 172.21.15.90:0/2747463813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56476d7e5800 msgr2=0x56476c997180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.277+0000 7f13eb400640 1 --2- 172.21.15.90:0/2747463813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56476d7e5800 0x56476c997180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.277+0000 7f13eb400640 1 -- 172.21.15.90:0/2747463813 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56476c95c780 con 0x56476d71c000 2024-04-25T01:18:25.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.277+0000 7f13eaa00640 1 --2- 172.21.15.90:0/2747463813 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56476d71c400 0x56476c996680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:25.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.277+0000 7f13eb400640 1 --2- 172.21.15.90:0/2747463813 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56476d71c000 0x56476c996c00 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x56476d745860 tx=0x56476d6ba8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.278+0000 7f13e3c00640 1 -- 172.21.15.90:0/2747463813 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x56476c94e820 con 0x56476d71c000 2024-04-25T01:18:25.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.278+0000 7f13e3c00640 1 -- 172.21.15.90:0/2747463813 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x56476c9541c0 con 0x56476d71c000 2024-04-25T01:18:25.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.278+0000 7f13ed398280 1 -- 172.21.15.90:0/2747463813 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56476c95d4a0 con 0x56476d71c000 2024-04-25T01:18:25.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.278+0000 7f13e3c00640 1 -- 172.21.15.90:0/2747463813 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x56476c94e9c0 con 0x56476d71c000 2024-04-25T01:18:25.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.278+0000 7f13ed398280 1 -- 172.21.15.90:0/2747463813 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56476c95c960 con 0x56476d71c000 2024-04-25T01:18:25.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.278+0000 7f13ed398280 4 main: resolving pool 7 2024-04-25T01:18:25.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.279+0000 7f13e3c00640 1 -- 172.21.15.90:0/2747463813 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x56476d842900 con 0x56476d71c000 2024-04-25T01:18:25.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.280+0000 7f13e3c00640 1 --2- 172.21.15.90:0/2747463813 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x56476d723000 0x56476c997700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.280+0000 7f13e3c00640 1 -- 172.21.15.90:0/2747463813 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x56476d724600 con 0x56476d71c000 2024-04-25T01:18:25.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.280+0000 7f13ed398280 4 main: creating IoCtx.. 2024-04-25T01:18:25.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.280+0000 7f13ebe00640 1 --2- 172.21.15.90:0/2747463813 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x56476d723000 0x56476c997700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.280+0000 7f13ed398280 1 --2- 172.21.15.90:0/2747463813 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x56476d71c800 0x56476c996100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.280+0000 7f13ed398280 1 -- 172.21.15.90:0/2747463813 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:1 7.d 7:b50e409b:::mds_snaptable:head [writefull 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) -- 0x56476d71cc00 con 0x56476d71c800 2024-04-25T01:18:25.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.281+0000 7f13eaa00640 1 --2- 172.21.15.90:0/2747463813 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x56476d71c800 0x56476c996100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.281+0000 7f13ebe00640 1 --2- 172.21.15.90:0/2747463813 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x56476d723000 0x56476c997700 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x56476d6b2810 tx=0x56476cc26ae0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.281+0000 7f13eaa00640 1 --2- 172.21.15.90:0/2747463813 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x56476d71c800 0x56476c996100 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.284+0000 7f13eaa00640 1 -- 172.21.15.90:0/2747463813 <== osd.1 v2:172.21.15.90:6800/2685473653 1 ==== osd_op_reply(1 mds_snaptable [writefull 0~70] v50'1 uv1 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x56476d6aa000 con 0x56476d71c800 2024-04-25T01:18:25.269 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T01:18:25.269 INFO:teuthology.orchestra.run.smithi090.stdout: "result": 0 2024-04-25T01:18:25.270 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T01:18:25.270 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:25.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.284+0000 7f13ed398280 1 -- 172.21.15.90:0/2747463813 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x56476d71c800 msgr2=0x56476c996100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.284+0000 7f13ed398280 1 --2- 172.21.15.90:0/2747463813 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x56476d71c800 0x56476c996100 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.284+0000 7f13ed398280 1 -- 172.21.15.90:0/2747463813 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x56476d723000 msgr2=0x56476c997700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.284+0000 7f13ed398280 1 --2- 172.21.15.90:0/2747463813 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x56476d723000 0x56476c997700 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x56476d6b2810 tx=0x56476cc26ae0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.284+0000 7f13ed398280 1 -- 172.21.15.90:0/2747463813 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56476d71c000 msgr2=0x56476c996c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.284+0000 7f13ed398280 1 --2- 172.21.15.90:0/2747463813 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56476d71c000 0x56476c996c00 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x56476d745860 tx=0x56476d6ba8a0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.284+0000 7f13eaa00640 1 -- 172.21.15.90:0/2747463813 reap_dead start 2024-04-25T01:18:25.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.285+0000 7f13ed398280 1 -- 172.21.15.90:0/2747463813 shutdown_connections 2024-04-25T01:18:25.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.285+0000 7f13ed398280 1 -- 172.21.15.90:0/2747463813 >> 172.21.15.90:0/2747463813 conn(0x56476d7e5c00 msgr2=0x56476d6a7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:25.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.285+0000 7f13ed398280 1 -- 172.21.15.90:0/2747463813 shutdown_connections 2024-04-25T01:18:25.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.285+0000 7f13ed398280 1 -- 172.21.15.90:0/2747463813 wait complete. 2024-04-25T01:18:25.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.285+0000 7f13ed398280 1 -- 172.21.15.90:0/2396806224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56476c928c00 msgr2=0x56476c994580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.285+0000 7f13ed398280 1 --2- 172.21.15.90:0/2396806224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56476c928c00 0x56476c994580 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x56476d7451a0 tx=0x56476d6bc180 comp rx=0 tx=0).stop 2024-04-25T01:18:25.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.285+0000 7f13ed398280 1 -- 172.21.15.90:0/2396806224 shutdown_connections 2024-04-25T01:18:25.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.285+0000 7f13ed398280 1 --2- 172.21.15.90:0/2396806224 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56476c929c00 0x56476c995080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.285+0000 7f13ed398280 1 --2- 172.21.15.90:0/2396806224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56476c929800 0x56476c994b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.285+0000 7f13ed398280 1 --2- 172.21.15.90:0/2396806224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56476c928c00 0x56476c994580 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.285+0000 7f13ed398280 1 -- 172.21.15.90:0/2396806224 >> 172.21.15.90:0/2396806224 conn(0x56476c929000 msgr2=0x56476d6a6800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:25.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.285+0000 7f13ed398280 1 -- 172.21.15.90:0/2396806224 shutdown_connections 2024-04-25T01:18:25.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.285+0000 7f13ed398280 1 -- 172.21.15.90:0/2396806224 wait complete. 2024-04-25T01:18:25.275 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-table-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'recovery_fs:0', 'reset', 'snap'] in time 0:00:00.108307, result: { "result": 0 } 2024-04-25T01:18:25.276 DEBUG:teuthology.orchestra.run.smithi090:> cephfs-table-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 recovery_fs:0 reset inode 2024-04-25T01:18:25.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.374+0000 7fc0c7765280 1 -- start start 2024-04-25T01:18:25.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.374+0000 7fc0c7765280 1 Processor -- start 2024-04-25T01:18:25.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.375+0000 7fc0c7765280 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5646aa170c00 0x5646aa1dc580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.375+0000 7fc0c7765280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646aa171800 0x5646aa1dcb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.375+0000 7fc0c7765280 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5646aa171c00 0x5646aa1dd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.375+0000 7fc0c7765280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5646aaf60180 con 0x5646aa171800 2024-04-25T01:18:25.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.375+0000 7fc0c7765280 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5646aaf60300 con 0x5646aa171c00 2024-04-25T01:18:25.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.375+0000 7fc0c7765280 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5646aaf60480 con 0x5646aa170c00 2024-04-25T01:18:25.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.376+0000 7fc0c4e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646aa171800 0x5646aa1dcb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.376+0000 7fc0c4e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646aa171800 0x5646aa1dcb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47616/0 (socket says 172.21.15.90:47616) 2024-04-25T01:18:25.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.376+0000 7fc0c4e00640 1 -- 172.21.15.90:0/2872880625 learned_addr learned my addr 172.21.15.90:0/2872880625 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:25.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.376+0000 7fc0c5800640 1 --2- 172.21.15.90:0/2872880625 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5646aa170c00 0x5646aa1dc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.376+0000 7fc0c6200640 1 --2- 172.21.15.90:0/2872880625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5646aa171c00 0x5646aa1dd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.376+0000 7fc0c4e00640 1 -- 172.21.15.90:0/2872880625 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5646aa170c00 msgr2=0x5646aa1dc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:25.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.376+0000 7fc0c4e00640 1 --2- 172.21.15.90:0/2872880625 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5646aa170c00 0x5646aa1dc580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.376+0000 7fc0c4e00640 1 -- 172.21.15.90:0/2872880625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5646aa171c00 msgr2=0x5646aa1dd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:25.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.376+0000 7fc0c4e00640 1 --2- 172.21.15.90:0/2872880625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5646aa171c00 0x5646aa1dd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.376+0000 7fc0c4e00640 1 -- 172.21.15.90:0/2872880625 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646aa1a4780 con 0x5646aa171800 2024-04-25T01:18:25.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.376+0000 7fc0c6200640 1 --2- 172.21.15.90:0/2872880625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5646aa171c00 0x5646aa1dd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:25.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.376+0000 7fc0c4e00640 1 --2- 172.21.15.90:0/2872880625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646aa171800 0x5646aa1dcb00 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x5646aafa8a20 tx=0x5646aaf04de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b93b7d417c4d84a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.377+0000 7fc0c2600640 1 -- 172.21.15.90:0/2872880625 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5646aa1969c0 con 0x5646aa171800 2024-04-25T01:18:25.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.377+0000 7fc0c2600640 1 -- 172.21.15.90:0/2872880625 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x5646aa19cfc0 con 0x5646aa171800 2024-04-25T01:18:25.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.377+0000 7fc0c7765280 1 -- 172.21.15.90:0/2872880625 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646aa1a54a0 con 0x5646aa171800 2024-04-25T01:18:25.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.377+0000 7fc0c7765280 1 -- 172.21.15.90:0/2872880625 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) -- 0x5646aa1a4000 con 0x5646aa171800 2024-04-25T01:18:25.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.377+0000 7fc0c7765280 4 waiting for MDS map... 2024-04-25T01:18:25.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.377+0000 7fc0c2600640 1 -- 172.21.15.90:0/2872880625 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5646aa196680 con 0x5646aa171800 2024-04-25T01:18:25.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.377+0000 7fc0c2600640 1 -- 172.21.15.90:0/2872880625 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x5646aaf46a00 con 0x5646aa171800 2024-04-25T01:18:25.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.377+0000 7fc0c2600640 1 -- 172.21.15.90:0/2872880625 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 23) ==== 4135+0+0 (secure 0 0 0) 0x5646aa1cb180 con 0x5646aa171800 2024-04-25T01:18:25.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.377+0000 7fc0c7765280 4 Got MDS map 23 2024-04-25T01:18:25.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.377+0000 7fc0c7765280 10 main: main 2024-04-25T01:18:25.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.377+0000 7fc0c7765280 4 main: connecting to RADOS... 2024-04-25T01:18:25.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.378+0000 7fc0c7765280 1 Processor -- start 2024-04-25T01:18:25.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.378+0000 7fc0c7765280 1 -- start start 2024-04-25T01:18:25.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.379+0000 7fc0c7765280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646ab049000 0x5646aa1dd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.379+0000 7fc0c7765280 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5646ab049400 0x5646aa1ddb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.379+0000 7fc0c7765280 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5646ab049800 0x5646aa1de100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.379+0000 7fc0c7765280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5646aaf60600 con 0x5646ab049000 2024-04-25T01:18:25.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.379+0000 7fc0c7765280 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5646aaf60780 con 0x5646ab049400 2024-04-25T01:18:25.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.379+0000 7fc0c7765280 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5646aaf60900 con 0x5646ab049800 2024-04-25T01:18:25.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.379+0000 7fc0c6200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646ab049000 0x5646aa1dd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.379+0000 7fc0c6200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646ab049000 0x5646aa1dd600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47618/0 (socket says 172.21.15.90:47618) 2024-04-25T01:18:25.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.379+0000 7fc0c6200640 1 -- 172.21.15.90:0/2081630821 learned_addr learned my addr 172.21.15.90:0/2081630821 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:25.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.380+0000 7fc0c6200640 1 -- 172.21.15.90:0/2081630821 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5646ab049800 msgr2=0x5646aa1de100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:25.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.380+0000 7fc0c6200640 1 --2- 172.21.15.90:0/2081630821 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5646ab049800 0x5646aa1de100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.380+0000 7fc0c6200640 1 -- 172.21.15.90:0/2081630821 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5646ab049400 msgr2=0x5646aa1ddb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:25.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.380+0000 7fc0c6200640 1 --2- 172.21.15.90:0/2081630821 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5646ab049400 0x5646aa1ddb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.380+0000 7fc0c6200640 1 -- 172.21.15.90:0/2081630821 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646aa1a43c0 con 0x5646ab049000 2024-04-25T01:18:25.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.380+0000 7fc0c6200640 1 --2- 172.21.15.90:0/2081630821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646ab049000 0x5646aa1dd600 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x5646ab04c6f0 tx=0x5646ab04b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaafe0e056eda543 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.380+0000 7fc0bfe00640 1 -- 172.21.15.90:0/2081630821 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5646aa1969c0 con 0x5646ab049000 2024-04-25T01:18:25.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.380+0000 7fc0bfe00640 1 -- 172.21.15.90:0/2081630821 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x5646aa19cc40 con 0x5646ab049000 2024-04-25T01:18:25.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.380+0000 7fc0bfe00640 1 -- 172.21.15.90:0/2081630821 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5646aa196680 con 0x5646ab049000 2024-04-25T01:18:25.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.381+0000 7fc0c7765280 1 -- 172.21.15.90:0/2081630821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646ab049000 msgr2=0x5646aa1dd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.381+0000 7fc0c7765280 1 --2- 172.21.15.90:0/2081630821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646ab049000 0x5646aa1dd600 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x5646ab04c6f0 tx=0x5646ab04b800 comp rx=0 tx=0).stop 2024-04-25T01:18:25.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.381+0000 7fc0c7765280 1 -- 172.21.15.90:0/2081630821 shutdown_connections 2024-04-25T01:18:25.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.381+0000 7fc0c7765280 1 --2- 172.21.15.90:0/2081630821 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5646ab049800 0x5646aa1de100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.381+0000 7fc0c7765280 1 --2- 172.21.15.90:0/2081630821 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5646ab049400 0x5646aa1ddb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.381+0000 7fc0c7765280 1 --2- 172.21.15.90:0/2081630821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646ab049000 0x5646aa1dd600 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.381+0000 7fc0c7765280 1 -- 172.21.15.90:0/2081630821 >> 172.21.15.90:0/2081630821 conn(0x5646ab048c00 msgr2=0x5646aa275800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:25.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.381+0000 7fc0c7765280 1 -- 172.21.15.90:0/2081630821 shutdown_connections 2024-04-25T01:18:25.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.381+0000 7fc0c7765280 1 -- 172.21.15.90:0/2081630821 wait complete. 2024-04-25T01:18:25.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.381+0000 7fc0c7765280 1 Processor -- start 2024-04-25T01:18:25.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.382+0000 7fc0c7765280 1 -- start start 2024-04-25T01:18:25.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.382+0000 7fc0c7765280 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5646ab049000 0x5646aa1df180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.382+0000 7fc0c7765280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646ab049400 0x5646aa1dec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.382+0000 7fc0c7765280 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5646ab049800 0x5646aa1de680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.382+0000 7fc0c7765280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5646aaf61980 con 0x5646ab049400 2024-04-25T01:18:25.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.382+0000 7fc0c7765280 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5646aaf60300 con 0x5646ab049800 2024-04-25T01:18:25.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.382+0000 7fc0c7765280 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x5646aaf60c00 con 0x5646ab049000 2024-04-25T01:18:25.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.383+0000 7fc0c5800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646ab049400 0x5646aa1dec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.383+0000 7fc0c5800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646ab049400 0x5646aa1dec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47632/0 (socket says 172.21.15.90:47632) 2024-04-25T01:18:25.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.383+0000 7fc0c5800640 1 -- 172.21.15.90:0/3563071823 learned_addr learned my addr 172.21.15.90:0/3563071823 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:25.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.383+0000 7fc0c6200640 1 --2- 172.21.15.90:0/3563071823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5646ab049000 0x5646aa1df180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.383+0000 7fc0c4e00640 1 --2- 172.21.15.90:0/3563071823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5646ab049800 0x5646aa1de680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.383+0000 7fc0c5800640 1 -- 172.21.15.90:0/3563071823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5646ab049000 msgr2=0x5646aa1df180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.383+0000 7fc0c5800640 1 --2- 172.21.15.90:0/3563071823 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5646ab049000 0x5646aa1df180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.383+0000 7fc0c5800640 1 -- 172.21.15.90:0/3563071823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5646ab049800 msgr2=0x5646aa1de680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.383+0000 7fc0c5800640 1 --2- 172.21.15.90:0/3563071823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5646ab049800 0x5646aa1de680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.383+0000 7fc0c5800640 1 -- 172.21.15.90:0/3563071823 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646aa1a4960 con 0x5646ab049400 2024-04-25T01:18:25.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.383+0000 7fc0c5800640 1 --2- 172.21.15.90:0/3563071823 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646ab049400 0x5646aa1dec00 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x5646ab04d4a0 tx=0x5646ab04b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.383+0000 7fc0be000640 1 -- 172.21.15.90:0/3563071823 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5646aa1969c0 con 0x5646ab049400 2024-04-25T01:18:25.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.384+0000 7fc0be000640 1 -- 172.21.15.90:0/3563071823 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x5646aa19ca80 con 0x5646ab049400 2024-04-25T01:18:25.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.384+0000 7fc0be000640 1 -- 172.21.15.90:0/3563071823 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5646aa196680 con 0x5646ab049400 2024-04-25T01:18:25.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.384+0000 7fc0c7765280 1 -- 172.21.15.90:0/3563071823 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5646aa1a5680 con 0x5646ab049400 2024-04-25T01:18:25.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.384+0000 7fc0c7765280 1 -- 172.21.15.90:0/3563071823 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646aa1a5860 con 0x5646ab049400 2024-04-25T01:18:25.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.384+0000 7fc0c7765280 4 main: resolving pool 7 2024-04-25T01:18:25.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.385+0000 7fc0be000640 1 -- 172.21.15.90:0/3563071823 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x5646ab098900 con 0x5646ab049400 2024-04-25T01:18:25.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.386+0000 7fc0be000640 1 --2- 172.21.15.90:0/3563071823 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x5646ab0af400 0x5646aa1df700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.386+0000 7fc0be000640 1 -- 172.21.15.90:0/3563071823 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x5646ab0b0e00 con 0x5646ab049400 2024-04-25T01:18:25.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.386+0000 7fc0c7765280 4 main: creating IoCtx.. 2024-04-25T01:18:25.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.386+0000 7fc0c7765280 1 --2- 172.21.15.90:0/3563071823 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x5646ab0af800 0x5646aa1de100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.386+0000 7fc0c6200640 1 --2- 172.21.15.90:0/3563071823 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x5646ab0af400 0x5646aa1df700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.386+0000 7fc0c7765280 1 -- 172.21.15.90:0/3563071823 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:1 7.13 7:c91d4a1d:::mds0_inotable:head [writefull 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) -- 0x5646ab0afc00 con 0x5646ab0af800 2024-04-25T01:18:25.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.387+0000 7fc0c6200640 1 --2- 172.21.15.90:0/3563071823 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x5646ab0af400 0x5646aa1df700 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x5646ab04c6f0 tx=0x5646aaf040c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.387+0000 7fc0c4e00640 1 --2- 172.21.15.90:0/3563071823 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x5646ab0af800 0x5646aa1de100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.388+0000 7fc0c4e00640 1 --2- 172.21.15.90:0/3563071823 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x5646ab0af800 0x5646aa1de100 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.391+0000 7fc0c4e00640 1 -- 172.21.15.90:0/3563071823 <== osd.7 v2:172.21.15.103:6800/1145679154 1 ==== osd_op_reply(1 mds0_inotable [writefull 0~34] v50'2 uv2 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x5646aaefa000 con 0x5646ab0af800 2024-04-25T01:18:25.375 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T01:18:25.375 INFO:teuthology.orchestra.run.smithi090.stdout: "0": { 2024-04-25T01:18:25.375 INFO:teuthology.orchestra.run.smithi090.stdout: "data": {}, 2024-04-25T01:18:25.375 INFO:teuthology.orchestra.run.smithi090.stdout: "result": 0 2024-04-25T01:18:25.375 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T01:18:25.376 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T01:18:25.376 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:25.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.391+0000 7fc0c7765280 1 -- 172.21.15.90:0/3563071823 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x5646ab0af800 msgr2=0x5646aa1de100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.391+0000 7fc0c7765280 1 --2- 172.21.15.90:0/3563071823 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x5646ab0af800 0x5646aa1de100 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.391+0000 7fc0c7765280 1 -- 172.21.15.90:0/3563071823 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x5646ab0af400 msgr2=0x5646aa1df700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.391+0000 7fc0c7765280 1 --2- 172.21.15.90:0/3563071823 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x5646ab0af400 0x5646aa1df700 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x5646ab04c6f0 tx=0x5646aaf040c0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.391+0000 7fc0c7765280 1 -- 172.21.15.90:0/3563071823 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646ab049400 msgr2=0x5646aa1dec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.391+0000 7fc0c7765280 1 --2- 172.21.15.90:0/3563071823 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646ab049400 0x5646aa1dec00 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x5646ab04d4a0 tx=0x5646ab04b920 comp rx=0 tx=0).stop 2024-04-25T01:18:25.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.391+0000 7fc0c5800640 1 -- 172.21.15.90:0/3563071823 reap_dead start 2024-04-25T01:18:25.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.391+0000 7fc0c7765280 1 -- 172.21.15.90:0/3563071823 shutdown_connections 2024-04-25T01:18:25.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.391+0000 7fc0c7765280 1 -- 172.21.15.90:0/3563071823 >> 172.21.15.90:0/3563071823 conn(0x5646ab048c00 msgr2=0x5646aa275800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:25.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.392+0000 7fc0c7765280 1 -- 172.21.15.90:0/3563071823 shutdown_connections 2024-04-25T01:18:25.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.392+0000 7fc0c7765280 1 -- 172.21.15.90:0/3563071823 wait complete. 2024-04-25T01:18:25.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.392+0000 7fc0c7765280 1 -- 172.21.15.90:0/2872880625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646aa171800 msgr2=0x5646aa1dcb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.392+0000 7fc0c7765280 1 --2- 172.21.15.90:0/2872880625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646aa171800 0x5646aa1dcb00 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x5646aafa8a20 tx=0x5646aaf04de0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.392+0000 7fc0c7765280 1 -- 172.21.15.90:0/2872880625 shutdown_connections 2024-04-25T01:18:25.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.392+0000 7fc0c7765280 1 --2- 172.21.15.90:0/2872880625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5646aa171c00 0x5646aa1dd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.392+0000 7fc0c7765280 1 --2- 172.21.15.90:0/2872880625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646aa171800 0x5646aa1dcb00 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.392+0000 7fc0c7765280 1 --2- 172.21.15.90:0/2872880625 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x5646aa170c00 0x5646aa1dc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.392+0000 7fc0c7765280 1 -- 172.21.15.90:0/2872880625 >> 172.21.15.90:0/2872880625 conn(0x5646aa171000 msgr2=0x5646aa275000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:25.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.392+0000 7fc0c7765280 1 -- 172.21.15.90:0/2872880625 shutdown_connections 2024-04-25T01:18:25.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.392+0000 7fc0c7765280 1 -- 172.21.15.90:0/2872880625 wait complete. 2024-04-25T01:18:25.381 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-table-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'recovery_fs:0', 'reset', 'inode'] in time 0:00:00.105681, result: { "0": { "data": {}, "result": 0 } } 2024-04-25T01:18:25.382 DEBUG:teuthology.orchestra.run.smithi090:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_extents --alternate-pool recovery_fs_metadata --filesystem cephfs cephfs_data 2024-04-25T01:18:25.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.481+0000 7f6fe1c17740 1 -- start start 2024-04-25T01:18:25.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.481+0000 7f6fe1c17740 1 Processor -- start 2024-04-25T01:18:25.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.482+0000 7f6fe1c17740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a25b646c00 0x55a25b6ea580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.482+0000 7f6fe1c17740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a25b647800 0x55a25b6eab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.482+0000 7f6fe1c17740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a25b647c00 0x55a25b6eb080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.482+0000 7f6fe1c17740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a25c3ae600 con 0x55a25b647c00 2024-04-25T01:18:25.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.482+0000 7f6fe1c17740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a25c3ae480 con 0x55a25b647800 2024-04-25T01:18:25.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.482+0000 7f6fe1c17740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55a25c3ae300 con 0x55a25b646c00 2024-04-25T01:18:25.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.482+0000 7f6fdfc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a25b647c00 0x55a25b6eb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.482+0000 7f6fdfc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a25b647c00 0x55a25b6eb080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47650/0 (socket says 172.21.15.90:47650) 2024-04-25T01:18:25.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.482+0000 7f6fdfc00640 1 -- 172.21.15.90:0/4204169444 learned_addr learned my addr 172.21.15.90:0/4204169444 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:25.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.483+0000 7f6fdf200640 1 --2- 172.21.15.90:0/4204169444 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a25b646c00 0x55a25b6ea580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.483+0000 7f6fde800640 1 --2- 172.21.15.90:0/4204169444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a25b647800 0x55a25b6eab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.483+0000 7f6fdfc00640 1 -- 172.21.15.90:0/4204169444 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a25b646c00 msgr2=0x55a25b6ea580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:25.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.483+0000 7f6fdfc00640 1 --2- 172.21.15.90:0/4204169444 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a25b646c00 0x55a25b6ea580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.483+0000 7f6fdfc00640 1 -- 172.21.15.90:0/4204169444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a25b647800 msgr2=0x55a25b6eab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:25.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.483+0000 7f6fdfc00640 1 --2- 172.21.15.90:0/4204169444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a25b647800 0x55a25b6eab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.483+0000 7f6fdfc00640 1 -- 172.21.15.90:0/4204169444 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a25b673680 con 0x55a25b647c00 2024-04-25T01:18:25.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.483+0000 7f6fdf200640 1 --2- 172.21.15.90:0/4204169444 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a25b646c00 0x55a25b6ea580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:25.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.483+0000 7f6fdfc00640 1 --2- 172.21.15.90:0/4204169444 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a25b647c00 0x55a25b6eb080 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x55a25c455d10 tx=0x55a25c3e2180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a446144b58ad9f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.483+0000 7f6fdc000640 1 -- 172.21.15.90:0/4204169444 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55a25b66c680 con 0x55a25b647c00 2024-04-25T01:18:25.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.483+0000 7f6fdc000640 1 -- 172.21.15.90:0/4204169444 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55a25b690c40 con 0x55a25b647c00 2024-04-25T01:18:25.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.483+0000 7f6fdc000640 1 -- 172.21.15.90:0/4204169444 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55a25b66cb60 con 0x55a25b647c00 2024-04-25T01:18:25.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.483+0000 7f6fe1c17740 1 -- 172.21.15.90:0/4204169444 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a25b673860 con 0x55a25b647c00 2024-04-25T01:18:25.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.483+0000 7f6fe1c17740 1 -- 172.21.15.90:0/4204169444 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) -- 0x55a25b6732c0 con 0x55a25b647c00 2024-04-25T01:18:25.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.483+0000 7f6fe1c17740 4 waiting for MDS map... 2024-04-25T01:18:25.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.484+0000 7f6fdc000640 1 -- 172.21.15.90:0/4204169444 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x55a25c512600 con 0x55a25b647c00 2024-04-25T01:18:25.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.484+0000 7f6fdc000640 1 -- 172.21.15.90:0/4204169444 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 23) ==== 4135+0+0 (secure 0 0 0) 0x55a25b69f180 con 0x55a25b647c00 2024-04-25T01:18:25.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.484+0000 7f6fe1c17740 4 Got MDS map 23 2024-04-25T01:18:25.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.484+0000 7f6fe1c17740 4 datascan.main: Using metadata pool output 2024-04-25T01:18:25.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.484+0000 7f6fe1c17740 4 datascan.main: connecting to RADOS... 2024-04-25T01:18:25.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.485+0000 7f6fe1c17740 1 Processor -- start 2024-04-25T01:18:25.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.485+0000 7f6fe1c17740 1 -- start start 2024-04-25T01:18:25.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.486+0000 7f6fe1c17740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a25c519c00 0x55a25b6eb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.486+0000 7f6fe1c17740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a25c570000 0x55a25b6ebb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.486+0000 7f6fe1c17740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a25c570400 0x55a25b6ec100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.486+0000 7f6fe1c17740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a25c3ae180 con 0x55a25c519c00 2024-04-25T01:18:25.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.486+0000 7f6fe1c17740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a25b6a6480 con 0x55a25c570000 2024-04-25T01:18:25.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.486+0000 7f6fe1c17740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55a25b6a6900 con 0x55a25c570400 2024-04-25T01:18:25.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.486+0000 7f6fde800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a25c519c00 0x55a25b6eb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.486+0000 7f6fde800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a25c519c00 0x55a25b6eb600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47662/0 (socket says 172.21.15.90:47662) 2024-04-25T01:18:25.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.486+0000 7f6fde800640 1 -- 172.21.15.90:0/2735276586 learned_addr learned my addr 172.21.15.90:0/2735276586 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:25.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.486+0000 7f6fde800640 1 -- 172.21.15.90:0/2735276586 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a25c570400 msgr2=0x55a25b6ec100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:25.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.486+0000 7f6fde800640 1 --2- 172.21.15.90:0/2735276586 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a25c570400 0x55a25b6ec100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.486+0000 7f6fde800640 1 -- 172.21.15.90:0/2735276586 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a25c570000 msgr2=0x55a25b6ebb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:25.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.486+0000 7f6fde800640 1 --2- 172.21.15.90:0/2735276586 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a25c570000 0x55a25b6ebb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.486+0000 7f6fde800640 1 -- 172.21.15.90:0/2735276586 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a25b6734a0 con 0x55a25c519c00 2024-04-25T01:18:25.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.487+0000 7f6fde800640 1 --2- 172.21.15.90:0/2735276586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a25c519c00 0x55a25b6eb600 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x55a25c511230 tx=0x55a25c5258c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0d044576f2fee79 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.487+0000 7f6fd9800640 1 -- 172.21.15.90:0/2735276586 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55a25b66c680 con 0x55a25c519c00 2024-04-25T01:18:25.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.487+0000 7f6fd9800640 1 -- 172.21.15.90:0/2735276586 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55a25b690a80 con 0x55a25c519c00 2024-04-25T01:18:25.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.487+0000 7f6fd9800640 1 -- 172.21.15.90:0/2735276586 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55a25b66cb60 con 0x55a25c519c00 2024-04-25T01:18:25.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.487+0000 7f6fe1c17740 1 -- 172.21.15.90:0/2735276586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a25c519c00 msgr2=0x55a25b6eb600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.487+0000 7f6fe1c17740 1 --2- 172.21.15.90:0/2735276586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a25c519c00 0x55a25b6eb600 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x55a25c511230 tx=0x55a25c5258c0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.487+0000 7f6fe1c17740 1 -- 172.21.15.90:0/2735276586 shutdown_connections 2024-04-25T01:18:25.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.487+0000 7f6fe1c17740 1 --2- 172.21.15.90:0/2735276586 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a25c570400 0x55a25b6ec100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.487+0000 7f6fe1c17740 1 --2- 172.21.15.90:0/2735276586 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a25c570000 0x55a25b6ebb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.487+0000 7f6fe1c17740 1 --2- 172.21.15.90:0/2735276586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a25c519c00 0x55a25b6eb600 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.487+0000 7f6fe1c17740 1 -- 172.21.15.90:0/2735276586 >> 172.21.15.90:0/2735276586 conn(0x55a25c519800 msgr2=0x55a25b732800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:25.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.487+0000 7f6fe1c17740 1 -- 172.21.15.90:0/2735276586 shutdown_connections 2024-04-25T01:18:25.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.487+0000 7f6fe1c17740 1 -- 172.21.15.90:0/2735276586 wait complete. 2024-04-25T01:18:25.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.488+0000 7f6fe1c17740 1 Processor -- start 2024-04-25T01:18:25.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.488+0000 7f6fe1c17740 1 -- start start 2024-04-25T01:18:25.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.489+0000 7f6fe1c17740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a25c519800 0x55a25b6ed180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.489+0000 7f6fe1c17740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a25c571000 0x55a25b6ecc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.489+0000 7f6fe1c17740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a25c570c00 0x55a25b6ec680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.489+0000 7f6fe1c17740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a25c3af800 con 0x55a25c571000 2024-04-25T01:18:25.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.489+0000 7f6fe1c17740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a25c3ae180 con 0x55a25c519800 2024-04-25T01:18:25.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.489+0000 7f6fe1c17740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55a25c3aed80 con 0x55a25c570c00 2024-04-25T01:18:25.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.489+0000 7f6fdf200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a25c571000 0x55a25b6ecc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.489+0000 7f6fdf200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a25c571000 0x55a25b6ecc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47668/0 (socket says 172.21.15.90:47668) 2024-04-25T01:18:25.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.489+0000 7f6fdf200640 1 -- 172.21.15.90:0/3796371542 learned_addr learned my addr 172.21.15.90:0/3796371542 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:25.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.490+0000 7f6fdf200640 1 -- 172.21.15.90:0/3796371542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a25c570c00 msgr2=0x55a25b6ec680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.490+0000 7f6fdf200640 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a25c570c00 0x55a25b6ec680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.490+0000 7f6fdf200640 1 -- 172.21.15.90:0/3796371542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a25c519800 msgr2=0x55a25b6ed180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.490+0000 7f6fdf200640 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a25c519800 0x55a25b6ed180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.490+0000 7f6fdf200640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a25b673680 con 0x55a25c571000 2024-04-25T01:18:25.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.490+0000 7f6fdf200640 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a25c571000 0x55a25b6ecc00 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x55a25c455170 tx=0x55a25c3de960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.490+0000 7f6fd7a00640 1 -- 172.21.15.90:0/3796371542 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55a25b66c9c0 con 0x55a25c571000 2024-04-25T01:18:25.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.490+0000 7f6fd7a00640 1 -- 172.21.15.90:0/3796371542 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55a25b690540 con 0x55a25c571000 2024-04-25T01:18:25.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.490+0000 7f6fd7a00640 1 -- 172.21.15.90:0/3796371542 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55a25b66c680 con 0x55a25c571000 2024-04-25T01:18:25.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.490+0000 7f6fe1c17740 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a25b673860 con 0x55a25c571000 2024-04-25T01:18:25.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.490+0000 7f6fe1c17740 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a25b6730e0 con 0x55a25c571000 2024-04-25T01:18:25.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.490+0000 7f6fe1c17740 4 datascan.init: forcing metadata pool 'recovery_fs_metadata' 2024-04-25T01:18:25.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.492+0000 7f6fd7a00640 1 -- 172.21.15.90:0/3796371542 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x55a25c564900 con 0x55a25c571000 2024-04-25T01:18:25.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.493+0000 7f6fd7a00640 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x55a25b9be000 0x55a25b6ed700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.493+0000 7f6fd7a00640 1 -- 172.21.15.90:0/3796371542 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x55a25c582c00 con 0x55a25c571000 2024-04-25T01:18:25.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.493+0000 7f6fe1c17740 4 datascan.main: data pool 'cephfs_data' has ID 6 2024-04-25T01:18:25.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.493+0000 7f6fe1c17740 4 datascan.main: opening data pool 'cephfs_data' 2024-04-25T01:18:25.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.493+0000 7f6fe1c17740 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x55a25c570800 0x55a25b6ec100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.493+0000 7f6fe1c17740 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:1 6.0 6:00000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c570400 con 0x55a25c570800 2024-04-25T01:18:25.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.493+0000 7f6fde800640 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x55a25b9be000 0x55a25b6ed700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.493+0000 7f6fde800640 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x55a25b9be000 0x55a25b6ed700 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x55a25c511380 tx=0x55a25c525800 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.494+0000 7f6fde800640 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x55a25c570800 0x55a25b6ec100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.494+0000 7f6fde800640 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x55a25c570800 0x55a25b6ec100 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.495+0000 7f6fde800640 1 -- 172.21.15.90:0/3796371542 <== osd.7 v2:172.21.15.103:6800/1145679154 1 ==== osd_op_reply(1 [pgnls start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc000 con 0x55a25c570800 2024-04-25T01:18:25.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.495+0000 7f6fd8e00640 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] conn(0x55a25b9be400 0x55a25b9d4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.496+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:2 6.20 6:04000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25b9be800 con 0x55a25b9be400 2024-04-25T01:18:25.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.496+0000 7f6fdfc00640 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] conn(0x55a25b9be400 0x55a25b9d4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.496+0000 7f6fdfc00640 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] conn(0x55a25b9be400 0x55a25b9d4000 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.497+0000 7f6fdfc00640 1 -- 172.21.15.90:0/3796371542 <== osd.2 v2:172.21.15.90:6808/906792737 1 ==== osd_op_reply(2 [pgnls start_epoch 50 out=49b] v42'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc500 con 0x55a25b9be400 2024-04-25T01:18:25.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.497+0000 7f6fd9800640 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x55a25b9bf000 0x55a25b9d4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.497+0000 7f6fd9800640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:3 6.10 6:08000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25b9bf400 con 0x55a25b9bf000 2024-04-25T01:18:25.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.498+0000 7f6fdf200640 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x55a25b9bf000 0x55a25b9d4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.498+0000 7f6fdf200640 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x55a25b9bf000 0x55a25b9d4580 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.499+0000 7f6fdf200640 1 -- 172.21.15.90:0/3796371542 <== osd.4 v2:172.21.15.103:6821/1971061680 1 ==== osd_op_reply(3 [pgnls start_epoch 50 out=49b] v42'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc780 con 0x55a25b9bf000 2024-04-25T01:18:25.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.499+0000 7f6fd8e00640 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x55a25b9bec00 0x55a25b9d4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.499+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:4 6.30 6:0c000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25b9bfc00 con 0x55a25b9bec00 2024-04-25T01:18:25.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.500+0000 7f6fde800640 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x55a25b9bec00 0x55a25b9d4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.500+0000 7f6fde800640 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x55a25b9bec00 0x55a25b9d4b00 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.501+0000 7f6fde800640 1 -- 172.21.15.90:0/3796371542 <== osd.1 v2:172.21.15.90:6800/2685473653 1 ==== osd_op_reply(4 [pgnls start_epoch 50 out=49b] v42'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc000 con 0x55a25b9bec00 2024-04-25T01:18:25.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.501+0000 7f6fd9800640 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x55a25b9bf800 0x55a25b9d5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.501+0000 7f6fd9800640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:5 6.8 6:10000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43c000 con 0x55a25b9bf800 2024-04-25T01:18:25.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.501+0000 7f6fdfc00640 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x55a25b9bf800 0x55a25b9d5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.501+0000 7f6fdfc00640 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x55a25b9bf800 0x55a25b9d5600 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.502+0000 7f6fdfc00640 1 -- 172.21.15.90:0/3796371542 <== osd.3 v2:172.21.15.90:6812/8683104 1 ==== osd_op_reply(5 [pgnls start_epoch 50 out=49b] v42'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc500 con 0x55a25b9bf800 2024-04-25T01:18:25.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.502+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:6 6.28 6:14000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43cc00 con 0x55a25b9bf000 2024-04-25T01:18:25.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.503+0000 7f6fdf200640 1 -- 172.21.15.90:0/3796371542 <== osd.4 v2:172.21.15.103:6821/1971061680 2 ==== osd_op_reply(6 [pgnls start_epoch 50 out=49b] v42'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc780 con 0x55a25b9bf000 2024-04-25T01:18:25.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.503+0000 7f6fd9800640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:7 6.18 6:18000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43c400 con 0x55a25b9bf000 2024-04-25T01:18:25.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.503+0000 7f6fdf200640 1 -- 172.21.15.90:0/3796371542 <== osd.4 v2:172.21.15.103:6821/1971061680 3 ==== osd_op_reply(7 [pgnls start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc780 con 0x55a25b9bf000 2024-04-25T01:18:25.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.503+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:8 6.38 6:1c000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43d000 con 0x55a25b9bf000 2024-04-25T01:18:25.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.504+0000 7f6fdf200640 1 -- 172.21.15.90:0/3796371542 <== osd.4 v2:172.21.15.103:6821/1971061680 4 ==== osd_op_reply(8 [pgnls start_epoch 50 out=49b] v42'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc780 con 0x55a25b9bf000 2024-04-25T01:18:25.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.505+0000 7f6fd9800640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:9 6.4 6:20000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43c800 con 0x55a25c570800 2024-04-25T01:18:25.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.506+0000 7f6fde800640 1 -- 172.21.15.90:0/3796371542 <== osd.7 v2:172.21.15.103:6800/1145679154 2 ==== osd_op_reply(9 [pgnls start_epoch 50 out=49b] v42'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc000 con 0x55a25c570800 2024-04-25T01:18:25.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.506+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:10 6.24 6:24000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43d400 con 0x55a25b9bec00 2024-04-25T01:18:25.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.506+0000 7f6fde800640 1 -- 172.21.15.90:0/3796371542 <== osd.1 v2:172.21.15.90:6800/2685473653 2 ==== osd_op_reply(10 [pgnls start_epoch 50 out=49b] v42'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc000 con 0x55a25b9bec00 2024-04-25T01:18:25.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.506+0000 7f6fd9800640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:11 6.14 6:28000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25b9bfc00 con 0x55a25b9bf000 2024-04-25T01:18:25.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.507+0000 7f6fdf200640 1 -- 172.21.15.90:0/3796371542 <== osd.4 v2:172.21.15.103:6821/1971061680 5 ==== osd_op_reply(11 [pgnls start_epoch 50 out=49b] v42'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc780 con 0x55a25b9bf000 2024-04-25T01:18:25.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.507+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:12 6.34 6:2c000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43d800 con 0x55a25c570800 2024-04-25T01:18:25.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.508+0000 7f6fde800640 1 -- 172.21.15.90:0/3796371542 <== osd.7 v2:172.21.15.103:6800/1145679154 3 ==== osd_op_reply(12 [pgnls start_epoch 50 out=49b] v42'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc000 con 0x55a25c570800 2024-04-25T01:18:25.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.508+0000 7f6fd9800640 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x55a25c423400 0x55a25b9d5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.508+0000 7f6fd9800640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:13 6.c 6:30000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c571c00 con 0x55a25c423400 2024-04-25T01:18:25.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.509+0000 7f6fdf200640 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x55a25c423400 0x55a25b9d5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.510+0000 7f6fdf200640 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x55a25c423400 0x55a25b9d5b80 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.511+0000 7f6fdf200640 1 -- 172.21.15.90:0/3796371542 <== osd.6 v2:172.21.15.103:6816/2008388906 1 ==== osd_op_reply(13 [pgnls start_epoch 50 out=49b] v42'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc780 con 0x55a25c423400 2024-04-25T01:18:25.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.511+0000 7f6fd8e00640 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x55a25c43c800 0x55a25b9d5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.511+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:14 6.2c 6:34000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43cc00 con 0x55a25c43c800 2024-04-25T01:18:25.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.512+0000 7f6fde800640 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x55a25c43c800 0x55a25b9d5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.512+0000 7f6fde800640 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x55a25c43c800 0x55a25b9d5080 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.513+0000 7f6fde800640 1 -- 172.21.15.90:0/3796371542 <== osd.0 v2:172.21.15.90:6816/2625481430 1 ==== osd_op_reply(14 [pgnls start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc000 con 0x55a25c43c800 2024-04-25T01:18:25.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.513+0000 7f6fd9800640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:15 6.1c 6:38000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c570400 con 0x55a25b9bf800 2024-04-25T01:18:25.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.513+0000 7f6fdfc00640 1 -- 172.21.15.90:0/3796371542 <== osd.3 v2:172.21.15.90:6812/8683104 2 ==== osd_op_reply(15 [pgnls start_epoch 50 out=49b] v42'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc500 con 0x55a25b9bf800 2024-04-25T01:18:25.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.513+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:16 6.3c 6:3c000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43c400 con 0x55a25b9bf000 2024-04-25T01:18:25.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.514+0000 7f6fdf200640 1 -- 172.21.15.90:0/3796371542 <== osd.4 v2:172.21.15.103:6821/1971061680 6 ==== osd_op_reply(16 [pgnls start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc780 con 0x55a25b9bf000 2024-04-25T01:18:25.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.514+0000 7f6fd9800640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:17 6.2 6:40000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43dc00 con 0x55a25b9bf800 2024-04-25T01:18:25.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.515+0000 7f6fdfc00640 1 -- 172.21.15.90:0/3796371542 <== osd.3 v2:172.21.15.90:6812/8683104 3 ==== osd_op_reply(17 [pgnls start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc500 con 0x55a25b9bf800 2024-04-25T01:18:25.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.515+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:18 6.22 6:44000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43d000 con 0x55a25b9bf800 2024-04-25T01:18:25.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.515+0000 7f6fdfc00640 1 -- 172.21.15.90:0/3796371542 <== osd.3 v2:172.21.15.90:6812/8683104 4 ==== osd_op_reply(18 [pgnls start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc500 con 0x55a25b9bf800 2024-04-25T01:18:25.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.515+0000 7f6fd9800640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:19 6.12 6:48000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43c000 con 0x55a25c43c800 2024-04-25T01:18:25.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.515+0000 7f6fde800640 1 -- 172.21.15.90:0/3796371542 <== osd.0 v2:172.21.15.90:6816/2625481430 2 ==== osd_op_reply(19 [pgnls start_epoch 50 out=49b] v42'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc000 con 0x55a25c43c800 2024-04-25T01:18:25.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.515+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:20 6.32 6:4c000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25b9bf400 con 0x55a25b9be400 2024-04-25T01:18:25.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.516+0000 7f6fdfc00640 1 -- 172.21.15.90:0/3796371542 <== osd.2 v2:172.21.15.90:6808/906792737 2 ==== osd_op_reply(20 [pgnls start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc500 con 0x55a25b9be400 2024-04-25T01:18:25.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.516+0000 7f6fd9800640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:21 6.a 6:50000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43dc00 con 0x55a25c43c800 2024-04-25T01:18:25.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.516+0000 7f6fde800640 1 -- 172.21.15.90:0/3796371542 <== osd.0 v2:172.21.15.90:6816/2625481430 3 ==== osd_op_reply(21 [pgnls start_epoch 50 out=49b] v42'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc000 con 0x55a25c43c800 2024-04-25T01:18:25.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.516+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:22 6.2a 6:54000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43d400 con 0x55a25c43c800 2024-04-25T01:18:25.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.516+0000 7f6fde800640 1 -- 172.21.15.90:0/3796371542 <== osd.0 v2:172.21.15.90:6816/2625481430 4 ==== osd_op_reply(22 [pgnls start_epoch 50 out=49b] v42'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc000 con 0x55a25c43c800 2024-04-25T01:18:25.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.516+0000 7f6fd9800640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:23 6.1a 6:58000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43d000 con 0x55a25b9bf800 2024-04-25T01:18:25.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.517+0000 7f6fdfc00640 1 -- 172.21.15.90:0/3796371542 <== osd.3 v2:172.21.15.90:6812/8683104 5 ==== osd_op_reply(23 [pgnls start_epoch 50 out=49b] v42'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc500 con 0x55a25b9bf800 2024-04-25T01:18:25.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.517+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:24 6.3a 6:5c000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43d800 con 0x55a25c423400 2024-04-25T01:18:25.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.518+0000 7f6fdf200640 1 -- 172.21.15.90:0/3796371542 <== osd.6 v2:172.21.15.103:6816/2008388906 2 ==== osd_op_reply(24 [pgnls start_epoch 50 out=49b] v42'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc780 con 0x55a25c423400 2024-04-25T01:18:25.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.518+0000 7f6fd9800640 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x55a25c570400 0x55a25b9d6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.518+0000 7f6fd9800640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- osd_op(unknown.0.0:25 6.6 6:60000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25b9be800 con 0x55a25c570400 2024-04-25T01:18:25.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.519+0000 7f6fdfc00640 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x55a25c570400 0x55a25b9d6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.520+0000 7f6fdfc00640 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x55a25c570400 0x55a25b9d6100 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.521+0000 7f6fdfc00640 1 -- 172.21.15.90:0/3796371542 <== osd.5 v2:172.21.15.103:6801/1348846168 1 ==== osd_op_reply(25 [pgnls start_epoch 50 out=49b] v42'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc500 con 0x55a25c570400 2024-04-25T01:18:25.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.522+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:26 6.26 6:64000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43cc00 con 0x55a25b9bf000 2024-04-25T01:18:25.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.522+0000 7f6fdf200640 1 -- 172.21.15.90:0/3796371542 <== osd.4 v2:172.21.15.103:6821/1971061680 7 ==== osd_op_reply(26 [pgnls start_epoch 50 out=49b] v42'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc780 con 0x55a25b9bf000 2024-04-25T01:18:25.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.522+0000 7f6fd9800640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:27 6.16 6:68000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25b9f8000 con 0x55a25b9be400 2024-04-25T01:18:25.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.523+0000 7f6fdfc00640 1 -- 172.21.15.90:0/3796371542 <== osd.2 v2:172.21.15.90:6808/906792737 3 ==== osd_op_reply(27 [pgnls start_epoch 50 out=49b] v42'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc500 con 0x55a25b9be400 2024-04-25T01:18:25.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.523+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:28 6.36 6:6c000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43c000 con 0x55a25b9bec00 2024-04-25T01:18:25.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.523+0000 7f6fde800640 1 -- 172.21.15.90:0/3796371542 <== osd.1 v2:172.21.15.90:6800/2685473653 3 ==== osd_op_reply(28 [pgnls start_epoch 50 out=49b] v42'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc000 con 0x55a25b9bec00 2024-04-25T01:18:25.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.523+0000 7f6fd9800640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:29 6.e 6:70000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25b9f8400 con 0x55a25b9bec00 2024-04-25T01:18:25.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.524+0000 7f6fde800640 1 -- 172.21.15.90:0/3796371542 <== osd.1 v2:172.21.15.90:6800/2685473653 4 ==== osd_op_reply(29 [pgnls start_epoch 50 out=49b] v42'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc000 con 0x55a25b9bec00 2024-04-25T01:18:25.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.524+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:30 6.2e 6:74000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43dc00 con 0x55a25b9bf800 2024-04-25T01:18:25.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.524+0000 7f6fdfc00640 1 -- 172.21.15.90:0/3796371542 <== osd.3 v2:172.21.15.90:6812/8683104 6 ==== osd_op_reply(30 [pgnls start_epoch 50 out=49b] v42'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc500 con 0x55a25b9bf800 2024-04-25T01:18:25.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.524+0000 7f6fd9800640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:31 6.1e 6:78000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25b9f8800 con 0x55a25b9bf800 2024-04-25T01:18:25.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.525+0000 7f6fdfc00640 1 -- 172.21.15.90:0/3796371542 <== osd.3 v2:172.21.15.90:6812/8683104 7 ==== osd_op_reply(31 [pgnls start_epoch 50 out=49b] v42'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc500 con 0x55a25b9bf800 2024-04-25T01:18:25.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.525+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:32 6.3e 6:7c000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25b9f8000 con 0x55a25c423400 2024-04-25T01:18:25.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.526+0000 7f6fdf200640 1 -- 172.21.15.90:0/3796371542 <== osd.6 v2:172.21.15.103:6816/2008388906 3 ==== osd_op_reply(32 [pgnls start_epoch 50 out=49b] v42'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc780 con 0x55a25c423400 2024-04-25T01:18:25.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.526+0000 7f6fd9800640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:33 6.1 6:80000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25b9f8c00 con 0x55a25c423400 2024-04-25T01:18:25.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.527+0000 7f6fdf200640 1 -- 172.21.15.90:0/3796371542 <== osd.6 v2:172.21.15.103:6816/2008388906 4 ==== osd_op_reply(33 [pgnls start_epoch 50 out=49b] v42'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc780 con 0x55a25c423400 2024-04-25T01:18:25.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.527+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:34 6.21 6:84000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25b9f8800 con 0x55a25b9bec00 2024-04-25T01:18:25.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.528+0000 7f6fde800640 1 -- 172.21.15.90:0/3796371542 <== osd.1 v2:172.21.15.90:6800/2685473653 5 ==== osd_op_reply(34 [pgnls start_epoch 50 out=49b] v42'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc000 con 0x55a25b9bec00 2024-04-25T01:18:25.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.528+0000 7f6fd9800640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:35 6.11 6:88000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25b9f9000 con 0x55a25c423400 2024-04-25T01:18:25.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.529+0000 7f6fdf200640 1 -- 172.21.15.90:0/3796371542 <== osd.6 v2:172.21.15.103:6816/2008388906 5 ==== osd_op_reply(35 [pgnls start_epoch 50 out=49b] v42'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc780 con 0x55a25c423400 2024-04-25T01:18:25.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.529+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:36 6.31 6:8c000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25b9f9c00 con 0x55a25b9bf800 2024-04-25T01:18:25.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.530+0000 7f6fdfc00640 1 -- 172.21.15.90:0/3796371542 <== osd.3 v2:172.21.15.90:6812/8683104 8 ==== osd_op_reply(36 [pgnls start_epoch 50 out=49b] v42'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc500 con 0x55a25b9bf800 2024-04-25T01:18:25.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.530+0000 7f6fd9800640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:37 6.9 6:90000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25b9f9400 con 0x55a25b9bf000 2024-04-25T01:18:25.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.531+0000 7f6fdf200640 1 -- 172.21.15.90:0/3796371542 <== osd.4 v2:172.21.15.103:6821/1971061680 8 ==== osd_op_reply(37 [pgnls start_epoch 50 out=49b] v42'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc780 con 0x55a25b9bf000 2024-04-25T01:18:25.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.531+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:38 6.29 6:94000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43d000 con 0x55a25b9bf800 2024-04-25T01:18:25.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.531+0000 7f6fdfc00640 1 -- 172.21.15.90:0/3796371542 <== osd.3 v2:172.21.15.90:6812/8683104 9 ==== osd_op_reply(38 [pgnls start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc500 con 0x55a25b9bf800 2024-04-25T01:18:25.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.531+0000 7f6fd9800640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:39 6.19 6:98000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25b9f9800 con 0x55a25c43c800 2024-04-25T01:18:25.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.532+0000 7f6fde800640 1 -- 172.21.15.90:0/3796371542 <== osd.0 v2:172.21.15.90:6816/2625481430 5 ==== osd_op_reply(39 [pgnls start_epoch 50 out=49b] v42'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc000 con 0x55a25c43c800 2024-04-25T01:18:25.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.532+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:40 6.39 6:9c000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43dc00 con 0x55a25b9bec00 2024-04-25T01:18:25.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.532+0000 7f6fde800640 1 -- 172.21.15.90:0/3796371542 <== osd.1 v2:172.21.15.90:6800/2685473653 6 ==== osd_op_reply(40 [pgnls start_epoch 50 out=49b] v42'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc000 con 0x55a25b9bec00 2024-04-25T01:18:25.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.532+0000 7f6fd9800640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:41 6.5 6:a0000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43d400 con 0x55a25c43c800 2024-04-25T01:18:25.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.533+0000 7f6fde800640 1 -- 172.21.15.90:0/3796371542 <== osd.0 v2:172.21.15.90:6816/2625481430 6 ==== osd_op_reply(41 [pgnls start_epoch 50 out=49b] v42'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc000 con 0x55a25c43c800 2024-04-25T01:18:25.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.533+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:42 6.25 6:a4000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25b9bf400 con 0x55a25c43c800 2024-04-25T01:18:25.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.534+0000 7f6fde800640 1 -- 172.21.15.90:0/3796371542 <== osd.0 v2:172.21.15.90:6816/2625481430 7 ==== osd_op_reply(42 [pgnls start_epoch 50 out=49b] v42'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc000 con 0x55a25c43c800 2024-04-25T01:18:25.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.534+0000 7f6fd9800640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:43 6.15 6:a8000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43c000 con 0x55a25b9be400 2024-04-25T01:18:25.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.534+0000 7f6fdfc00640 1 -- 172.21.15.90:0/3796371542 <== osd.2 v2:172.21.15.90:6808/906792737 4 ==== osd_op_reply(43 [pgnls start_epoch 50 out=49b] v42'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc500 con 0x55a25b9be400 2024-04-25T01:18:25.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.534+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:44 6.35 6:ac000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25b9be800 con 0x55a25c43c800 2024-04-25T01:18:25.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.535+0000 7f6fde800640 1 -- 172.21.15.90:0/3796371542 <== osd.0 v2:172.21.15.90:6816/2625481430 8 ==== osd_op_reply(44 [pgnls start_epoch 50 out=49b] v42'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc000 con 0x55a25c43c800 2024-04-25T01:18:25.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.535+0000 7f6fd9800640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:45 6.d 6:b0000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43dc00 con 0x55a25b9bec00 2024-04-25T01:18:25.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.536+0000 7f6fde800640 1 -- 172.21.15.90:0/3796371542 <== osd.1 v2:172.21.15.90:6800/2685473653 7 ==== osd_op_reply(45 [pgnls start_epoch 50 out=49b] v42'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc000 con 0x55a25b9bec00 2024-04-25T01:18:25.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.536+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- osd_op(unknown.0.0:46 6.2d 6:b4000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25b9f8400 con 0x55a25c570400 2024-04-25T01:18:25.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.537+0000 7f6fdfc00640 1 -- 172.21.15.90:0/3796371542 <== osd.5 v2:172.21.15.103:6801/1348846168 2 ==== osd_op_reply(46 [pgnls start_epoch 50 out=49b] v42'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc500 con 0x55a25c570400 2024-04-25T01:18:25.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.537+0000 7f6fd9800640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:47 6.1d 6:b8000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43c400 con 0x55a25c423400 2024-04-25T01:18:25.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.538+0000 7f6fdf200640 1 -- 172.21.15.90:0/3796371542 <== osd.6 v2:172.21.15.103:6816/2008388906 6 ==== osd_op_reply(47 [pgnls start_epoch 50 out=49b] v42'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc780 con 0x55a25c423400 2024-04-25T01:18:25.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.538+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:48 6.3d 6:bc000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25b9f8800 con 0x55a25b9bf000 2024-04-25T01:18:25.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.539+0000 7f6fdf200640 1 -- 172.21.15.90:0/3796371542 <== osd.4 v2:172.21.15.103:6821/1971061680 9 ==== osd_op_reply(48 [pgnls start_epoch 50 out=49b] v42'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc780 con 0x55a25b9bf000 2024-04-25T01:18:25.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.539+0000 7f6fd9800640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:49 6.3 6:c0000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43d800 con 0x55a25c43c800 2024-04-25T01:18:25.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.539+0000 7f6fde800640 1 -- 172.21.15.90:0/3796371542 <== osd.0 v2:172.21.15.90:6816/2625481430 9 ==== osd_op_reply(49 [pgnls start_epoch 50 out=49b] v42'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc000 con 0x55a25c43c800 2024-04-25T01:18:25.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.540+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:50 6.23 6:c4000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25b9f9800 con 0x55a25b9bf800 2024-04-25T01:18:25.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.540+0000 7f6fdfc00640 1 -- 172.21.15.90:0/3796371542 <== osd.3 v2:172.21.15.90:6812/8683104 10 ==== osd_op_reply(50 [pgnls start_epoch 50 out=49b] v42'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc500 con 0x55a25b9bf800 2024-04-25T01:18:25.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.540+0000 7f6fd9800640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:51 6.13 6:c8000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43cc00 con 0x55a25c570800 2024-04-25T01:18:25.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.541+0000 7f6fde800640 1 -- 172.21.15.90:0/3796371542 <== osd.7 v2:172.21.15.103:6800/1145679154 4 ==== osd_op_reply(51 [pgnls start_epoch 50 out=49b] v42'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc000 con 0x55a25c570800 2024-04-25T01:18:25.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.541+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:52 6.33 6:cc000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25b9f8000 con 0x55a25b9bec00 2024-04-25T01:18:25.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.542+0000 7f6fde800640 1 -- 172.21.15.90:0/3796371542 <== osd.1 v2:172.21.15.90:6800/2685473653 8 ==== osd_op_reply(52 [pgnls start_epoch 50 out=49b] v42'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc000 con 0x55a25b9bec00 2024-04-25T01:18:25.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.542+0000 7f6fd9800640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:53 6.b 6:d0000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c571c00 con 0x55a25b9bf000 2024-04-25T01:18:25.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.543+0000 7f6fdf200640 1 -- 172.21.15.90:0/3796371542 <== osd.4 v2:172.21.15.103:6821/1971061680 10 ==== osd_op_reply(53 [pgnls start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc780 con 0x55a25b9bf000 2024-04-25T01:18:25.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.543+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:54 6.2b 6:d4000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25b9f8000 con 0x55a25c43c800 2024-04-25T01:18:25.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.543+0000 7f6fde800640 1 -- 172.21.15.90:0/3796371542 <== osd.0 v2:172.21.15.90:6816/2625481430 10 ==== osd_op_reply(54 [pgnls start_epoch 50 out=49b] v42'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc000 con 0x55a25c43c800 2024-04-25T01:18:25.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.543+0000 7f6fd9800640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:55 6.1b 6:d8000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25b9bfc00 con 0x55a25c423400 2024-04-25T01:18:25.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.544+0000 7f6fdf200640 1 -- 172.21.15.90:0/3796371542 <== osd.6 v2:172.21.15.103:6816/2008388906 7 ==== osd_op_reply(55 [pgnls start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc780 con 0x55a25c423400 2024-04-25T01:18:25.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.544+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:56 6.3b 6:dc000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43d400 con 0x55a25c570800 2024-04-25T01:18:25.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.545+0000 7f6fde800640 1 -- 172.21.15.90:0/3796371542 <== osd.7 v2:172.21.15.103:6800/1145679154 5 ==== osd_op_reply(56 [pgnls start_epoch 50 out=49b] v42'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc000 con 0x55a25c570800 2024-04-25T01:18:25.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.545+0000 7f6fd9800640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:57 6.7 6:e0000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25b9bfc00 con 0x55a25b9bf800 2024-04-25T01:18:25.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.546+0000 7f6fdfc00640 1 -- 172.21.15.90:0/3796371542 <== osd.3 v2:172.21.15.90:6812/8683104 11 ==== osd_op_reply(57 [pgnls start_epoch 50 out=81b] v42'8 uv8 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x55a25c3cc500 con 0x55a25b9bf800 2024-04-25T01:18:25.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.546+0000 7f6fe1c17740 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:58 6.7 6:e2e07930:::10000000001.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50) -- 0x55a25c570000 con 0x55a25b9bf800 2024-04-25T01:18:25.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.547+0000 7f6fdfc00640 1 -- 172.21.15.90:0/3796371542 <== osd.3 v2:172.21.15.90:6812/8683104 12 ==== osd_op_reply(58 10000000001.00000000 [stat out=16b] v0'0 uv8 ondisk = 0) ==== 164+0+16 (crc 0 0 0) 0x55a25c3cc500 con 0x55a25b9bf800 2024-04-25T01:18:25.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.547+0000 7f6fe1c17740 10 datascan.operator(): handling object 1099511627777.0 2024-04-25T01:18:25.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.547+0000 7f6fe1c17740 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:59 6.7 6:e2e07930:::10000000001.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) -- 0x55a25c570000 con 0x55a25b9bf800 2024-04-25T01:18:25.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.550+0000 7f6fdfc00640 1 -- 172.21.15.90:0/3796371542 <== osd.3 v2:172.21.15.90:6812/8683104 13 ==== osd_op_reply(59 10000000001.00000000 [call] v50'9 uv9 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x55a25c3cc500 con 0x55a25b9bf800 2024-04-25T01:18:25.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.551+0000 7f6fe1c17740 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:60 6.27 6:e4000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25b9bfc00 con 0x55a25b9be400 2024-04-25T01:18:25.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.551+0000 7f6fdfc00640 1 -- 172.21.15.90:0/3796371542 <== osd.2 v2:172.21.15.90:6808/906792737 5 ==== osd_op_reply(60 [pgnls start_epoch 50 out=49b] v42'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc500 con 0x55a25b9be400 2024-04-25T01:18:25.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.551+0000 7f6fd9800640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:61 6.17 6:e8000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c571c00 con 0x55a25b9bec00 2024-04-25T01:18:25.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.552+0000 7f6fde800640 1 -- 172.21.15.90:0/3796371542 <== osd.1 v2:172.21.15.90:6800/2685473653 9 ==== osd_op_reply(61 [pgnls start_epoch 50 out=49b] v42'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc000 con 0x55a25b9bec00 2024-04-25T01:18:25.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.552+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:62 6.37 6:ec000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43dc00 con 0x55a25c570800 2024-04-25T01:18:25.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.553+0000 7f6fde800640 1 -- 172.21.15.90:0/3796371542 <== osd.7 v2:172.21.15.103:6800/1145679154 6 ==== osd_op_reply(62 [pgnls start_epoch 50 out=49b] v42'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc000 con 0x55a25c570800 2024-04-25T01:18:25.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.553+0000 7f6fd9800640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- osd_op(unknown.0.0:63 6.f 6:f0000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25b9f8c00 con 0x55a25c570400 2024-04-25T01:18:25.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.555+0000 7f6fdfc00640 1 -- 172.21.15.90:0/3796371542 <== osd.5 v2:172.21.15.103:6801/1348846168 3 ==== osd_op_reply(63 [pgnls start_epoch 50 out=49b] v42'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc500 con 0x55a25c570400 2024-04-25T01:18:25.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.555+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:64 6.2f 6:f4000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43d800 con 0x55a25b9bf800 2024-04-25T01:18:25.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.555+0000 7f6fdfc00640 1 -- 172.21.15.90:0/3796371542 <== osd.3 v2:172.21.15.90:6812/8683104 14 ==== osd_op_reply(64 [pgnls start_epoch 50 out=81b] v42'9 uv1 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x55a25c3cc500 con 0x55a25b9bf800 2024-04-25T01:18:25.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.555+0000 7f6fe1c17740 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:65 6.2f 6:f526278d:::10000000001.00000001:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50) -- 0x55a25b9f9c00 con 0x55a25b9bf800 2024-04-25T01:18:25.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.556+0000 7f6fdfc00640 1 -- 172.21.15.90:0/3796371542 <== osd.3 v2:172.21.15.90:6812/8683104 15 ==== osd_op_reply(65 10000000001.00000001 [stat out=16b] v0'0 uv1 ondisk = 0) ==== 164+0+16 (crc 0 0 0) 0x55a25c3cc500 con 0x55a25b9bf800 2024-04-25T01:18:25.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.556+0000 7f6fe1c17740 10 datascan.operator(): handling object 1099511627777.1 2024-04-25T01:18:25.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.556+0000 7f6fe1c17740 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:66 6.7 6:e2e07930:::10000000001.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) -- 0x55a25b9f8400 con 0x55a25b9bf800 2024-04-25T01:18:25.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.559+0000 7f6fdfc00640 1 -- 172.21.15.90:0/3796371542 <== osd.3 v2:172.21.15.90:6812/8683104 16 ==== osd_op_reply(66 10000000001.00000000 [call] v50'10 uv10 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x55a25c3cc500 con 0x55a25b9bf800 2024-04-25T01:18:25.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.559+0000 7f6fe1c17740 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:67 6.1f 6:f8000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25b9f9800 con 0x55a25c423400 2024-04-25T01:18:25.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.560+0000 7f6fdf200640 1 -- 172.21.15.90:0/3796371542 <== osd.6 v2:172.21.15.103:6816/2008388906 8 ==== osd_op_reply(67 [pgnls start_epoch 50 out=49b] v42'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc780 con 0x55a25c423400 2024-04-25T01:18:25.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.560+0000 7f6fd8e00640 1 -- 172.21.15.90:0/3796371542 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:68 6.3f 6:fc000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x55a25c43cc00 con 0x55a25b9bf000 2024-04-25T01:18:25.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.562+0000 7f6fdf200640 1 -- 172.21.15.90:0/3796371542 <== osd.4 v2:172.21.15.103:6821/1971061680 11 ==== osd_op_reply(68 [pgnls start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55a25c3cc780 con 0x55a25b9bf000 2024-04-25T01:18:25.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.562+0000 7f6fe1c17740 1 -- 172.21.15.90:0/3796371542 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x55a25c43c800 msgr2=0x55a25b9d5080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.562+0000 7f6fe1c17740 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x55a25c43c800 0x55a25b9d5080 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.562+0000 7f6fe1c17740 1 -- 172.21.15.90:0/3796371542 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x55a25b9bec00 msgr2=0x55a25b9d4b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.562+0000 7f6fe1c17740 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x55a25b9bec00 0x55a25b9d4b00 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.562+0000 7f6fe1c17740 1 -- 172.21.15.90:0/3796371542 >> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] conn(0x55a25b9be400 msgr2=0x55a25b9d4000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.562+0000 7f6fe1c17740 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] conn(0x55a25b9be400 0x55a25b9d4000 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.562+0000 7f6fe1c17740 1 -- 172.21.15.90:0/3796371542 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x55a25b9bf800 msgr2=0x55a25b9d5600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.562+0000 7f6fe1c17740 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x55a25b9bf800 0x55a25b9d5600 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.562+0000 7f6fdf200640 1 -- 172.21.15.90:0/3796371542 reap_dead start 2024-04-25T01:18:25.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.562+0000 7f6fe1c17740 1 -- 172.21.15.90:0/3796371542 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x55a25b9bf000 msgr2=0x55a25b9d4580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.562+0000 7f6fe1c17740 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x55a25b9bf000 0x55a25b9d4580 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.562+0000 7f6fe1c17740 1 -- 172.21.15.90:0/3796371542 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x55a25c570400 msgr2=0x55a25b9d6100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.562+0000 7f6fdf200640 1 -- 172.21.15.90:0/3796371542 reap_dead start 2024-04-25T01:18:25.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.562+0000 7f6fe1c17740 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x55a25c570400 0x55a25b9d6100 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.562+0000 7f6fe1c17740 1 -- 172.21.15.90:0/3796371542 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x55a25c423400 msgr2=0x55a25b9d5b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.562+0000 7f6fe1c17740 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x55a25c423400 0x55a25b9d5b80 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.562+0000 7f6fe1c17740 1 -- 172.21.15.90:0/3796371542 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x55a25c570800 msgr2=0x55a25b6ec100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.562+0000 7f6fe1c17740 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x55a25c570800 0x55a25b6ec100 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.563+0000 7f6fe1c17740 1 -- 172.21.15.90:0/3796371542 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x55a25b9be000 msgr2=0x55a25b6ed700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.563+0000 7f6fe1c17740 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x55a25b9be000 0x55a25b6ed700 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x55a25c511380 tx=0x55a25c525800 comp rx=0 tx=0).stop 2024-04-25T01:18:25.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.563+0000 7f6fe1c17740 1 -- 172.21.15.90:0/3796371542 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a25c571000 msgr2=0x55a25b6ecc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.563+0000 7f6fe1c17740 1 --2- 172.21.15.90:0/3796371542 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a25c571000 0x55a25b6ecc00 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x55a25c455170 tx=0x55a25c3de960 comp rx=0 tx=0).stop 2024-04-25T01:18:25.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.563+0000 7f6fdf200640 1 -- 172.21.15.90:0/3796371542 reap_dead start 2024-04-25T01:18:25.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.563+0000 7f6fe1c17740 1 -- 172.21.15.90:0/3796371542 shutdown_connections 2024-04-25T01:18:25.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.563+0000 7f6fe1c17740 1 -- 172.21.15.90:0/3796371542 >> 172.21.15.90:0/3796371542 conn(0x55a25c519c00 msgr2=0x55a25b732800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:25.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.563+0000 7f6fe1c17740 1 -- 172.21.15.90:0/3796371542 shutdown_connections 2024-04-25T01:18:25.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.563+0000 7f6fe1c17740 1 -- 172.21.15.90:0/3796371542 wait complete. 2024-04-25T01:18:25.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.564+0000 7f6fe1c17740 1 -- 172.21.15.90:0/4204169444 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a25b647c00 msgr2=0x55a25b6eb080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.564+0000 7f6fe1c17740 1 --2- 172.21.15.90:0/4204169444 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a25b647c00 0x55a25b6eb080 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x55a25c455d10 tx=0x55a25c3e2180 comp rx=0 tx=0).stop 2024-04-25T01:18:25.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.564+0000 7f6fe1c17740 1 -- 172.21.15.90:0/4204169444 shutdown_connections 2024-04-25T01:18:25.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.564+0000 7f6fe1c17740 1 --2- 172.21.15.90:0/4204169444 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a25b647c00 0x55a25b6eb080 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.564+0000 7f6fe1c17740 1 --2- 172.21.15.90:0/4204169444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a25b647800 0x55a25b6eab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.564+0000 7f6fe1c17740 1 --2- 172.21.15.90:0/4204169444 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55a25b646c00 0x55a25b6ea580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.565+0000 7f6fe1c17740 1 -- 172.21.15.90:0/4204169444 >> 172.21.15.90:0/4204169444 conn(0x55a25b647000 msgr2=0x55a25b733000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:25.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.565+0000 7f6fe1c17740 1 -- 172.21.15.90:0/4204169444 shutdown_connections 2024-04-25T01:18:25.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.565+0000 7f6fe1c17740 1 -- 172.21.15.90:0/4204169444 wait complete. 2024-04-25T01:18:25.561 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_extents', '--alternate-pool', 'recovery_fs_metadata', '--filesystem', 'cephfs', 'cephfs_data'] in time 0:00:00.179285, result: 2024-04-25T01:18:25.561 DEBUG:teuthology.orchestra.run.smithi090:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_inodes --alternate-pool recovery_fs_metadata --filesystem cephfs --force-corrupt --force-init cephfs_data 2024-04-25T01:18:25.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.688+0000 7fa96f41a740 1 -- start start 2024-04-25T01:18:25.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.688+0000 7fa96f41a740 1 Processor -- start 2024-04-25T01:18:25.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.689+0000 7fa96f41a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563023ed6400 0x563023e05600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.689+0000 7fa96f41a740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563023ed6000 0x563023e04b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.689+0000 7fa96f41a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563023d61800 0x563023e05b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.689+0000 7fa96f41a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563023edc480 con 0x563023d61800 2024-04-25T01:18:25.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.689+0000 7fa96f41a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563023edc600 con 0x563023ed6400 2024-04-25T01:18:25.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.689+0000 7fa96f41a740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x563023edc780 con 0x563023ed6000 2024-04-25T01:18:25.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.689+0000 7fa96d400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563023d61800 0x563023e05b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.689+0000 7fa96d400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563023d61800 0x563023e05b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47680/0 (socket says 172.21.15.90:47680) 2024-04-25T01:18:25.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.690+0000 7fa96d400640 1 -- 172.21.15.90:0/1725878873 learned_addr learned my addr 172.21.15.90:0/1725878873 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:25.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.690+0000 7fa96c000640 1 --2- 172.21.15.90:0/1725878873 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563023ed6000 0x563023e04b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.690+0000 7fa96ca00640 1 --2- 172.21.15.90:0/1725878873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563023ed6400 0x563023e05600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.690+0000 7fa96d400640 1 -- 172.21.15.90:0/1725878873 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563023ed6000 msgr2=0x563023e04b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:25.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.690+0000 7fa96d400640 1 --2- 172.21.15.90:0/1725878873 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563023ed6000 0x563023e04b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.690+0000 7fa96d400640 1 -- 172.21.15.90:0/1725878873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563023ed6400 msgr2=0x563023e05600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:25.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.690+0000 7fa96d400640 1 --2- 172.21.15.90:0/1725878873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563023ed6400 0x563023e05600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.690+0000 7fa96d400640 1 -- 172.21.15.90:0/1725878873 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563023d8d680 con 0x563023d61800 2024-04-25T01:18:25.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.690+0000 7fa96d400640 1 --2- 172.21.15.90:0/1725878873 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563023d61800 0x563023e05b80 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x563024b83b30 tx=0x563024afc2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f47ebbd986c6664c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.690+0000 7fa969800640 1 -- 172.21.15.90:0/1725878873 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x563023d869c0 con 0x563023d61800 2024-04-25T01:18:25.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.690+0000 7fa969800640 1 -- 172.21.15.90:0/1725878873 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x563023daa1c0 con 0x563023d61800 2024-04-25T01:18:25.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.690+0000 7fa969800640 1 -- 172.21.15.90:0/1725878873 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x563023d86b60 con 0x563023d61800 2024-04-25T01:18:25.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.691+0000 7fa96f41a740 1 -- 172.21.15.90:0/1725878873 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563023d8d2c0 con 0x563023d61800 2024-04-25T01:18:25.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.691+0000 7fa96f41a740 1 -- 172.21.15.90:0/1725878873 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) -- 0x563023d8da40 con 0x563023d61800 2024-04-25T01:18:25.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.691+0000 7fa96f41a740 4 waiting for MDS map... 2024-04-25T01:18:25.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.691+0000 7fa969800640 1 -- 172.21.15.90:0/1725878873 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x563024b6e600 con 0x563023d61800 2024-04-25T01:18:25.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.691+0000 7fa969800640 1 -- 172.21.15.90:0/1725878873 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 23) ==== 4135+0+0 (secure 0 0 0) 0x563023db9180 con 0x563023d61800 2024-04-25T01:18:25.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.691+0000 7fa96f41a740 4 Got MDS map 23 2024-04-25T01:18:25.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.691+0000 7fa96f41a740 4 datascan.main: Using metadata pool output 2024-04-25T01:18:25.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.691+0000 7fa96f41a740 4 datascan.main: connecting to RADOS... 2024-04-25T01:18:25.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.692+0000 7fa96f41a740 1 Processor -- start 2024-04-25T01:18:25.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.692+0000 7fa96f41a740 1 -- start start 2024-04-25T01:18:25.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.693+0000 7fa96f41a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563023d61c00 0x563023e06100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.693+0000 7fa96f41a740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563024c2dc00 0x563023e06680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.693+0000 7fa96f41a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563024c90000 0x563023e06c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.693+0000 7fa96f41a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563023dc0180 con 0x563023d61c00 2024-04-25T01:18:25.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.693+0000 7fa96f41a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563023edd680 con 0x563024c90000 2024-04-25T01:18:25.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.693+0000 7fa96f41a740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x563023edd500 con 0x563024c2dc00 2024-04-25T01:18:25.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.693+0000 7fa96c000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563023d61c00 0x563023e06100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.693+0000 7fa96c000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563023d61c00 0x563023e06100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47692/0 (socket says 172.21.15.90:47692) 2024-04-25T01:18:25.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.693+0000 7fa96c000640 1 -- 172.21.15.90:0/2843893971 learned_addr learned my addr 172.21.15.90:0/2843893971 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:25.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.694+0000 7fa96c000640 1 -- 172.21.15.90:0/2843893971 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563024c2dc00 msgr2=0x563023e06680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:25.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.694+0000 7fa96c000640 1 --2- 172.21.15.90:0/2843893971 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563024c2dc00 0x563023e06680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.694+0000 7fa96c000640 1 -- 172.21.15.90:0/2843893971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563024c90000 msgr2=0x563023e06c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:25.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.694+0000 7fa96c000640 1 --2- 172.21.15.90:0/2843893971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563024c90000 0x563023e06c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.694+0000 7fa96c000640 1 -- 172.21.15.90:0/2843893971 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563023d8dc20 con 0x563023d61c00 2024-04-25T01:18:25.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.694+0000 7fa96c000640 1 --2- 172.21.15.90:0/2843893971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563023d61c00 0x563023e06100 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x563024c3cae0 tx=0x563024afc120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26612989d05808e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.694+0000 7fa967000640 1 -- 172.21.15.90:0/2843893971 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x563023d86680 con 0x563023d61c00 2024-04-25T01:18:25.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.694+0000 7fa967000640 1 -- 172.21.15.90:0/2843893971 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x563023daae00 con 0x563023d61c00 2024-04-25T01:18:25.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.694+0000 7fa967000640 1 -- 172.21.15.90:0/2843893971 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x563023d869c0 con 0x563023d61c00 2024-04-25T01:18:25.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.695+0000 7fa96f41a740 1 -- 172.21.15.90:0/2843893971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563023d61c00 msgr2=0x563023e06100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.695+0000 7fa96f41a740 1 --2- 172.21.15.90:0/2843893971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563023d61c00 0x563023e06100 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x563024c3cae0 tx=0x563024afc120 comp rx=0 tx=0).stop 2024-04-25T01:18:25.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.695+0000 7fa96f41a740 1 -- 172.21.15.90:0/2843893971 shutdown_connections 2024-04-25T01:18:25.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.695+0000 7fa96f41a740 1 --2- 172.21.15.90:0/2843893971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563024c90000 0x563023e06c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.695+0000 7fa96f41a740 1 --2- 172.21.15.90:0/2843893971 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563024c2dc00 0x563023e06680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.695+0000 7fa96f41a740 1 --2- 172.21.15.90:0/2843893971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563023d61c00 0x563023e06100 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.695+0000 7fa96f41a740 1 -- 172.21.15.90:0/2843893971 >> 172.21.15.90:0/2843893971 conn(0x563023d60c00 msgr2=0x563024b01800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:25.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.695+0000 7fa96f41a740 1 -- 172.21.15.90:0/2843893971 shutdown_connections 2024-04-25T01:18:25.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.695+0000 7fa96f41a740 1 -- 172.21.15.90:0/2843893971 wait complete. 2024-04-25T01:18:25.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.695+0000 7fa96f41a740 1 Processor -- start 2024-04-25T01:18:25.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.696+0000 7fa96f41a740 1 -- start start 2024-04-25T01:18:25.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.696+0000 7fa96f41a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563023d61c00 0x563023e06100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.696+0000 7fa96f41a740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563024c2dc00 0x563023e06680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.696+0000 7fa96f41a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563024c92000 0x563023e06c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.696+0000 7fa96f41a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563023edc900 con 0x563023d61c00 2024-04-25T01:18:25.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.696+0000 7fa96f41a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563023edca80 con 0x563024c92000 2024-04-25T01:18:25.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.696+0000 7fa96f41a740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x563023edc180 con 0x563024c2dc00 2024-04-25T01:18:25.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.696+0000 7fa96c000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563023d61c00 0x563023e06100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.697+0000 7fa96c000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563023d61c00 0x563023e06100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47704/0 (socket says 172.21.15.90:47704) 2024-04-25T01:18:25.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.697+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 learned_addr learned my addr 172.21.15.90:0/1433430244 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:25.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.697+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563024c2dc00 msgr2=0x563023e06680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.697+0000 7fa96c000640 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563024c2dc00 0x563023e06680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.697+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563024c92000 msgr2=0x563023e06c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.697+0000 7fa96c000640 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563024c92000 0x563023e06c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.697+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563023d8cb40 con 0x563023d61c00 2024-04-25T01:18:25.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.697+0000 7fa96c000640 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563023d61c00 0x563023e06100 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x563024c3ce10 tx=0x563024afc000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.697+0000 7fa965200640 1 -- 172.21.15.90:0/1433430244 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x563023d86b60 con 0x563023d61c00 2024-04-25T01:18:25.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.697+0000 7fa965200640 1 -- 172.21.15.90:0/1433430244 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x563023daac40 con 0x563023d61c00 2024-04-25T01:18:25.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.697+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x563023d8c3c0 con 0x563023d61c00 2024-04-25T01:18:25.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.697+0000 7fa965200640 1 -- 172.21.15.90:0/1433430244 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x563023d864e0 con 0x563023d61c00 2024-04-25T01:18:25.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.698+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563023d8d4a0 con 0x563023d61c00 2024-04-25T01:18:25.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.698+0000 7fa96f41a740 4 datascan.init: forcing metadata pool 'recovery_fs_metadata' 2024-04-25T01:18:25.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.699+0000 7fa965200640 1 -- 172.21.15.90:0/1433430244 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x563024c80900 con 0x563023d61c00 2024-04-25T01:18:25.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.700+0000 7fa965200640 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x5630240ca000 0x5630240d0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.700+0000 7fa965200640 1 -- 172.21.15.90:0/1433430244 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x563023d5f000 con 0x563023d61c00 2024-04-25T01:18:25.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.700+0000 7fa96f41a740 4 datascan.main: data pool 'cephfs_data' has ID 6 2024-04-25T01:18:25.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.700+0000 7fa96f41a740 4 datascan.main: opening data pool 'cephfs_data' 2024-04-25T01:18:25.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.700+0000 7fa96f41a740 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x563024c92400 0x563023e07180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.700+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- osd_op(unknown.0.0:1 7.14 7:287034c4:::1.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50) -- 0x563024c92800 con 0x563024c92400 2024-04-25T01:18:25.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.700+0000 7fa96ca00640 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x5630240ca000 0x5630240d0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.701+0000 7fa96ca00640 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x5630240ca000 0x5630240d0000 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x563024afb170 tx=0x563024058ae0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.701+0000 7fa96c000640 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x563024c92400 0x563023e07180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.702+0000 7fa96c000640 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x563024c92400 0x563023e07180 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.702+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.5 v2:172.21.15.103:6801/1348846168 1 ==== osd_op_reply(1 1.00000000.inode [stat out=16b] v0'0 uv1 ondisk = 0) ==== 160+0+16 (crc 0 0 0) 0x563024ae6000 con 0x563024c92400 2024-04-25T01:18:25.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.703+0000 7fa96f41a740 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x563024c92c00 0x563023e07700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.703+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:2 7.1f 7:fe07bba1:::100.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50) -- 0x563024c93000 con 0x563024c92c00 2024-04-25T01:18:25.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.703+0000 7fa96d400640 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x563024c92c00 0x563023e07700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.703+0000 7fa96d400640 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x563024c92c00 0x563023e07700 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.704+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.0 v2:172.21.15.90:6816/2625481430 1 ==== osd_op_reply(2 100.00000000.inode [stat out=16b] v0'0 uv1 ondisk = 0) ==== 162+0+16 (crc 0 0 0) 0x563024ae6500 con 0x563024c92c00 2024-04-25T01:18:25.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.704+0000 7fa96f41a740 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x563024c93400 0x5630240d0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.704+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:3 6.0 6:00000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x563024c93800 con 0x563024c93400 2024-04-25T01:18:25.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.705+0000 7fa96ca00640 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x563024c93400 0x5630240d0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.705+0000 7fa96ca00640 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x563024c93400 0x5630240d0580 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.706+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.7 v2:172.21.15.103:6800/1145679154 1 ==== osd_op_reply(3 [pgnls-filter start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6780 con 0x563024c93400 2024-04-25T01:18:25.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.707+0000 7fa966600640 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] conn(0x5630240cb000 0x5630240d2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.707+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:4 6.20 6:04000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240cb400 con 0x5630240cb000 2024-04-25T01:18:25.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.707+0000 7fa96c000640 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] conn(0x5630240cb000 0x5630240d2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.707+0000 7fa96c000640 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] conn(0x5630240cb000 0x5630240d2100 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.708+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.2 v2:172.21.15.90:6808/906792737 1 ==== osd_op_reply(4 [pgnls-filter start_epoch 50 out=49b] v42'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x5630240cb000 2024-04-25T01:18:25.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.708+0000 7fa967000640 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x5630240cbc00 0x5630240d2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.708+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:5 6.10 6:08000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fc000 con 0x5630240cbc00 2024-04-25T01:18:25.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.708+0000 7fa96d400640 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x5630240cbc00 0x5630240d2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.709+0000 7fa96d400640 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x5630240cbc00 0x5630240d2680 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.710+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.4 v2:172.21.15.103:6821/1971061680 1 ==== osd_op_reply(5 [pgnls-filter start_epoch 50 out=49b] v42'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x5630240cbc00 2024-04-25T01:18:25.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.710+0000 7fa966600640 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x5630240cb800 0x5630240d2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.710+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:6 6.30 6:0c000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fc800 con 0x5630240cb800 2024-04-25T01:18:25.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.710+0000 7fa96ca00640 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x5630240cb800 0x5630240d2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.711+0000 7fa96ca00640 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x5630240cb800 0x5630240d2c00 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.711+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.1 v2:172.21.15.90:6800/2685473653 1 ==== osd_op_reply(6 [pgnls-filter start_epoch 50 out=49b] v42'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6780 con 0x5630240cb800 2024-04-25T01:18:25.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.711+0000 7fa967000640 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x5630240fc400 0x5630240d3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.711+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:7 6.8 6:10000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fd400 con 0x5630240fc400 2024-04-25T01:18:25.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.712+0000 7fa96c000640 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x5630240fc400 0x5630240d3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.712+0000 7fa96c000640 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x5630240fc400 0x5630240d3700 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.712+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.3 v2:172.21.15.90:6812/8683104 1 ==== osd_op_reply(7 [pgnls-filter start_epoch 50 out=49b] v42'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x5630240fc400 2024-04-25T01:18:25.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.712+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:8 6.28 6:14000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fcc00 con 0x5630240cbc00 2024-04-25T01:18:25.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.713+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.4 v2:172.21.15.103:6821/1971061680 2 ==== osd_op_reply(8 [pgnls-filter start_epoch 50 out=49b] v42'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x5630240cbc00 2024-04-25T01:18:25.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.713+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:9 6.18 6:18000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fd800 con 0x5630240cbc00 2024-04-25T01:18:25.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.714+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.4 v2:172.21.15.103:6821/1971061680 3 ==== osd_op_reply(9 [pgnls-filter start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x5630240cbc00 2024-04-25T01:18:25.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.714+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:10 6.38 6:1c000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fd000 con 0x5630240cbc00 2024-04-25T01:18:25.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.714+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.4 v2:172.21.15.103:6821/1971061680 4 ==== osd_op_reply(10 [pgnls-filter start_epoch 50 out=49b] v42'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x5630240cbc00 2024-04-25T01:18:25.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.714+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:11 6.4 6:20000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fdc00 con 0x563024c93400 2024-04-25T01:18:25.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.715+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.7 v2:172.21.15.103:6800/1145679154 2 ==== osd_op_reply(11 [pgnls-filter start_epoch 50 out=49b] v42'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6780 con 0x563024c93400 2024-04-25T01:18:25.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.715+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:12 6.24 6:24000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fc800 con 0x5630240cb800 2024-04-25T01:18:25.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.716+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.1 v2:172.21.15.90:6800/2685473653 2 ==== osd_op_reply(12 [pgnls-filter start_epoch 50 out=49b] v42'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6780 con 0x5630240cb800 2024-04-25T01:18:25.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.716+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:13 6.14 6:28000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410e000 con 0x5630240cbc00 2024-04-25T01:18:25.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.717+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.4 v2:172.21.15.103:6821/1971061680 5 ==== osd_op_reply(13 [pgnls-filter start_epoch 50 out=49b] v42'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x5630240cbc00 2024-04-25T01:18:25.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.717+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:14 6.34 6:2c000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fdc00 con 0x563024c93400 2024-04-25T01:18:25.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.718+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.7 v2:172.21.15.103:6800/1145679154 3 ==== osd_op_reply(14 [pgnls-filter start_epoch 50 out=49b] v42'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6780 con 0x563024c93400 2024-04-25T01:18:25.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.718+0000 7fa967000640 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x56302410e400 0x563024110000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.718+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:15 6.c 6:30000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410e800 con 0x56302410e400 2024-04-25T01:18:25.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.719+0000 7fa96d400640 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x56302410e400 0x563024110000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.720+0000 7fa96d400640 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x56302410e400 0x563024110000 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.720+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.6 v2:172.21.15.103:6816/2008388906 1 ==== osd_op_reply(15 [pgnls-filter start_epoch 50 out=49b] v42'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x56302410e400 2024-04-25T01:18:25.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.720+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:16 6.2c 6:34000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x563024c93800 con 0x563024c92c00 2024-04-25T01:18:25.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.721+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.0 v2:172.21.15.90:6816/2625481430 2 ==== osd_op_reply(16 [pgnls-filter start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x563024c92c00 2024-04-25T01:18:25.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.721+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:17 6.1c 6:38000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410ec00 con 0x5630240fc400 2024-04-25T01:18:25.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.721+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.3 v2:172.21.15.90:6812/8683104 2 ==== osd_op_reply(17 [pgnls-filter start_epoch 50 out=49b] v42'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x5630240fc400 2024-04-25T01:18:25.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.722+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:18 6.3c 6:3c000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240cac00 con 0x5630240cbc00 2024-04-25T01:18:25.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.723+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.4 v2:172.21.15.103:6821/1971061680 6 ==== osd_op_reply(18 [pgnls-filter start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x5630240cbc00 2024-04-25T01:18:25.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.723+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:19 6.2 6:40000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fc000 con 0x5630240fc400 2024-04-25T01:18:25.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.723+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.3 v2:172.21.15.90:6812/8683104 3 ==== osd_op_reply(19 [pgnls-filter start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x5630240fc400 2024-04-25T01:18:25.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.723+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:20 6.22 6:44000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410ec00 con 0x5630240fc400 2024-04-25T01:18:25.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.724+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.3 v2:172.21.15.90:6812/8683104 4 ==== osd_op_reply(20 [pgnls-filter start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x5630240fc400 2024-04-25T01:18:25.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.724+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:21 6.12 6:48000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fcc00 con 0x563024c92c00 2024-04-25T01:18:25.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.724+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.0 v2:172.21.15.90:6816/2625481430 3 ==== osd_op_reply(21 [pgnls-filter start_epoch 50 out=49b] v42'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x563024c92c00 2024-04-25T01:18:25.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.724+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:22 6.32 6:4c000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fd400 con 0x5630240cb000 2024-04-25T01:18:25.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.725+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.2 v2:172.21.15.90:6808/906792737 2 ==== osd_op_reply(22 [pgnls-filter start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x5630240cb000 2024-04-25T01:18:25.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.725+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:23 6.a 6:50000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fd800 con 0x563024c92c00 2024-04-25T01:18:25.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.725+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.0 v2:172.21.15.90:6816/2625481430 4 ==== osd_op_reply(23 [pgnls-filter start_epoch 50 out=49b] v42'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x563024c92c00 2024-04-25T01:18:25.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.725+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:24 6.2a 6:54000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fc000 con 0x563024c92c00 2024-04-25T01:18:25.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.726+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.0 v2:172.21.15.90:6816/2625481430 5 ==== osd_op_reply(24 [pgnls-filter start_epoch 50 out=49b] v42'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x563024c92c00 2024-04-25T01:18:25.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.726+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:25 6.1a 6:58000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fd000 con 0x5630240fc400 2024-04-25T01:18:25.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.726+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.3 v2:172.21.15.90:6812/8683104 5 ==== osd_op_reply(25 [pgnls-filter start_epoch 50 out=49b] v42'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x5630240fc400 2024-04-25T01:18:25.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.726+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:26 6.3a 6:5c000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240cb400 con 0x56302410e400 2024-04-25T01:18:25.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.727+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.6 v2:172.21.15.103:6816/2008388906 2 ==== osd_op_reply(26 [pgnls-filter start_epoch 50 out=49b] v42'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x56302410e400 2024-04-25T01:18:25.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.727+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- osd_op(unknown.0.0:27 6.6 6:60000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x563024c93000 con 0x563024c92400 2024-04-25T01:18:25.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.728+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.5 v2:172.21.15.103:6801/1348846168 2 ==== osd_op_reply(27 [pgnls-filter start_epoch 50 out=49b] v42'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x563024c92400 2024-04-25T01:18:25.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.728+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:28 6.26 6:64000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x563024c92800 con 0x5630240cbc00 2024-04-25T01:18:25.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.729+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.4 v2:172.21.15.103:6821/1971061680 7 ==== osd_op_reply(28 [pgnls-filter start_epoch 50 out=49b] v42'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x5630240cbc00 2024-04-25T01:18:25.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.729+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:29 6.16 6:68000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240cac00 con 0x5630240cb000 2024-04-25T01:18:25.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.730+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.2 v2:172.21.15.90:6808/906792737 3 ==== osd_op_reply(29 [pgnls-filter start_epoch 50 out=49b] v42'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x5630240cb000 2024-04-25T01:18:25.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.730+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:30 6.36 6:6c000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410e800 con 0x5630240cb800 2024-04-25T01:18:25.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.730+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.1 v2:172.21.15.90:6800/2685473653 3 ==== osd_op_reply(30 [pgnls-filter start_epoch 50 out=49b] v42'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6780 con 0x5630240cb800 2024-04-25T01:18:25.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.731+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:31 6.e 6:70000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240cb400 con 0x5630240cb800 2024-04-25T01:18:25.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.731+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.1 v2:172.21.15.90:6800/2685473653 4 ==== osd_op_reply(31 [pgnls-filter start_epoch 50 out=49b] v42'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6780 con 0x5630240cb800 2024-04-25T01:18:25.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.731+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:32 6.2e 6:74000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410e000 con 0x5630240fc400 2024-04-25T01:18:25.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.732+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.3 v2:172.21.15.90:6812/8683104 6 ==== osd_op_reply(32 [pgnls-filter start_epoch 50 out=49b] v42'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x5630240fc400 2024-04-25T01:18:25.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.732+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:33 6.1e 6:78000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fcc00 con 0x5630240fc400 2024-04-25T01:18:25.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.732+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.3 v2:172.21.15.90:6812/8683104 7 ==== osd_op_reply(33 [pgnls-filter start_epoch 50 out=49b] v42'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x5630240fc400 2024-04-25T01:18:25.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.732+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:34 6.3e 6:7c000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410f000 con 0x56302410e400 2024-04-25T01:18:25.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.733+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.6 v2:172.21.15.103:6816/2008388906 3 ==== osd_op_reply(34 [pgnls-filter start_epoch 50 out=49b] v42'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x56302410e400 2024-04-25T01:18:25.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.733+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:35 6.1 6:80000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fd800 con 0x56302410e400 2024-04-25T01:18:25.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.734+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.6 v2:172.21.15.103:6816/2008388906 4 ==== osd_op_reply(35 [pgnls-filter start_epoch 50 out=49b] v42'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x56302410e400 2024-04-25T01:18:25.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.734+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:36 6.21 6:84000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410f400 con 0x5630240cb800 2024-04-25T01:18:25.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.735+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.1 v2:172.21.15.90:6800/2685473653 5 ==== osd_op_reply(36 [pgnls-filter start_epoch 50 out=49b] v42'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6780 con 0x5630240cb800 2024-04-25T01:18:25.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.735+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:37 6.11 6:88000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fc000 con 0x56302410e400 2024-04-25T01:18:25.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.736+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.6 v2:172.21.15.103:6816/2008388906 5 ==== osd_op_reply(37 [pgnls-filter start_epoch 50 out=49b] v42'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x56302410e400 2024-04-25T01:18:25.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.736+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:38 6.31 6:8c000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410f800 con 0x5630240fc400 2024-04-25T01:18:25.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.737+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.3 v2:172.21.15.90:6812/8683104 8 ==== osd_op_reply(38 [pgnls-filter start_epoch 50 out=49b] v42'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x5630240fc400 2024-04-25T01:18:25.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.737+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:39 6.9 6:90000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x563024c93800 con 0x5630240cbc00 2024-04-25T01:18:25.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.738+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.4 v2:172.21.15.103:6821/1971061680 8 ==== osd_op_reply(39 [pgnls-filter start_epoch 50 out=49b] v42'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x5630240cbc00 2024-04-25T01:18:25.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.738+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:40 6.29 6:94000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410fc00 con 0x5630240fc400 2024-04-25T01:18:25.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.739+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.3 v2:172.21.15.90:6812/8683104 9 ==== osd_op_reply(40 [pgnls-filter start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x5630240fc400 2024-04-25T01:18:25.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.739+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:41 6.19 6:98000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240cb400 con 0x563024c92c00 2024-04-25T01:18:25.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.739+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.0 v2:172.21.15.90:6816/2625481430 6 ==== osd_op_reply(41 [pgnls-filter start_epoch 50 out=49b] v42'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x563024c92c00 2024-04-25T01:18:25.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.739+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:42 6.39 6:9c000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410e800 con 0x5630240cb800 2024-04-25T01:18:25.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.740+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.1 v2:172.21.15.90:6800/2685473653 6 ==== osd_op_reply(42 [pgnls-filter start_epoch 50 out=49b] v42'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6780 con 0x5630240cb800 2024-04-25T01:18:25.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.740+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:43 6.5 6:a0000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240cac00 con 0x563024c92c00 2024-04-25T01:18:25.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.741+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.0 v2:172.21.15.90:6816/2625481430 7 ==== osd_op_reply(43 [pgnls-filter start_epoch 50 out=49b] v42'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x563024c92c00 2024-04-25T01:18:25.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.741+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:44 6.25 6:a4000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410f400 con 0x563024c92c00 2024-04-25T01:18:25.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.741+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.0 v2:172.21.15.90:6816/2625481430 8 ==== osd_op_reply(44 [pgnls-filter start_epoch 50 out=49b] v42'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x563024c92c00 2024-04-25T01:18:25.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.741+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:45 6.15 6:a8000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x563024c93000 con 0x5630240cb000 2024-04-25T01:18:25.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.742+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.2 v2:172.21.15.90:6808/906792737 4 ==== osd_op_reply(45 [pgnls-filter start_epoch 50 out=49b] v42'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x5630240cb000 2024-04-25T01:18:25.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.742+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:46 6.35 6:ac000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410ec00 con 0x563024c92c00 2024-04-25T01:18:25.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.742+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.0 v2:172.21.15.90:6816/2625481430 9 ==== osd_op_reply(46 [pgnls-filter start_epoch 50 out=49b] v42'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x563024c92c00 2024-04-25T01:18:25.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.743+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:47 6.d 6:b0000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fc800 con 0x5630240cb800 2024-04-25T01:18:25.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.743+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.1 v2:172.21.15.90:6800/2685473653 7 ==== osd_op_reply(47 [pgnls-filter start_epoch 50 out=49b] v42'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6780 con 0x5630240cb800 2024-04-25T01:18:25.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.743+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- osd_op(unknown.0.0:48 6.2d 6:b4000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410e000 con 0x563024c92400 2024-04-25T01:18:25.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.745+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.5 v2:172.21.15.103:6801/1348846168 3 ==== osd_op_reply(48 [pgnls-filter start_epoch 50 out=49b] v42'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x563024c92400 2024-04-25T01:18:25.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.745+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:49 6.1d 6:b8000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fdc00 con 0x56302410e400 2024-04-25T01:18:25.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.746+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.6 v2:172.21.15.103:6816/2008388906 6 ==== osd_op_reply(49 [pgnls-filter start_epoch 50 out=49b] v42'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x56302410e400 2024-04-25T01:18:25.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.746+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:50 6.3d 6:bc000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410f000 con 0x5630240cbc00 2024-04-25T01:18:25.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.747+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.4 v2:172.21.15.103:6821/1971061680 9 ==== osd_op_reply(50 [pgnls-filter start_epoch 50 out=49b] v42'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x5630240cbc00 2024-04-25T01:18:25.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.747+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:51 6.3 6:c0000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fd400 con 0x563024c92c00 2024-04-25T01:18:25.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.748+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.0 v2:172.21.15.90:6816/2625481430 10 ==== osd_op_reply(51 [pgnls-filter start_epoch 50 out=49b] v42'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x563024c92c00 2024-04-25T01:18:25.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.748+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:52 6.23 6:c4000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410f400 con 0x5630240fc400 2024-04-25T01:18:25.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.749+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.3 v2:172.21.15.90:6812/8683104 10 ==== osd_op_reply(52 [pgnls-filter start_epoch 50 out=49b] v42'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x5630240fc400 2024-04-25T01:18:25.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.749+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:53 6.13 6:c8000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fd000 con 0x563024c93400 2024-04-25T01:18:25.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.750+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.7 v2:172.21.15.103:6800/1145679154 4 ==== osd_op_reply(53 [pgnls-filter start_epoch 50 out=49b] v42'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6780 con 0x563024c93400 2024-04-25T01:18:25.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.750+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:54 6.33 6:cc000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240cb400 con 0x5630240cb800 2024-04-25T01:18:25.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.751+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.1 v2:172.21.15.90:6800/2685473653 8 ==== osd_op_reply(54 [pgnls-filter start_epoch 50 out=49b] v42'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6780 con 0x5630240cb800 2024-04-25T01:18:25.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.751+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:55 6.b 6:d0000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302411c000 con 0x5630240cbc00 2024-04-25T01:18:25.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.752+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.4 v2:172.21.15.103:6821/1971061680 10 ==== osd_op_reply(55 [pgnls-filter start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x5630240cbc00 2024-04-25T01:18:25.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.752+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:56 6.2b 6:d4000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240cac00 con 0x563024c92c00 2024-04-25T01:18:25.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.753+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.0 v2:172.21.15.90:6816/2625481430 11 ==== osd_op_reply(56 [pgnls-filter start_epoch 50 out=49b] v42'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x563024c92c00 2024-04-25T01:18:25.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.753+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:57 6.1b 6:d8000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302411c400 con 0x56302410e400 2024-04-25T01:18:25.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.754+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.6 v2:172.21.15.103:6816/2008388906 7 ==== osd_op_reply(57 [pgnls-filter start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x56302410e400 2024-04-25T01:18:25.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.754+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:58 6.3b 6:dc000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x563024c92800 con 0x563024c93400 2024-04-25T01:18:25.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.755+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.7 v2:172.21.15.103:6800/1145679154 5 ==== osd_op_reply(58 [pgnls-filter start_epoch 50 out=49b] v42'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6780 con 0x563024c93400 2024-04-25T01:18:25.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.755+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:59 6.7 6:e0000000::::head [pgnls-filter start_epoch 50 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302411c800 con 0x5630240fc400 2024-04-25T01:18:25.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.756+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.3 v2:172.21.15.90:6812/8683104 11 ==== osd_op_reply(59 [pgnls-filter start_epoch 50 out=81b] v50'10 uv10 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x563024ae6000 con 0x5630240fc400 2024-04-25T01:18:25.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.757+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:60 6.0 6:00000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x563024c93c00 con 0x563024c93400 2024-04-25T01:18:25.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.758+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.7 v2:172.21.15.103:6800/1145679154 6 ==== osd_op_reply(60 [pgnls-filter start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6780 con 0x563024c93400 2024-04-25T01:18:25.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.758+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:61 6.20 6:04000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302411cc00 con 0x5630240cb000 2024-04-25T01:18:25.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.759+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.2 v2:172.21.15.90:6808/906792737 5 ==== osd_op_reply(61 [pgnls-filter start_epoch 50 out=49b] v42'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x5630240cb000 2024-04-25T01:18:25.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.759+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:62 6.10 6:08000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x563024c93800 con 0x5630240cbc00 2024-04-25T01:18:25.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.760+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.4 v2:172.21.15.103:6821/1971061680 11 ==== osd_op_reply(62 [pgnls-filter start_epoch 50 out=49b] v42'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x5630240cbc00 2024-04-25T01:18:25.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.760+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:63 6.30 6:0c000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302411d000 con 0x5630240cb800 2024-04-25T01:18:25.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.761+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.1 v2:172.21.15.90:6800/2685473653 9 ==== osd_op_reply(63 [pgnls-filter start_epoch 50 out=49b] v42'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6780 con 0x5630240cb800 2024-04-25T01:18:25.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.761+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:64 6.8 6:10000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fd800 con 0x5630240fc400 2024-04-25T01:18:25.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.762+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.3 v2:172.21.15.90:6812/8683104 12 ==== osd_op_reply(64 [pgnls-filter start_epoch 50 out=49b] v42'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x5630240fc400 2024-04-25T01:18:25.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.762+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:65 6.28 6:14000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302411d400 con 0x5630240cbc00 2024-04-25T01:18:25.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.763+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.4 v2:172.21.15.103:6821/1971061680 12 ==== osd_op_reply(65 [pgnls-filter start_epoch 50 out=49b] v42'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x5630240cbc00 2024-04-25T01:18:25.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.763+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:66 6.18 6:18000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fc000 con 0x5630240cbc00 2024-04-25T01:18:25.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.764+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.4 v2:172.21.15.103:6821/1971061680 13 ==== osd_op_reply(66 [pgnls-filter start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x5630240cbc00 2024-04-25T01:18:25.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.764+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:67 6.38 6:1c000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302411d800 con 0x5630240cbc00 2024-04-25T01:18:25.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.766+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.4 v2:172.21.15.103:6821/1971061680 14 ==== osd_op_reply(67 [pgnls-filter start_epoch 50 out=49b] v42'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x5630240cbc00 2024-04-25T01:18:25.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.766+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:68 6.4 6:20000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x563024c93800 con 0x563024c93400 2024-04-25T01:18:25.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.766+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.7 v2:172.21.15.103:6800/1145679154 7 ==== osd_op_reply(68 [pgnls-filter start_epoch 50 out=49b] v42'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6780 con 0x563024c93400 2024-04-25T01:18:25.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.766+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:69 6.24 6:24000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302411dc00 con 0x5630240cb800 2024-04-25T01:18:25.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.767+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.1 v2:172.21.15.90:6800/2685473653 10 ==== osd_op_reply(69 [pgnls-filter start_epoch 50 out=49b] v42'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6780 con 0x5630240cb800 2024-04-25T01:18:25.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.767+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:70 6.14 6:28000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x563024c92800 con 0x5630240cbc00 2024-04-25T01:18:25.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.769+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.4 v2:172.21.15.103:6821/1971061680 15 ==== osd_op_reply(70 [pgnls-filter start_epoch 50 out=49b] v42'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x5630240cbc00 2024-04-25T01:18:25.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.769+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:71 6.34 6:2c000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410ec00 con 0x563024c93400 2024-04-25T01:18:25.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.770+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.7 v2:172.21.15.103:6800/1145679154 8 ==== osd_op_reply(71 [pgnls-filter start_epoch 50 out=49b] v42'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6780 con 0x563024c93400 2024-04-25T01:18:25.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.770+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:72 6.c 6:30000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fdc00 con 0x56302410e400 2024-04-25T01:18:25.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.771+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.6 v2:172.21.15.103:6816/2008388906 8 ==== osd_op_reply(72 [pgnls-filter start_epoch 50 out=49b] v42'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x56302410e400 2024-04-25T01:18:25.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.771+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:73 6.2c 6:34000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410f000 con 0x563024c92c00 2024-04-25T01:18:25.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.772+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.0 v2:172.21.15.90:6816/2625481430 12 ==== osd_op_reply(73 [pgnls-filter start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x563024c92c00 2024-04-25T01:18:25.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.772+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:74 6.1c 6:38000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fd400 con 0x5630240fc400 2024-04-25T01:18:25.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.773+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.3 v2:172.21.15.90:6812/8683104 13 ==== osd_op_reply(74 [pgnls-filter start_epoch 50 out=49b] v42'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x5630240fc400 2024-04-25T01:18:25.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.773+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:75 6.3c 6:3c000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410e800 con 0x5630240cbc00 2024-04-25T01:18:25.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.774+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.4 v2:172.21.15.103:6821/1971061680 16 ==== osd_op_reply(75 [pgnls-filter start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x5630240cbc00 2024-04-25T01:18:25.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.774+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:76 6.2 6:40000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fc800 con 0x5630240fc400 2024-04-25T01:18:25.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.775+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.3 v2:172.21.15.90:6812/8683104 14 ==== osd_op_reply(76 [pgnls-filter start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x5630240fc400 2024-04-25T01:18:25.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.776+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:77 6.22 6:44000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410f800 con 0x5630240fc400 2024-04-25T01:18:25.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.776+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.3 v2:172.21.15.90:6812/8683104 15 ==== osd_op_reply(77 [pgnls-filter start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x5630240fc400 2024-04-25T01:18:25.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.777+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:78 6.12 6:48000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fd000 con 0x563024c92c00 2024-04-25T01:18:25.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.777+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.0 v2:172.21.15.90:6816/2625481430 13 ==== osd_op_reply(78 [pgnls-filter start_epoch 50 out=49b] v42'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x563024c92c00 2024-04-25T01:18:25.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.777+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:79 6.32 6:4c000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410fc00 con 0x5630240cb000 2024-04-25T01:18:25.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.778+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.2 v2:172.21.15.90:6808/906792737 6 ==== osd_op_reply(79 [pgnls-filter start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x5630240cb000 2024-04-25T01:18:25.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.778+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:80 6.a 6:50000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fcc00 con 0x563024c92c00 2024-04-25T01:18:25.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.779+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.0 v2:172.21.15.90:6816/2625481430 14 ==== osd_op_reply(80 [pgnls-filter start_epoch 50 out=49b] v42'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x563024c92c00 2024-04-25T01:18:25.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.779+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:81 6.2a 6:54000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410e000 con 0x563024c92c00 2024-04-25T01:18:25.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.780+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.0 v2:172.21.15.90:6816/2625481430 15 ==== osd_op_reply(81 [pgnls-filter start_epoch 50 out=49b] v42'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x563024c92c00 2024-04-25T01:18:25.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.780+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:82 6.1a 6:58000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240cac00 con 0x5630240fc400 2024-04-25T01:18:25.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.781+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.3 v2:172.21.15.90:6812/8683104 16 ==== osd_op_reply(82 [pgnls-filter start_epoch 50 out=49b] v42'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x5630240fc400 2024-04-25T01:18:25.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.781+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:83 6.3a 6:5c000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410f400 con 0x56302410e400 2024-04-25T01:18:25.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.783+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.6 v2:172.21.15.103:6816/2008388906 9 ==== osd_op_reply(83 [pgnls-filter start_epoch 50 out=49b] v42'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x56302410e400 2024-04-25T01:18:25.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.783+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- osd_op(unknown.0.0:84 6.6 6:60000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240cb400 con 0x563024c92400 2024-04-25T01:18:25.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.784+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.5 v2:172.21.15.103:6801/1348846168 4 ==== osd_op_reply(84 [pgnls-filter start_epoch 50 out=49b] v42'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x563024c92400 2024-04-25T01:18:25.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.784+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:85 6.26 6:64000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x563024c93c00 con 0x5630240cbc00 2024-04-25T01:18:25.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.785+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.4 v2:172.21.15.103:6821/1971061680 17 ==== osd_op_reply(85 [pgnls-filter start_epoch 50 out=49b] v42'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x5630240cbc00 2024-04-25T01:18:25.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.785+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:86 6.16 6:68000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fd800 con 0x5630240cb000 2024-04-25T01:18:25.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.786+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.2 v2:172.21.15.90:6808/906792737 7 ==== osd_op_reply(86 [pgnls-filter start_epoch 50 out=49b] v42'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x5630240cb000 2024-04-25T01:18:25.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.786+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:87 6.36 6:6c000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x563024c93000 con 0x5630240cb800 2024-04-25T01:18:25.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.787+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.1 v2:172.21.15.90:6800/2685473653 11 ==== osd_op_reply(87 [pgnls-filter start_epoch 50 out=49b] v42'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6780 con 0x5630240cb800 2024-04-25T01:18:25.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.787+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:88 6.e 6:70000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fd400 con 0x5630240cb800 2024-04-25T01:18:25.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.788+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.1 v2:172.21.15.90:6800/2685473653 12 ==== osd_op_reply(88 [pgnls-filter start_epoch 50 out=49b] v42'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6780 con 0x5630240cb800 2024-04-25T01:18:25.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.788+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:89 6.2e 6:74000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302411cc00 con 0x5630240fc400 2024-04-25T01:18:25.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.789+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.3 v2:172.21.15.90:6812/8683104 17 ==== osd_op_reply(89 [pgnls-filter start_epoch 50 out=49b] v42'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x5630240fc400 2024-04-25T01:18:25.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.789+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:90 6.1e 6:78000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fc800 con 0x5630240fc400 2024-04-25T01:18:25.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.790+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.3 v2:172.21.15.90:6812/8683104 18 ==== osd_op_reply(90 [pgnls-filter start_epoch 50 out=49b] v42'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x5630240fc400 2024-04-25T01:18:25.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.790+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:91 6.3e 6:7c000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302411d800 con 0x56302410e400 2024-04-25T01:18:25.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.791+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.6 v2:172.21.15.103:6816/2008388906 10 ==== osd_op_reply(91 [pgnls-filter start_epoch 50 out=49b] v42'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x56302410e400 2024-04-25T01:18:25.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.792+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:92 6.1 6:80000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fc000 con 0x56302410e400 2024-04-25T01:18:25.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.793+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.6 v2:172.21.15.103:6816/2008388906 11 ==== osd_op_reply(92 [pgnls-filter start_epoch 50 out=49b] v42'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x56302410e400 2024-04-25T01:18:25.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.793+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:93 6.21 6:84000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302411c000 con 0x5630240cb800 2024-04-25T01:18:25.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.794+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.1 v2:172.21.15.90:6800/2685473653 13 ==== osd_op_reply(93 [pgnls-filter start_epoch 50 out=49b] v42'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6780 con 0x5630240cb800 2024-04-25T01:18:25.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.794+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:94 6.11 6:88000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fdc00 con 0x56302410e400 2024-04-25T01:18:25.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.795+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.6 v2:172.21.15.103:6816/2008388906 12 ==== osd_op_reply(94 [pgnls-filter start_epoch 50 out=49b] v42'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x56302410e400 2024-04-25T01:18:25.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.795+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:95 6.31 6:8c000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302411c400 con 0x5630240fc400 2024-04-25T01:18:25.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.796+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.3 v2:172.21.15.90:6812/8683104 19 ==== osd_op_reply(95 [pgnls-filter start_epoch 50 out=49b] v42'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x5630240fc400 2024-04-25T01:18:25.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.796+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:96 6.9 6:90000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fd000 con 0x5630240cbc00 2024-04-25T01:18:25.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.798+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.4 v2:172.21.15.103:6821/1971061680 18 ==== osd_op_reply(96 [pgnls-filter start_epoch 50 out=49b] v42'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x5630240cbc00 2024-04-25T01:18:25.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.798+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:97 6.29 6:94000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410f800 con 0x5630240fc400 2024-04-25T01:18:25.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.798+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.3 v2:172.21.15.90:6812/8683104 20 ==== osd_op_reply(97 [pgnls-filter start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x5630240fc400 2024-04-25T01:18:25.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.798+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:98 6.19 6:98000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fcc00 con 0x563024c92c00 2024-04-25T01:18:25.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.799+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.0 v2:172.21.15.90:6816/2625481430 16 ==== osd_op_reply(98 [pgnls-filter start_epoch 50 out=49b] v42'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x563024c92c00 2024-04-25T01:18:25.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.799+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:99 6.39 6:9c000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410fc00 con 0x5630240cb800 2024-04-25T01:18:25.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.800+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.1 v2:172.21.15.90:6800/2685473653 14 ==== osd_op_reply(99 [pgnls-filter start_epoch 50 out=49b] v42'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6780 con 0x5630240cb800 2024-04-25T01:18:25.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.800+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:100 6.5 6:a0000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x563024c92800 con 0x563024c92c00 2024-04-25T01:18:25.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.801+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.0 v2:172.21.15.90:6816/2625481430 17 ==== osd_op_reply(100 [pgnls-filter start_epoch 50 out=49b] v42'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x563024c92c00 2024-04-25T01:18:25.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.802+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:101 6.25 6:a4000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410f000 con 0x563024c92c00 2024-04-25T01:18:25.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.802+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.0 v2:172.21.15.90:6816/2625481430 18 ==== osd_op_reply(101 [pgnls-filter start_epoch 50 out=49b] v42'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x563024c92c00 2024-04-25T01:18:25.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.803+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:102 6.15 6:a8000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240cac00 con 0x5630240cb000 2024-04-25T01:18:25.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.803+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.2 v2:172.21.15.90:6808/906792737 8 ==== osd_op_reply(102 [pgnls-filter start_epoch 50 out=49b] v42'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x5630240cb000 2024-04-25T01:18:25.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.804+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:103 6.35 6:ac000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410e800 con 0x563024c92c00 2024-04-25T01:18:25.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.805+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.0 v2:172.21.15.90:6816/2625481430 19 ==== osd_op_reply(103 [pgnls-filter start_epoch 50 out=49b] v42'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x563024c92c00 2024-04-25T01:18:25.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.805+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:104 6.d 6:b0000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240cb400 con 0x5630240cb800 2024-04-25T01:18:25.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.805+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.1 v2:172.21.15.90:6800/2685473653 15 ==== osd_op_reply(104 [pgnls-filter start_epoch 50 out=49b] v42'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6780 con 0x5630240cb800 2024-04-25T01:18:25.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.806+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- osd_op(unknown.0.0:105 6.2d 6:b4000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410e000 con 0x563024c92400 2024-04-25T01:18:25.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.807+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.5 v2:172.21.15.103:6801/1348846168 5 ==== osd_op_reply(105 [pgnls-filter start_epoch 50 out=49b] v42'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x563024c92400 2024-04-25T01:18:25.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.807+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:106 6.1d 6:b8000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302411d000 con 0x56302410e400 2024-04-25T01:18:25.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.808+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.6 v2:172.21.15.103:6816/2008388906 13 ==== osd_op_reply(106 [pgnls-filter start_epoch 50 out=49b] v42'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x56302410e400 2024-04-25T01:18:25.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.808+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:107 6.3d 6:bc000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410f400 con 0x5630240cbc00 2024-04-25T01:18:25.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.809+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.4 v2:172.21.15.103:6821/1971061680 19 ==== osd_op_reply(107 [pgnls-filter start_epoch 50 out=49b] v42'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x5630240cbc00 2024-04-25T01:18:25.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.809+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:108 6.3 6:c0000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302411dc00 con 0x563024c92c00 2024-04-25T01:18:25.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.811+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.0 v2:172.21.15.90:6816/2625481430 20 ==== osd_op_reply(108 [pgnls-filter start_epoch 50 out=49b] v42'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x563024c92c00 2024-04-25T01:18:25.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.811+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:109 6.23 6:c4000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410f000 con 0x5630240fc400 2024-04-25T01:18:25.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.812+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.3 v2:172.21.15.90:6812/8683104 21 ==== osd_op_reply(109 [pgnls-filter start_epoch 50 out=49b] v42'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x5630240fc400 2024-04-25T01:18:25.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.812+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:110 6.13 6:c8000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302411c000 con 0x563024c93400 2024-04-25T01:18:25.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.813+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.7 v2:172.21.15.103:6800/1145679154 9 ==== osd_op_reply(110 [pgnls-filter start_epoch 50 out=49b] v42'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6780 con 0x563024c93400 2024-04-25T01:18:25.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.813+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:111 6.33 6:cc000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410e800 con 0x5630240cb800 2024-04-25T01:18:25.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.814+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.1 v2:172.21.15.90:6800/2685473653 16 ==== osd_op_reply(111 [pgnls-filter start_epoch 50 out=49b] v42'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6780 con 0x5630240cb800 2024-04-25T01:18:25.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.814+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:112 6.b 6:d0000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302411d400 con 0x5630240cbc00 2024-04-25T01:18:25.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.815+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.4 v2:172.21.15.103:6821/1971061680 20 ==== osd_op_reply(112 [pgnls-filter start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x5630240cbc00 2024-04-25T01:18:25.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.815+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:113 6.2b 6:d4000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410f400 con 0x563024c92c00 2024-04-25T01:18:25.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.816+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.0 v2:172.21.15.90:6816/2625481430 21 ==== osd_op_reply(113 [pgnls-filter start_epoch 50 out=49b] v42'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x563024c92c00 2024-04-25T01:18:25.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.816+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:114 6.1b 6:d8000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302411c800 con 0x56302410e400 2024-04-25T01:18:25.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.817+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.6 v2:172.21.15.103:6816/2008388906 14 ==== osd_op_reply(114 [pgnls-filter start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x56302410e400 2024-04-25T01:18:25.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.817+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:115 6.3b 6:dc000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302411d800 con 0x563024c93400 2024-04-25T01:18:25.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.818+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.7 v2:172.21.15.103:6800/1145679154 10 ==== osd_op_reply(115 [pgnls-filter start_epoch 50 out=49b] v42'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6780 con 0x563024c93400 2024-04-25T01:18:25.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.818+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:116 6.7 6:e0000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x5630240fd400 con 0x5630240fc400 2024-04-25T01:18:25.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.819+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.3 v2:172.21.15.90:6812/8683104 22 ==== osd_op_reply(116 [pgnls-filter start_epoch 50 out=81b] v50'10 uv10 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x563024ae6000 con 0x5630240fc400 2024-04-25T01:18:25.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.819+0000 7fa96f41a740 20 datascan.forall_objects: OSD matched oid 10000000001.00000000 2024-04-25T01:18:25.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.819+0000 7fa96f41a740 10 datascan.operator(): handling object 10000000001.0 2024-04-25T01:18:25.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.820+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:117 6.7 6:e2e07930:::10000000001.00000000:head [getxattr scan_ceiling in=12b,getxattr scan_max_size in=13b,getxattr scan_max_mtime in=14b,getxattr scan_pool_id in=12b,getxattr parent in=6b,getxattr layout in=6b,getxattr symlink in=7b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50) -- 0x5630240fc800 con 0x5630240fc400 2024-04-25T01:18:25.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.821+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.3 v2:172.21.15.90:6812/8683104 23 ==== osd_op_reply(117 10000000001.00000000 [getxattr (22) out=22b,getxattr (8) out=8b,getxattr (8) out=8b,getxattr,getxattr (95) out=95b,getxattr (30) out=30b,getxattr] v0'0 uv10 ondisk = 0) ==== 416+0+163 (crc 0 0 0) 0x563024ae6000 con 0x5630240fc400 2024-04-25T01:18:25.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.821+0000 7fa96f41a740 20 datascan.operator(): loaded layout from xattr: pi: 6 os: 4194304 sc: 1 su: 4194304 2024-04-25T01:18:25.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.821+0000 7fa96f41a740 10 datascan.inject_with_backtrace: inode: 0x0x10000000001 2024-04-25T01:18:25.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.821+0000 7fa96f41a740 10 datascan.inject_with_backtrace: backptr: 0x0x10000000000/sixmegs 2024-04-25T01:18:25.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.821+0000 7fa96f41a740 20 datascan.get_frag_of: dirino=0x10000000000 target_dname=sixmegs 2024-04-25T01:18:25.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.821+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:118 7.10 7:0cf6ad0f:::10000000000.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50) -- 0x5630240fd400 con 0x563024c93400 2024-04-25T01:18:25.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.822+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.7 v2:172.21.15.103:6800/1145679154 11 ==== osd_op_reply(118 10000000000.00000000 [getxattr] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 164+0+0 (crc 0 0 0) 0x563024ae6780 con 0x563024c93400 2024-04-25T01:18:25.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.822+0000 7fa96f41a740 4 datascan.get_frag_of: Unexpected error on '10000000000.00000000': (2) No such file or directory 2024-04-25T01:18:25.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.822+0000 7fa96f41a740 20 datascan.inject_with_backtrace: don't know fragment for 0x0x10000000000/sixmegs, will insert to root 2024-04-25T01:18:25.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.822+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:119 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50) -- 0x56302410f800 con 0x563024c93400 2024-04-25T01:18:25.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.823+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.7 v2:172.21.15.103:6800/1145679154 12 ==== osd_op_reply(119 10000000000.00000000 [omap-get-header] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 164+0+0 (crc 0 0 0) 0x563024ae6780 con 0x563024c93400 2024-04-25T01:18:25.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.823+0000 7fa96f41a740 10 datascan.find_or_create_dirfrag: read_version = 0 2024-04-25T01:18:25.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.823+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:120 7.10 7:0cf6ad0f:::10000000000.00000000:head [create,omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) -- 0x56302410e000 con 0x563024c93400 2024-04-25T01:18:25.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.828+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.7 v2:172.21.15.103:6800/1145679154 13 ==== osd_op_reply(120 10000000000.00000000 [create,omap-set-header] v50'1 uv1 ondisk = 0) ==== 206+0+0 (crc 0 0 0) 0x563024ae6780 con 0x563024c93400 2024-04-25T01:18:25.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.828+0000 7fa96f41a740 10 datascan.find_or_create_dirfrag: Created dirfrag: 0x0x10000000000 2024-04-25T01:18:25.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.828+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:121 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50) -- 0x56302410f000 con 0x563024c93400 2024-04-25T01:18:25.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.829+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.7 v2:172.21.15.103:6800/1145679154 14 ==== osd_op_reply(121 10000000000.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv1 ondisk = 0) ==== 164+0+4 (crc 0 0 0) 0x563024ae6780 con 0x563024c93400 2024-04-25T01:18:25.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.829+0000 7fa96f41a740 20 datascan.read_dentry: oid=10000000000.00000000 dname=sixmegs frag=*, r=0 2024-04-25T01:18:25.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.829+0000 7fa96f41a740 20 datascan.read_dentry: sixmegs_head not found in result 2024-04-25T01:18:25.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.829+0000 7fa96f41a740 10 datascan.inject_with_backtrace: Linking inode 0x0x10000000001 at 0x0x10000000000/sixmegs with size=6291456 bytes 2024-04-25T01:18:25.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.829+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:122 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=508b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) -- 0x5630240cac00 con 0x563024c93400 2024-04-25T01:18:25.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.833+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.7 v2:172.21.15.103:6800/1145679154 15 ==== osd_op_reply(122 10000000000.00000000 [omap-set-vals] v50'2 uv2 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x563024ae6780 con 0x563024c93400 2024-04-25T01:18:25.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.833+0000 7fa96f41a740 20 datascan.inject_linkage: Injected dentry 0x0x10000000000/sixmegs pointing to 0x0x10000000001 2024-04-25T01:18:25.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.833+0000 7fa96f41a740 10 datascan.inject_with_backtrace: backptr: 0x0x1/subdir 2024-04-25T01:18:25.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.833+0000 7fa96f41a740 20 datascan.get_frag_of: dirino=0x1 target_dname=subdir 2024-04-25T01:18:25.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.833+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:123 7.1f 7:ff5b34d6:::1.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50) -- 0x56302411cc00 con 0x563024c92c00 2024-04-25T01:18:25.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.834+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.0 v2:172.21.15.90:6816/2625481430 22 ==== osd_op_reply(123 1.00000000 [getxattr] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 154+0+0 (crc 0 0 0) 0x563024ae6500 con 0x563024c92c00 2024-04-25T01:18:25.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.834+0000 7fa96f41a740 4 datascan.get_frag_of: Unexpected error on '1.00000000': (2) No such file or directory 2024-04-25T01:18:25.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.834+0000 7fa96f41a740 20 datascan.inject_with_backtrace: don't know fragment for 0x0x1/subdir, will insert to root 2024-04-25T01:18:25.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.834+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:124 7.1f 7:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50) -- 0x56302411c400 con 0x563024c92c00 2024-04-25T01:18:25.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.835+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.0 v2:172.21.15.90:6816/2625481430 23 ==== osd_op_reply(124 1.00000000 [omap-get-header] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 154+0+0 (crc 0 0 0) 0x563024ae6500 con 0x563024c92c00 2024-04-25T01:18:25.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.835+0000 7fa96f41a740 10 datascan.find_or_create_dirfrag: read_version = 0 2024-04-25T01:18:25.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.835+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:125 7.1f 7:ff5b34d6:::1.00000000:head [create,omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) -- 0x56302410fc00 con 0x563024c92c00 2024-04-25T01:18:25.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.840+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.0 v2:172.21.15.90:6816/2625481430 24 ==== osd_op_reply(125 1.00000000 [create,omap-set-header] v50'2 uv2 ondisk = 0) ==== 196+0+0 (crc 0 0 0) 0x563024ae6500 con 0x563024c92c00 2024-04-25T01:18:25.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.840+0000 7fa96f41a740 10 datascan.find_or_create_dirfrag: Created dirfrag: 0x0x1 2024-04-25T01:18:25.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.840+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:126 7.1f 7:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50) -- 0x56302410e800 con 0x563024c92c00 2024-04-25T01:18:25.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.841+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.0 v2:172.21.15.90:6816/2625481430 25 ==== osd_op_reply(126 1.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv2 ondisk = 0) ==== 154+0+4 (crc 0 0 0) 0x563024ae6500 con 0x563024c92c00 2024-04-25T01:18:25.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.841+0000 7fa96f41a740 20 datascan.read_dentry: oid=1.00000000 dname=subdir frag=*, r=0 2024-04-25T01:18:25.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.841+0000 7fa96f41a740 20 datascan.read_dentry: subdir_head not found in result 2024-04-25T01:18:25.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.841+0000 7fa96f41a740 10 datascan.inject_with_backtrace: Linking ancestor directory of inode 0x0x10000000000 at 0x0x1:subdir 2024-04-25T01:18:25.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.841+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:127 7.1f 7:ff5b34d6:::1.00000000:head [omap-set-vals in=507b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) -- 0x56302410f800 con 0x563024c92c00 2024-04-25T01:18:25.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.845+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.0 v2:172.21.15.90:6816/2625481430 26 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v50'3 uv3 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x563024ae6500 con 0x563024c92c00 2024-04-25T01:18:25.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.845+0000 7fa96f41a740 20 datascan.inject_linkage: Injected dentry 0x0x1/subdir pointing to 0x0x10000000000 2024-04-25T01:18:25.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.846+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:128 6.27 6:e4000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410e000 con 0x5630240cb000 2024-04-25T01:18:25.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.846+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.2 v2:172.21.15.90:6808/906792737 9 ==== osd_op_reply(128 [pgnls-filter start_epoch 50 out=49b] v42'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x5630240cb000 2024-04-25T01:18:25.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.847+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:129 6.17 6:e8000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x563024c93800 con 0x5630240cb800 2024-04-25T01:18:25.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.847+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.1 v2:172.21.15.90:6800/2685473653 17 ==== osd_op_reply(129 [pgnls-filter start_epoch 50 out=49b] v42'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6780 con 0x5630240cb800 2024-04-25T01:18:25.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.848+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:130 6.37 6:ec000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302411d000 con 0x563024c93400 2024-04-25T01:18:25.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.849+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 <== osd.7 v2:172.21.15.103:6800/1145679154 16 ==== osd_op_reply(130 [pgnls-filter start_epoch 50 out=49b] v42'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6780 con 0x563024c93400 2024-04-25T01:18:25.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.849+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- osd_op(unknown.0.0:131 6.f 6:f0000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x563024c93000 con 0x563024c92400 2024-04-25T01:18:25.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.850+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.5 v2:172.21.15.103:6801/1348846168 6 ==== osd_op_reply(131 [pgnls-filter start_epoch 50 out=49b] v42'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x563024c92400 2024-04-25T01:18:25.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.850+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:132 6.2f 6:f4000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x563024c93c00 con 0x5630240fc400 2024-04-25T01:18:25.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.851+0000 7fa96c000640 1 -- 172.21.15.90:0/1433430244 <== osd.3 v2:172.21.15.90:6812/8683104 24 ==== osd_op_reply(132 [pgnls-filter start_epoch 50 out=49b] v42'9 uv1 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6000 con 0x5630240fc400 2024-04-25T01:18:25.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.851+0000 7fa966600640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:133 6.1f 6:f8000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x56302410ec00 con 0x56302410e400 2024-04-25T01:18:25.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.852+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.6 v2:172.21.15.103:6816/2008388906 15 ==== osd_op_reply(133 [pgnls-filter start_epoch 50 out=49b] v42'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x56302410e400 2024-04-25T01:18:25.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.852+0000 7fa967000640 1 -- 172.21.15.90:0/1433430244 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:134 6.3f 6:fc000000::::head [pgnls-filter start_epoch 50 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x563024c92800 con 0x5630240cbc00 2024-04-25T01:18:25.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.853+0000 7fa96d400640 1 -- 172.21.15.90:0/1433430244 <== osd.4 v2:172.21.15.103:6821/1971061680 21 ==== osd_op_reply(134 [pgnls-filter start_epoch 50 out=49b] v42'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x563024ae6500 con 0x5630240cbc00 2024-04-25T01:18:25.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.854+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x563024c92c00 msgr2=0x563023e07700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.854+0000 7fa96f41a740 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x563024c92c00 0x563023e07700 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.854+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x5630240cb800 msgr2=0x5630240d2c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.854+0000 7fa96f41a740 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x5630240cb800 0x5630240d2c00 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.854+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 >> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] conn(0x5630240cb000 msgr2=0x5630240d2100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.854+0000 7fa96f41a740 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] conn(0x5630240cb000 0x5630240d2100 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.854+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x5630240fc400 msgr2=0x5630240d3700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.854+0000 7fa96f41a740 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x5630240fc400 0x5630240d3700 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.854+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x5630240cbc00 msgr2=0x5630240d2680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.854+0000 7fa96f41a740 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x5630240cbc00 0x5630240d2680 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.854+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x563024c92400 msgr2=0x563023e07180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.854+0000 7fa96f41a740 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x563024c92400 0x563023e07180 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.854+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x56302410e400 msgr2=0x563024110000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.854+0000 7fa96f41a740 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x56302410e400 0x563024110000 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.854+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x563024c93400 msgr2=0x5630240d0580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.854+0000 7fa96f41a740 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x563024c93400 0x5630240d0580 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.854+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 reap_dead start 2024-04-25T01:18:25.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.854+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 reap_dead start 2024-04-25T01:18:25.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.854+0000 7fa96ca00640 1 -- 172.21.15.90:0/1433430244 reap_dead start 2024-04-25T01:18:25.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.854+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x5630240ca000 msgr2=0x5630240d0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.854+0000 7fa96f41a740 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x5630240ca000 0x5630240d0000 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x563024afb170 tx=0x563024058ae0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.854+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563023d61c00 msgr2=0x563023e06100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.854+0000 7fa96f41a740 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563023d61c00 0x563023e06100 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x563024c3ce10 tx=0x563024afc000 comp rx=0 tx=0).stop 2024-04-25T01:18:25.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.855+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 shutdown_connections 2024-04-25T01:18:25.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.855+0000 7fa96f41a740 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x5630240ca000 0x5630240d0000 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.855+0000 7fa96f41a740 1 --2- 172.21.15.90:0/1433430244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563023d61c00 0x563023e06100 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.855+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 >> 172.21.15.90:0/1433430244 conn(0x563023d60c00 msgr2=0x563024b01800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:25.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.855+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 shutdown_connections 2024-04-25T01:18:25.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.855+0000 7fa96f41a740 1 -- 172.21.15.90:0/1433430244 wait complete. 2024-04-25T01:18:25.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.856+0000 7fa96f41a740 1 -- 172.21.15.90:0/1725878873 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563023d61800 msgr2=0x563023e05b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.856+0000 7fa96f41a740 1 --2- 172.21.15.90:0/1725878873 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563023d61800 0x563023e05b80 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x563024b83b30 tx=0x563024afc2a0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.856+0000 7fa96f41a740 1 -- 172.21.15.90:0/1725878873 shutdown_connections 2024-04-25T01:18:25.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.856+0000 7fa96f41a740 1 --2- 172.21.15.90:0/1725878873 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563023d61800 0x563023e05b80 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.856+0000 7fa96f41a740 1 --2- 172.21.15.90:0/1725878873 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x563023ed6000 0x563023e04b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.856+0000 7fa96f41a740 1 --2- 172.21.15.90:0/1725878873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563023ed6400 0x563023e05600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.856+0000 7fa96f41a740 1 -- 172.21.15.90:0/1725878873 >> 172.21.15.90:0/1725878873 conn(0x563023d61000 msgr2=0x563024b01000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:25.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.857+0000 7fa96f41a740 1 -- 172.21.15.90:0/1725878873 shutdown_connections 2024-04-25T01:18:25.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.857+0000 7fa96f41a740 1 -- 172.21.15.90:0/1725878873 wait complete. 2024-04-25T01:18:25.855 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_inodes', '--alternate-pool', 'recovery_fs_metadata', '--filesystem', 'cephfs', '--force-corrupt', '--force-init', 'cephfs_data'] in time 0:00:00.293914, result: 2024-04-25T01:18:25.855 DEBUG:teuthology.orchestra.run.smithi090:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_links --filesystem recovery_fs 2024-04-25T01:18:25.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.974+0000 7f7f6fc1a740 1 -- start start 2024-04-25T01:18:25.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.974+0000 7f7f6fc1a740 1 Processor -- start 2024-04-25T01:18:25.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.975+0000 7f7f6fc1a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ec4920c00 0x560ec49c4580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.975+0000 7f7f6fc1a740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ec4921800 0x560ec49c4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.975+0000 7f7f6fc1a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec4921c00 0x560ec49c5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.975+0000 7f7f6fc1a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560ec5688780 con 0x560ec4921c00 2024-04-25T01:18:25.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.975+0000 7f7f6fc1a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560ec5688600 con 0x560ec4920c00 2024-04-25T01:18:25.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.975+0000 7f7f6fc1a740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x560ec5688480 con 0x560ec4921800 2024-04-25T01:18:25.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.976+0000 7f7f6dc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec4921c00 0x560ec49c5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.976+0000 7f7f6dc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec4921c00 0x560ec49c5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47708/0 (socket says 172.21.15.90:47708) 2024-04-25T01:18:25.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.976+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1422867676 learned_addr learned my addr 172.21.15.90:0/1422867676 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:25.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.976+0000 7f7f6c800640 1 --2- 172.21.15.90:0/1422867676 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ec4921800 0x560ec49c4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.976+0000 7f7f6d200640 1 --2- 172.21.15.90:0/1422867676 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ec4920c00 0x560ec49c4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.976+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1422867676 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ec4921800 msgr2=0x560ec49c4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:25.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.976+0000 7f7f6dc00640 1 --2- 172.21.15.90:0/1422867676 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ec4921800 0x560ec49c4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.976+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1422867676 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ec4920c00 msgr2=0x560ec49c4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:25.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.976+0000 7f7f6dc00640 1 --2- 172.21.15.90:0/1422867676 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ec4920c00 0x560ec49c4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.976+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1422867676 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ec494d680 con 0x560ec4921c00 2024-04-25T01:18:25.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.976+0000 7f7f6dc00640 1 --2- 172.21.15.90:0/1422867676 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec4921c00 0x560ec49c5080 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x560ec573d260 tx=0x560ec56bcde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eeacd7217015bf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.977+0000 7f7f6a000640 1 -- 172.21.15.90:0/1422867676 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x560ec49469c0 con 0x560ec4921c00 2024-04-25T01:18:25.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.977+0000 7f7f6a000640 1 -- 172.21.15.90:0/1422867676 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x560ec496ae00 con 0x560ec4921c00 2024-04-25T01:18:25.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.977+0000 7f7f6a000640 1 -- 172.21.15.90:0/1422867676 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x560ec4946680 con 0x560ec4921c00 2024-04-25T01:18:25.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.977+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1422867676 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ec494d860 con 0x560ec4921c00 2024-04-25T01:18:25.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.977+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1422867676 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) -- 0x560ec494d2c0 con 0x560ec4921c00 2024-04-25T01:18:25.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.977+0000 7f7f6fc1a740 4 waiting for MDS map... 2024-04-25T01:18:25.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.977+0000 7f7f6a000640 1 -- 172.21.15.90:0/1422867676 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x560ec56fc000 con 0x560ec4921c00 2024-04-25T01:18:25.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.977+0000 7f7f6a000640 1 -- 172.21.15.90:0/1422867676 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 23) ==== 4135+0+0 (secure 0 0 0) 0x560ec4979180 con 0x560ec4921c00 2024-04-25T01:18:25.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.977+0000 7f7f6fc1a740 4 Got MDS map 23 2024-04-25T01:18:25.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.977+0000 7f7f6fc1a740 4 datascan.main: Using metadata pool output 2024-04-25T01:18:25.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.977+0000 7f7f6fc1a740 4 datascan.main: connecting to RADOS... 2024-04-25T01:18:25.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.978+0000 7f7f6fc1a740 1 Processor -- start 2024-04-25T01:18:25.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.978+0000 7f7f6fc1a740 1 -- start start 2024-04-25T01:18:25.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.979+0000 7f7f6fc1a740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ec57ff000 0x560ec49c5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.979+0000 7f7f6fc1a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec57ff400 0x560ec49c5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.979+0000 7f7f6fc1a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ec57ff800 0x560ec49c6100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.979+0000 7f7f6fc1a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560ec5688300 con 0x560ec57ff400 2024-04-25T01:18:25.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.979+0000 7f7f6fc1a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560ec5688000 con 0x560ec57ff800 2024-04-25T01:18:25.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.979+0000 7f7f6fc1a740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x560ec4980900 con 0x560ec57ff000 2024-04-25T01:18:25.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.979+0000 7f7f6d200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec57ff400 0x560ec49c5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.979+0000 7f7f6d200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec57ff400 0x560ec49c5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47720/0 (socket says 172.21.15.90:47720) 2024-04-25T01:18:25.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.979+0000 7f7f6d200640 1 -- 172.21.15.90:0/4037395961 learned_addr learned my addr 172.21.15.90:0/4037395961 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:25.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.980+0000 7f7f6d200640 1 -- 172.21.15.90:0/4037395961 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ec57ff000 msgr2=0x560ec49c5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:25.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.980+0000 7f7f6d200640 1 --2- 172.21.15.90:0/4037395961 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ec57ff000 0x560ec49c5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.980+0000 7f7f6d200640 1 -- 172.21.15.90:0/4037395961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ec57ff800 msgr2=0x560ec49c6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:25.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.980+0000 7f7f6d200640 1 --2- 172.21.15.90:0/4037395961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ec57ff800 0x560ec49c6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.980+0000 7f7f6d200640 1 -- 172.21.15.90:0/4037395961 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ec494d4a0 con 0x560ec57ff400 2024-04-25T01:18:25.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.980+0000 7f7f6d200640 1 --2- 172.21.15.90:0/4037395961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec57ff400 0x560ec49c5b80 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x560ec57fc9f0 tx=0x560ec5803800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5018acb3c78360ca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.980+0000 7f7f67800640 1 -- 172.21.15.90:0/4037395961 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x560ec49469c0 con 0x560ec57ff400 2024-04-25T01:18:25.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.980+0000 7f7f67800640 1 -- 172.21.15.90:0/4037395961 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x560ec496a540 con 0x560ec57ff400 2024-04-25T01:18:25.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.980+0000 7f7f67800640 1 -- 172.21.15.90:0/4037395961 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x560ec4946680 con 0x560ec57ff400 2024-04-25T01:18:25.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.981+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/4037395961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec57ff400 msgr2=0x560ec49c5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.981+0000 7f7f6fc1a740 1 --2- 172.21.15.90:0/4037395961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec57ff400 0x560ec49c5b80 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x560ec57fc9f0 tx=0x560ec5803800 comp rx=0 tx=0).stop 2024-04-25T01:18:25.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.981+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/4037395961 shutdown_connections 2024-04-25T01:18:25.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.981+0000 7f7f6fc1a740 1 --2- 172.21.15.90:0/4037395961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ec57ff800 0x560ec49c6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.981+0000 7f7f6fc1a740 1 --2- 172.21.15.90:0/4037395961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec57ff400 0x560ec49c5b80 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.981+0000 7f7f6fc1a740 1 --2- 172.21.15.90:0/4037395961 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ec57ff000 0x560ec49c5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.981+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/4037395961 >> 172.21.15.90:0/4037395961 conn(0x560ec57fec00 msgr2=0x560ec4a0d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:25.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.981+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/4037395961 shutdown_connections 2024-04-25T01:18:25.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.981+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/4037395961 wait complete. 2024-04-25T01:18:25.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.981+0000 7f7f6fc1a740 1 Processor -- start 2024-04-25T01:18:25.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.982+0000 7f7f6fc1a740 1 -- start start 2024-04-25T01:18:25.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.982+0000 7f7f6fc1a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ec57ff000 0x560ec49c7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.982+0000 7f7f6fc1a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec57ff400 0x560ec49c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.982+0000 7f7f6fc1a740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ec57ff800 0x560ec49c6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.982+0000 7f7f6fc1a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560ec5688300 con 0x560ec57ff400 2024-04-25T01:18:25.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.982+0000 7f7f6fc1a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560ec5688000 con 0x560ec57ff000 2024-04-25T01:18:25.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.982+0000 7f7f6fc1a740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x560ec5689500 con 0x560ec57ff800 2024-04-25T01:18:25.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.983+0000 7f7f6d200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec57ff400 0x560ec49c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.983+0000 7f7f6d200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec57ff400 0x560ec49c6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47730/0 (socket says 172.21.15.90:47730) 2024-04-25T01:18:25.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.983+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 learned_addr learned my addr 172.21.15.90:0/1398212762 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:25.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.983+0000 7f7f6c800640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ec57ff000 0x560ec49c7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.983+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ec57ff800 msgr2=0x560ec49c6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.983+0000 7f7f6c800640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ec57ff800 0x560ec49c6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.983+0000 7f7f6d200640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ec57ff800 0x560ec49c6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.983+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ec57ff000 msgr2=0x560ec49c7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:25.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.983+0000 7f7f6d200640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ec57ff000 0x560ec49c7180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:25.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.983+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ec494d0e0 con 0x560ec57ff400 2024-04-25T01:18:25.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.983+0000 7f7f6c800640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ec57ff800 0x560ec49c6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:25.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.983+0000 7f7f6d200640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec57ff400 0x560ec49c6c00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x560ec57fdcb0 tx=0x560ec56bcfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.984+0000 7f7f65a00640 1 -- 172.21.15.90:0/1398212762 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x560ec49464e0 con 0x560ec57ff400 2024-04-25T01:18:25.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.984+0000 7f7f65a00640 1 -- 172.21.15.90:0/1398212762 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x560ec496aa80 con 0x560ec57ff400 2024-04-25T01:18:25.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.984+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x560ec494da40 con 0x560ec57ff400 2024-04-25T01:18:25.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.984+0000 7f7f65a00640 1 -- 172.21.15.90:0/1398212762 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x560ec49469c0 con 0x560ec57ff400 2024-04-25T01:18:25.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.984+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ec494c3c0 con 0x560ec57ff400 2024-04-25T01:18:25.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.984+0000 7f7f6fc1a740 4 datascan.init: resolving metadata pool 7 2024-04-25T01:18:25.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.986+0000 7f7f65a00640 1 -- 172.21.15.90:0/1398212762 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x560ec584c900 con 0x560ec57ff400 2024-04-25T01:18:25.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.986+0000 7f7f65a00640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x560ec5862800 0x560ec49c7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.986+0000 7f7f65a00640 1 -- 172.21.15.90:0/1398212762 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x560ec5864a00 con 0x560ec57ff400 2024-04-25T01:18:25.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.986+0000 7f7f6fc1a740 4 datascan.init: found metadata pool 'recovery_fs_metadata' 2024-04-25T01:18:25.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.986+0000 7f7f6fc1a740 4 datascan.main: resolving metadata pool 7 2024-04-25T01:18:25.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.986+0000 7f7f6fc1a740 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x560ec5862c00 0x560ec49c6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.986+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:1 7.0 7:00000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec5863000 con 0x560ec5862c00 2024-04-25T01:18:25.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.986+0000 7f7f6c800640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x560ec5862800 0x560ec49c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.987+0000 7f7f6c800640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x560ec5862c00 0x560ec49c6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.987+0000 7f7f6c800640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x560ec5862800 0x560ec49c7700 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x560ec56baed0 tx=0x560ec4c2cf00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.988+0000 7f7f6c800640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x560ec5862c00 0x560ec49c6100 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.988+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 <== osd.1 v2:172.21.15.90:6800/2685473653 1 ==== osd_op_reply(1 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c000 con 0x560ec5862c00 2024-04-25T01:18:25.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.988+0000 7f7f6c800640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x560ec5863000 0x560ec4cb6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.988+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:2 7.10 7:08000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cb0800 con 0x560ec5863000 2024-04-25T01:18:25.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.989+0000 7f7f6dc00640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x560ec5863000 0x560ec4cb6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.990+0000 7f7f6dc00640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x560ec5863000 0x560ec4cb6000 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.991+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.7 v2:172.21.15.103:6800/1145679154 1 ==== osd_op_reply(2 [pgnls start_epoch 50 out=81b] v50'2 uv2 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x560ec4a9c500 con 0x560ec5863000 2024-04-25T01:18:25.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.991+0000 7f7f6dc00640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x560ec4cb0800 0x560ec4cb6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.991+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- osd_op(unknown.0.0:3 7.8 7:10000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec56b7c00 con 0x560ec4cb0800 2024-04-25T01:18:25.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.992+0000 7f7f6d200640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x560ec4cb0800 0x560ec4cb6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.993+0000 7f7f6d200640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x560ec4cb0800 0x560ec4cb6580 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.994+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 <== osd.5 v2:172.21.15.103:6801/1348846168 1 ==== osd_op_reply(3 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c780 con 0x560ec4cb0800 2024-04-25T01:18:25.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.994+0000 7f7f6d200640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x560ec56b7c00 0x560ec4cb6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.994+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:4 7.18 7:18000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cb0c00 con 0x560ec56b7c00 2024-04-25T01:18:25.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.995+0000 7f7f6c800640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x560ec56b7c00 0x560ec4cb6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.996+0000 7f7f6c800640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x560ec56b7c00 0x560ec4cb6b00 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.996+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 <== osd.4 v2:172.21.15.103:6821/1971061680 1 ==== osd_op_reply(4 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c000 con 0x560ec56b7c00 2024-04-25T01:18:25.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.996+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:5 7.4 7:20000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cb0c00 con 0x560ec56b7c00 2024-04-25T01:18:25.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.997+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 <== osd.4 v2:172.21.15.103:6821/1971061680 2 ==== osd_op_reply(5 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c000 con 0x560ec56b7c00 2024-04-25T01:18:25.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.997+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- osd_op(unknown.0.0:6 7.14 7:28000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cb0c00 con 0x560ec4cb0800 2024-04-25T01:18:25.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.998+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 <== osd.5 v2:172.21.15.103:6801/1348846168 2 ==== osd_op_reply(6 [pgnls start_epoch 50 out=77b] v50'1 uv1 ondisk = 1) ==== 144+0+77 (crc 0 0 0) 0x560ec4a9c780 con 0x560ec4cb0800 2024-04-25T01:18:25.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.998+0000 7f7f6d200640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] conn(0x560ec4cb0c00 0x560ec4cb7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.998+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:7 7.c 7:30000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cb1000 con 0x560ec4cb0c00 2024-04-25T01:18:25.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.999+0000 7f7f6dc00640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] conn(0x560ec4cb0c00 0x560ec4cb7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:25.999+0000 7f7f6dc00640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] conn(0x560ec4cb0c00 0x560ec4cb7080 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.000+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.2 v2:172.21.15.90:6808/906792737 1 ==== osd_op_reply(7 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c500 con 0x560ec4cb0c00 2024-04-25T01:18:25.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.000+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:8 7.1c 7:38000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cb1000 con 0x560ec4cb0c00 2024-04-25T01:18:25.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.000+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.2 v2:172.21.15.90:6808/906792737 2 ==== osd_op_reply(8 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c500 con 0x560ec4cb0c00 2024-04-25T01:18:25.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.000+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:9 7.2 7:40000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cb1000 con 0x560ec56b7c00 2024-04-25T01:18:25.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.001+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 <== osd.4 v2:172.21.15.103:6821/1971061680 3 ==== osd_op_reply(9 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c000 con 0x560ec56b7c00 2024-04-25T01:18:25.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.001+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:10 7.12 7:48000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cb1000 con 0x560ec5862c00 2024-04-25T01:18:25.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.002+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 <== osd.1 v2:172.21.15.90:6800/2685473653 2 ==== osd_op_reply(10 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c000 con 0x560ec5862c00 2024-04-25T01:18:25.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.002+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:11 7.a 7:50000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cb1000 con 0x560ec56b7c00 2024-04-25T01:18:25.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.002+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 <== osd.4 v2:172.21.15.103:6821/1971061680 4 ==== osd_op_reply(11 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c000 con 0x560ec56b7c00 2024-04-25T01:18:25.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.002+0000 7f7f6c800640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x560ec4cb1000 0x560ec4cb7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.002+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:12 7.1a 7:58000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cb1800 con 0x560ec4cb1000 2024-04-25T01:18:25.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.003+0000 7f7f6d200640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x560ec4cb1000 0x560ec4cb7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:25.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.003+0000 7f7f6d200640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x560ec4cb1000 0x560ec4cb7b80 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:25.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.004+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 <== osd.3 v2:172.21.15.90:6812/8683104 1 ==== osd_op_reply(12 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c780 con 0x560ec4cb1000 2024-04-25T01:18:25.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.004+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- osd_op(unknown.0.0:13 7.6 7:60000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cb1800 con 0x560ec4cb0800 2024-04-25T01:18:25.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.005+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 <== osd.5 v2:172.21.15.103:6801/1348846168 3 ==== osd_op_reply(13 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c780 con 0x560ec4cb0800 2024-04-25T01:18:25.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.005+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:14 7.16 7:68000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cb1800 con 0x560ec5862c00 2024-04-25T01:18:25.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.006+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 <== osd.1 v2:172.21.15.90:6800/2685473653 3 ==== osd_op_reply(14 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c000 con 0x560ec5862c00 2024-04-25T01:18:25.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.006+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:15 7.e 7:70000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cb1800 con 0x560ec5863000 2024-04-25T01:18:25.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.007+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.7 v2:172.21.15.103:6800/1145679154 2 ==== osd_op_reply(15 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c500 con 0x560ec5863000 2024-04-25T01:18:25.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.007+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:16 7.1e 7:78000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cb1800 con 0x560ec5863000 2024-04-25T01:18:25.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.007+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.7 v2:172.21.15.103:6800/1145679154 3 ==== osd_op_reply(16 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c500 con 0x560ec5863000 2024-04-25T01:18:25.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.007+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:17 7.1 7:80000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cb1800 con 0x560ec4cb0c00 2024-04-25T01:18:25.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.008+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.2 v2:172.21.15.90:6808/906792737 3 ==== osd_op_reply(17 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c500 con 0x560ec4cb0c00 2024-04-25T01:18:25.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.008+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:18 7.11 7:88000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cb1800 con 0x560ec4cb0c00 2024-04-25T01:18:25.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.008+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.2 v2:172.21.15.90:6808/906792737 4 ==== osd_op_reply(18 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c500 con 0x560ec4cb0c00 2024-04-25T01:18:25.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.009+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- osd_op(unknown.0.0:19 7.9 7:90000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cb1800 con 0x560ec4cb0800 2024-04-25T01:18:25.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.010+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 <== osd.5 v2:172.21.15.103:6801/1348846168 4 ==== osd_op_reply(19 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c780 con 0x560ec4cb0800 2024-04-25T01:18:25.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.010+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:20 7.19 7:98000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cb1800 con 0x560ec4cb1000 2024-04-25T01:18:25.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.010+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 <== osd.3 v2:172.21.15.90:6812/8683104 2 ==== osd_op_reply(20 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c780 con 0x560ec4cb1000 2024-04-25T01:18:25.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.010+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:21 7.5 7:a0000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cb1800 con 0x560ec4cb1000 2024-04-25T01:18:25.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.010+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 <== osd.3 v2:172.21.15.90:6812/8683104 3 ==== osd_op_reply(21 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c780 con 0x560ec4cb1000 2024-04-25T01:18:25.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.010+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:22 7.15 7:a8000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cb1800 con 0x560ec4cb1000 2024-04-25T01:18:25.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.011+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 <== osd.3 v2:172.21.15.90:6812/8683104 4 ==== osd_op_reply(22 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c780 con 0x560ec4cb1000 2024-04-25T01:18:25.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.011+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:23 7.d 7:b0000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cb1800 con 0x560ec5862c00 2024-04-25T01:18:25.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.012+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 <== osd.1 v2:172.21.15.90:6800/2685473653 4 ==== osd_op_reply(23 [pgnls start_epoch 50 out=74b] v50'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x560ec4a9c000 con 0x560ec5862c00 2024-04-25T01:18:25.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.012+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- osd_op(unknown.0.0:24 7.1d 7:b8000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cb1800 con 0x560ec4cb0800 2024-04-25T01:18:25.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.013+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 <== osd.5 v2:172.21.15.103:6801/1348846168 5 ==== osd_op_reply(24 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c780 con 0x560ec4cb0800 2024-04-25T01:18:25.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.013+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- osd_op(unknown.0.0:25 7.3 7:c0000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cb1800 con 0x560ec4cb0800 2024-04-25T01:18:25.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.013+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 <== osd.5 v2:172.21.15.103:6801/1348846168 6 ==== osd_op_reply(25 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c780 con 0x560ec4cb0800 2024-04-25T01:18:25.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.014+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:26 7.13 7:c8000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cb1800 con 0x560ec5863000 2024-04-25T01:18:25.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.015+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.7 v2:172.21.15.103:6800/1145679154 4 ==== osd_op_reply(26 [pgnls start_epoch 50 out=98b] v50'2 uv2 ondisk = 1) ==== 144+0+98 (crc 0 0 0) 0x560ec4a9c500 con 0x560ec5863000 2024-04-25T01:18:25.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.015+0000 7f7f6dc00640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x560ec4cb1800 0x560ec4cb8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:25.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.015+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:27 7.b 7:d0000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd0000 con 0x560ec4cb1800 2024-04-25T01:18:25.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.015+0000 7f7f6c800640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x560ec4cb1800 0x560ec4cb8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.017+0000 7f7f6c800640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x560ec4cb1800 0x560ec4cb8680 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:26.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.017+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 <== osd.6 v2:172.21.15.103:6816/2008388906 1 ==== osd_op_reply(27 [pgnls start_epoch 50 out=76b] v50'1 uv1 ondisk = 1) ==== 144+0+76 (crc 0 0 0) 0x560ec4a9c000 con 0x560ec4cb1800 2024-04-25T01:18:26.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.017+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:28 7.1b 7:d8000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd0000 con 0x560ec56b7c00 2024-04-25T01:18:26.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.018+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 <== osd.4 v2:172.21.15.103:6821/1971061680 5 ==== osd_op_reply(28 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c000 con 0x560ec56b7c00 2024-04-25T01:18:26.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.018+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:29 7.7 7:e0000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd0000 con 0x560ec4cb1000 2024-04-25T01:18:26.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.019+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 <== osd.3 v2:172.21.15.90:6812/8683104 5 ==== osd_op_reply(29 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c780 con 0x560ec4cb1000 2024-04-25T01:18:26.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.019+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:30 7.17 7:e8000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd0000 con 0x560ec5862c00 2024-04-25T01:18:26.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.020+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 <== osd.1 v2:172.21.15.90:6800/2685473653 5 ==== osd_op_reply(30 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c000 con 0x560ec5862c00 2024-04-25T01:18:26.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.020+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:31 7.f 7:f0000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd0000 con 0x560ec5863000 2024-04-25T01:18:26.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.021+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.7 v2:172.21.15.103:6800/1145679154 5 ==== osd_op_reply(31 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c500 con 0x560ec5863000 2024-04-25T01:18:26.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.021+0000 7f7f6dc00640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x560ec4cd0000 0x560ec4cb8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.021+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:32 7.1f 7:f8000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd0400 con 0x560ec4cd0000 2024-04-25T01:18:26.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.022+0000 7f7f6dc00640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x560ec4cd0000 0x560ec4cb8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.022+0000 7f7f6dc00640 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x560ec4cd0000 0x560ec4cb8c00 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:26.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.023+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.0 v2:172.21.15.90:6816/2625481430 1 ==== osd_op_reply(32 [pgnls start_epoch 50 out=101b] v50'3 uv3 ondisk = 1) ==== 144+0+101 (crc 0 0 0) 0x560ec4a9c500 con 0x560ec4cd0000 2024-04-25T01:18:26.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.023+0000 7f7f6fc1a740 10 datascan.scan_links: step 1: handling object 10000000000.00000000 2024-04-25T01:18:26.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.023+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:33 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50) -- 0x560ec5863400 con 0x560ec5863000 2024-04-25T01:18:26.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.024+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.7 v2:172.21.15.103:6800/1145679154 6 ==== osd_op_reply(33 10000000000.00000000 [omap-get-vals out=509b] v0'0 uv2 ondisk = 0) ==== 164+0+509 (crc 0 0 0) 0x560ec4a9c500 con 0x560ec5863000 2024-04-25T01:18:26.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.024+0000 7f7f6fc1a740 20 datascan.scan_links: injected ino detected 2024-04-25T01:18:26.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.024+0000 7f7f6fc1a740 10 datascan.scan_links: step 1: handling object 1.00000000.inode 2024-04-25T01:18:26.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.024+0000 7f7f6fc1a740 10 datascan.scan_links: Not a dirfrag: '1.00000000.inode' 2024-04-25T01:18:26.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.024+0000 7f7f6fc1a740 10 datascan.scan_links: step 1: handling object mds_snaptable 2024-04-25T01:18:26.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.024+0000 7f7f6fc1a740 10 datascan.scan_links: Not a dirfrag: 'mds_snaptable' 2024-04-25T01:18:26.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.024+0000 7f7f6fc1a740 10 datascan.scan_links: step 1: handling object mds0_inotable 2024-04-25T01:18:26.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.024+0000 7f7f6fc1a740 10 datascan.scan_links: Not a dirfrag: 'mds0_inotable' 2024-04-25T01:18:26.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.024+0000 7f7f6fc1a740 10 datascan.scan_links: step 1: handling object 100.00000000 2024-04-25T01:18:26.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.024+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:34 7.13 7:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50) -- 0x560ec5863800 con 0x560ec5863000 2024-04-25T01:18:26.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.025+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.7 v2:172.21.15.103:6800/1145679154 7 ==== osd_op_reply(34 100.00000000 [omap-get-vals out=5b] v0'0 uv1 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x560ec4a9c500 con 0x560ec5863000 2024-04-25T01:18:26.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.026+0000 7f7f6fc1a740 10 datascan.scan_links: step 1: handling object mds0_sessionmap 2024-04-25T01:18:26.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.026+0000 7f7f6fc1a740 10 datascan.scan_links: Not a dirfrag: 'mds0_sessionmap' 2024-04-25T01:18:26.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.026+0000 7f7f6fc1a740 10 datascan.scan_links: step 1: handling object 100.00000000.inode 2024-04-25T01:18:26.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.026+0000 7f7f6fc1a740 10 datascan.scan_links: Not a dirfrag: '100.00000000.inode' 2024-04-25T01:18:26.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.026+0000 7f7f6fc1a740 10 datascan.scan_links: step 1: handling object 1.00000000 2024-04-25T01:18:26.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.026+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:35 7.1f 7:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50) -- 0x560ec5863c00 con 0x560ec4cd0000 2024-04-25T01:18:26.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.026+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.0 v2:172.21.15.90:6816/2625481430 2 ==== osd_op_reply(35 1.00000000 [omap-get-vals out=508b] v0'0 uv3 ondisk = 0) ==== 154+0+508 (crc 0 0 0) 0x560ec4a9c500 con 0x560ec4cd0000 2024-04-25T01:18:26.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.026+0000 7f7f6fc1a740 20 datascan.scan_links: injected ino detected 2024-04-25T01:18:26.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.026+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:36 7.0 7:00000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec5862c00 2024-04-25T01:18:26.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.027+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 <== osd.1 v2:172.21.15.90:6800/2685473653 6 ==== osd_op_reply(36 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c000 con 0x560ec5862c00 2024-04-25T01:18:26.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.027+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:37 7.10 7:08000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec5863000 2024-04-25T01:18:26.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.028+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.7 v2:172.21.15.103:6800/1145679154 8 ==== osd_op_reply(37 [pgnls start_epoch 50 out=81b] v50'2 uv2 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x560ec4a9c500 con 0x560ec5863000 2024-04-25T01:18:26.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.028+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- osd_op(unknown.0.0:38 7.8 7:10000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec4cb0800 2024-04-25T01:18:26.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.029+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 <== osd.5 v2:172.21.15.103:6801/1348846168 7 ==== osd_op_reply(38 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c780 con 0x560ec4cb0800 2024-04-25T01:18:26.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.029+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:39 7.18 7:18000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec56b7c00 2024-04-25T01:18:26.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.030+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 <== osd.4 v2:172.21.15.103:6821/1971061680 6 ==== osd_op_reply(39 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c000 con 0x560ec56b7c00 2024-04-25T01:18:26.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.030+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:40 7.4 7:20000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec56b7c00 2024-04-25T01:18:26.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.031+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 <== osd.4 v2:172.21.15.103:6821/1971061680 7 ==== osd_op_reply(40 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c000 con 0x560ec56b7c00 2024-04-25T01:18:26.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.031+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- osd_op(unknown.0.0:41 7.14 7:28000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec4cb0800 2024-04-25T01:18:26.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.032+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 <== osd.5 v2:172.21.15.103:6801/1348846168 8 ==== osd_op_reply(41 [pgnls start_epoch 50 out=77b] v50'1 uv1 ondisk = 1) ==== 144+0+77 (crc 0 0 0) 0x560ec4a9c780 con 0x560ec4cb0800 2024-04-25T01:18:26.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.032+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:42 7.c 7:30000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec4cb0c00 2024-04-25T01:18:26.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.033+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.2 v2:172.21.15.90:6808/906792737 5 ==== osd_op_reply(42 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c500 con 0x560ec4cb0c00 2024-04-25T01:18:26.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.033+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:43 7.1c 7:38000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec4cb0c00 2024-04-25T01:18:26.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.033+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.2 v2:172.21.15.90:6808/906792737 6 ==== osd_op_reply(43 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c500 con 0x560ec4cb0c00 2024-04-25T01:18:26.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.033+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:44 7.2 7:40000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec56b7c00 2024-04-25T01:18:26.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.034+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 <== osd.4 v2:172.21.15.103:6821/1971061680 8 ==== osd_op_reply(44 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c000 con 0x560ec56b7c00 2024-04-25T01:18:26.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.034+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:45 7.12 7:48000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec5862c00 2024-04-25T01:18:26.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.035+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 <== osd.1 v2:172.21.15.90:6800/2685473653 7 ==== osd_op_reply(45 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c000 con 0x560ec5862c00 2024-04-25T01:18:26.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.035+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:46 7.a 7:50000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec56b7c00 2024-04-25T01:18:26.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.036+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 <== osd.4 v2:172.21.15.103:6821/1971061680 9 ==== osd_op_reply(46 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c000 con 0x560ec56b7c00 2024-04-25T01:18:26.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.036+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:47 7.1a 7:58000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec4cb1000 2024-04-25T01:18:26.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.037+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 <== osd.3 v2:172.21.15.90:6812/8683104 6 ==== osd_op_reply(47 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c780 con 0x560ec4cb1000 2024-04-25T01:18:26.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.037+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- osd_op(unknown.0.0:48 7.6 7:60000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec4cb0800 2024-04-25T01:18:26.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.038+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 <== osd.5 v2:172.21.15.103:6801/1348846168 9 ==== osd_op_reply(48 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c780 con 0x560ec4cb0800 2024-04-25T01:18:26.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.038+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:49 7.16 7:68000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec5862c00 2024-04-25T01:18:26.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.039+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 <== osd.1 v2:172.21.15.90:6800/2685473653 8 ==== osd_op_reply(49 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c000 con 0x560ec5862c00 2024-04-25T01:18:26.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.039+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:50 7.e 7:70000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec5863000 2024-04-25T01:18:26.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.040+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.7 v2:172.21.15.103:6800/1145679154 9 ==== osd_op_reply(50 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c500 con 0x560ec5863000 2024-04-25T01:18:26.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.040+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:51 7.1e 7:78000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec5863000 2024-04-25T01:18:26.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.041+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.7 v2:172.21.15.103:6800/1145679154 10 ==== osd_op_reply(51 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c500 con 0x560ec5863000 2024-04-25T01:18:26.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.041+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:52 7.1 7:80000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec4cb0c00 2024-04-25T01:18:26.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.041+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.2 v2:172.21.15.90:6808/906792737 7 ==== osd_op_reply(52 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c500 con 0x560ec4cb0c00 2024-04-25T01:18:26.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.041+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- osd_op(unknown.0.0:53 7.11 7:88000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec4cb0c00 2024-04-25T01:18:26.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.042+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.2 v2:172.21.15.90:6808/906792737 8 ==== osd_op_reply(53 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c500 con 0x560ec4cb0c00 2024-04-25T01:18:26.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.042+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- osd_op(unknown.0.0:54 7.9 7:90000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec4cb0800 2024-04-25T01:18:26.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.043+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 <== osd.5 v2:172.21.15.103:6801/1348846168 10 ==== osd_op_reply(54 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c780 con 0x560ec4cb0800 2024-04-25T01:18:26.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.043+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:55 7.19 7:98000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec4cb1000 2024-04-25T01:18:26.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.044+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 <== osd.3 v2:172.21.15.90:6812/8683104 7 ==== osd_op_reply(55 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c780 con 0x560ec4cb1000 2024-04-25T01:18:26.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.044+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:56 7.5 7:a0000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec4cb1000 2024-04-25T01:18:26.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.045+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 <== osd.3 v2:172.21.15.90:6812/8683104 8 ==== osd_op_reply(56 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c780 con 0x560ec4cb1000 2024-04-25T01:18:26.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.045+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:57 7.15 7:a8000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec4cb1000 2024-04-25T01:18:26.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.045+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 <== osd.3 v2:172.21.15.90:6812/8683104 9 ==== osd_op_reply(57 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c780 con 0x560ec4cb1000 2024-04-25T01:18:26.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.045+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:58 7.d 7:b0000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec5862c00 2024-04-25T01:18:26.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.046+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 <== osd.1 v2:172.21.15.90:6800/2685473653 9 ==== osd_op_reply(58 [pgnls start_epoch 50 out=74b] v50'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x560ec4a9c000 con 0x560ec5862c00 2024-04-25T01:18:26.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.046+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- osd_op(unknown.0.0:59 7.1d 7:b8000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec4cb0800 2024-04-25T01:18:26.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.047+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 <== osd.5 v2:172.21.15.103:6801/1348846168 11 ==== osd_op_reply(59 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c780 con 0x560ec4cb0800 2024-04-25T01:18:26.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.047+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- osd_op(unknown.0.0:60 7.3 7:c0000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec4cb0800 2024-04-25T01:18:26.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.049+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 <== osd.5 v2:172.21.15.103:6801/1348846168 12 ==== osd_op_reply(60 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c780 con 0x560ec4cb0800 2024-04-25T01:18:26.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.049+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:61 7.13 7:c8000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec5863000 2024-04-25T01:18:26.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.050+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.7 v2:172.21.15.103:6800/1145679154 11 ==== osd_op_reply(61 [pgnls start_epoch 50 out=98b] v50'2 uv2 ondisk = 1) ==== 144+0+98 (crc 0 0 0) 0x560ec4a9c500 con 0x560ec5863000 2024-04-25T01:18:26.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.050+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:62 7.b 7:d0000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec4cb1800 2024-04-25T01:18:26.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.050+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 <== osd.6 v2:172.21.15.103:6816/2008388906 2 ==== osd_op_reply(62 [pgnls start_epoch 50 out=76b] v50'1 uv1 ondisk = 1) ==== 144+0+76 (crc 0 0 0) 0x560ec4a9c000 con 0x560ec4cb1800 2024-04-25T01:18:26.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.050+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:63 7.1b 7:d8000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec56b7c00 2024-04-25T01:18:26.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.051+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 <== osd.4 v2:172.21.15.103:6821/1971061680 10 ==== osd_op_reply(63 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c000 con 0x560ec56b7c00 2024-04-25T01:18:26.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.051+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:64 7.7 7:e0000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec4cb1000 2024-04-25T01:18:26.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.052+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 <== osd.3 v2:172.21.15.90:6812/8683104 10 ==== osd_op_reply(64 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c780 con 0x560ec4cb1000 2024-04-25T01:18:26.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.052+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:65 7.17 7:e8000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec5862c00 2024-04-25T01:18:26.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.053+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 <== osd.1 v2:172.21.15.90:6800/2685473653 10 ==== osd_op_reply(65 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c000 con 0x560ec5862c00 2024-04-25T01:18:26.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.053+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:66 7.f 7:f0000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec5863000 2024-04-25T01:18:26.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.054+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.7 v2:172.21.15.103:6800/1145679154 12 ==== osd_op_reply(66 [pgnls start_epoch 50 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x560ec4a9c500 con 0x560ec5863000 2024-04-25T01:18:26.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.054+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:67 7.1f 7:f8000000::::head [pgnls start_epoch 50 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec4cd0000 2024-04-25T01:18:26.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.055+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.0 v2:172.21.15.90:6816/2625481430 3 ==== osd_op_reply(67 [pgnls start_epoch 50 out=101b] v50'3 uv3 ondisk = 1) ==== 144+0+101 (crc 0 0 0) 0x560ec4a9c500 con 0x560ec4cd0000 2024-04-25T01:18:26.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.055+0000 7f7f6fc1a740 10 datascan.scan_links: step 2: handling object 10000000000.00000000 2024-04-25T01:18:26.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.055+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:68 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd0c00 con 0x560ec5863000 2024-04-25T01:18:26.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.056+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.7 v2:172.21.15.103:6800/1145679154 13 ==== osd_op_reply(68 10000000000.00000000 [omap-get-vals out=509b] v0'0 uv2 ondisk = 0) ==== 164+0+509 (crc 0 0 0) 0x560ec4a9c500 con 0x560ec5863000 2024-04-25T01:18:26.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.056+0000 7f7f6fc1a740 20 datascan.scan_links: injected ino detected 2024-04-25T01:18:26.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.056+0000 7f7f6fc1a740 20 datascan.scan_links: adding 0x10000000001 for future processing to fix dnfirst 2024-04-25T01:18:26.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.056+0000 7f7f6fc1a740 10 datascan.scan_links: step 2: handling object 1.00000000.inode 2024-04-25T01:18:26.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.056+0000 7f7f6fc1a740 10 datascan.scan_links: Not a dirfrag: '1.00000000.inode' 2024-04-25T01:18:26.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.056+0000 7f7f6fc1a740 10 datascan.scan_links: step 2: handling object mds_snaptable 2024-04-25T01:18:26.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.056+0000 7f7f6fc1a740 10 datascan.scan_links: Not a dirfrag: 'mds_snaptable' 2024-04-25T01:18:26.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.056+0000 7f7f6fc1a740 10 datascan.scan_links: step 2: handling object mds0_inotable 2024-04-25T01:18:26.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.056+0000 7f7f6fc1a740 10 datascan.scan_links: Not a dirfrag: 'mds0_inotable' 2024-04-25T01:18:26.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.056+0000 7f7f6fc1a740 10 datascan.scan_links: step 2: handling object 100.00000000 2024-04-25T01:18:26.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.056+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:69 7.13 7:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd0800 con 0x560ec5863000 2024-04-25T01:18:26.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.058+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.7 v2:172.21.15.103:6800/1145679154 14 ==== osd_op_reply(69 100.00000000 [omap-get-vals out=5b] v0'0 uv1 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x560ec4a9c500 con 0x560ec5863000 2024-04-25T01:18:26.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.058+0000 7f7f6fc1a740 10 datascan.scan_links: step 2: handling object mds0_sessionmap 2024-04-25T01:18:26.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.058+0000 7f7f6fc1a740 10 datascan.scan_links: Not a dirfrag: 'mds0_sessionmap' 2024-04-25T01:18:26.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.058+0000 7f7f6fc1a740 10 datascan.scan_links: step 2: handling object 100.00000000.inode 2024-04-25T01:18:26.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.058+0000 7f7f6fc1a740 10 datascan.scan_links: Not a dirfrag: '100.00000000.inode' 2024-04-25T01:18:26.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.058+0000 7f7f6fc1a740 10 datascan.scan_links: step 2: handling object 1.00000000 2024-04-25T01:18:26.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.058+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:70 7.1f 7:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50) -- 0x560ec5863400 con 0x560ec4cd0000 2024-04-25T01:18:26.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.059+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.0 v2:172.21.15.90:6816/2625481430 4 ==== osd_op_reply(70 1.00000000 [omap-get-vals out=508b] v0'0 uv3 ondisk = 0) ==== 154+0+508 (crc 0 0 0) 0x560ec4a9c500 con 0x560ec4cd0000 2024-04-25T01:18:26.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.059+0000 7f7f6fc1a740 20 datascan.scan_links: injected ino detected 2024-04-25T01:18:26.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.059+0000 7f7f6fc1a740 20 datascan.scan_links: adding 0x10000000000 for future processing to fix dnfirst 2024-04-25T01:18:26.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.059+0000 7f7f6fc1a740 10 datascan.scan_links: processing 0 dup_primaries, 0 remote_links 2024-04-25T01:18:26.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.059+0000 7f7f6fc1a740 10 datascan.scan_links: removing dup dentries from 0 objects 2024-04-25T01:18:26.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.059+0000 7f7f6fc1a740 10 datascan.scan_links: processing 0 bad_nlink_inos 2024-04-25T01:18:26.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.059+0000 7f7f6fc1a740 10 datascan.scan_links: processing 2 injected_inos 2024-04-25T01:18:26.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.059+0000 7f7f6fc1a740 10 datascan.scan_links: handling injected_ino 0x10000000000 2024-04-25T01:18:26.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.059+0000 7f7f6fc1a740 20 datascan.scan_links: fixing linkage (dnfirst) of 0x1:subdir 2024-04-25T01:18:26.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.059+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:71 7.1f 7:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50) -- 0x560ec5863800 con 0x560ec4cd0000 2024-04-25T01:18:26.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.059+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.0 v2:172.21.15.90:6816/2625481430 5 ==== osd_op_reply(71 1.00000000 [omap-get-vals-by-keys out=507b] v0'0 uv3 ondisk = 0) ==== 154+0+507 (crc 0 0 0) 0x560ec4a9c500 con 0x560ec4cd0000 2024-04-25T01:18:26.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.059+0000 7f7f6fc1a740 20 datascan.read_dentry: oid=1.00000000 dname=subdir frag=*, r=0 2024-04-25T01:18:26.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.059+0000 7f7f6fc1a740 20 datascan.scan_links: first is now 2 2024-04-25T01:18:26.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.059+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:72 7.1f 7:ff5b34d6:::1.00000000:head [omap-set-vals in=507b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) -- 0x560ec5863c00 con 0x560ec4cd0000 2024-04-25T01:18:26.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.063+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.0 v2:172.21.15.90:6816/2625481430 6 ==== osd_op_reply(72 1.00000000 [omap-set-vals] v50'4 uv4 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x560ec4a9c500 con 0x560ec4cd0000 2024-04-25T01:18:26.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.063+0000 7f7f6fc1a740 20 datascan.inject_linkage: Injected dentry 0x0x1/subdir pointing to 0x0x10000000000 2024-04-25T01:18:26.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.063+0000 7f7f6fc1a740 10 datascan.scan_links: handling injected_ino 0x10000000001 2024-04-25T01:18:26.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.063+0000 7f7f6fc1a740 20 datascan.scan_links: fixing linkage (dnfirst) of 0x10000000000:sixmegs 2024-04-25T01:18:26.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.063+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:73 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd0400 con 0x560ec5863000 2024-04-25T01:18:26.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.064+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.7 v2:172.21.15.103:6800/1145679154 15 ==== osd_op_reply(73 10000000000.00000000 [omap-get-vals-by-keys out=508b] v0'0 uv2 ondisk = 0) ==== 164+0+508 (crc 0 0 0) 0x560ec4a9c500 con 0x560ec5863000 2024-04-25T01:18:26.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.064+0000 7f7f6fc1a740 20 datascan.read_dentry: oid=10000000000.00000000 dname=sixmegs frag=*, r=0 2024-04-25T01:18:26.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.064+0000 7f7f6fc1a740 20 datascan.scan_links: first is now 2 2024-04-25T01:18:26.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.064+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:74 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=508b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1400 con 0x560ec5863000 2024-04-25T01:18:26.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.069+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.7 v2:172.21.15.103:6800/1145679154 16 ==== osd_op_reply(74 10000000000.00000000 [omap-set-vals] v50'3 uv3 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x560ec4a9c500 con 0x560ec5863000 2024-04-25T01:18:26.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.069+0000 7f7f6fc1a740 20 datascan.inject_linkage: Injected dentry 0x0x10000000000/sixmegs pointing to 0x0x10000000001 2024-04-25T01:18:26.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.069+0000 7f7f6fc1a740 10 datascan.scan_links: updating inotable 2024-04-25T01:18:26.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.069+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:75 7.13 7:c91d4a1d:::mds0_inotable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1800 con 0x560ec5863000 2024-04-25T01:18:26.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.071+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.7 v2:172.21.15.103:6800/1145679154 17 ==== osd_op_reply(75 mds0_inotable [read 0~34 out=34b] v0'0 uv2 ondisk = 0) ==== 157+0+34 (crc 0 0 0) 0x560ec4a9c280 con 0x560ec5863000 2024-04-25T01:18:26.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.071+0000 7f7f6fc1a740 10 mds.0.inotable: skip_inos was [0x10000000000~0x10000000000] 2024-04-25T01:18:26.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.071+0000 7f7f6fc1a740 10 mds.0.inotable: skip_inos now [0x10000000002~0xfffffffffe] 2024-04-25T01:18:26.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.071+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:76 7.13 7:c91d4a1d:::mds0_inotable:head [writefull 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1c00 con 0x560ec5863000 2024-04-25T01:18:26.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.075+0000 7f7f6dc00640 1 -- 172.21.15.90:0/1398212762 <== osd.7 v2:172.21.15.103:6800/1145679154 18 ==== osd_op_reply(76 mds0_inotable [writefull 0~34] v50'3 uv3 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x560ec4a9c280 con 0x560ec5863000 2024-04-25T01:18:26.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.075+0000 7f7f6fc1a740 10 datascan.scan_links: updating snaptable 2024-04-25T01:18:26.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.075+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- osd_op(unknown.0.0:77 7.d 7:b50e409b:::mds_snaptable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50) -- 0x560ec4cd1000 con 0x560ec5862c00 2024-04-25T01:18:26.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.077+0000 7f7f6c800640 1 -- 172.21.15.90:0/1398212762 <== osd.1 v2:172.21.15.90:6800/2685473653 11 ==== osd_op_reply(77 mds_snaptable [read 0~70 out=70b] v0'0 uv1 ondisk = 0) ==== 157+0+70 (crc 0 0 0) 0x560ec4a9c000 con 0x560ec5862c00 2024-04-25T01:18:26.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.077+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x560ec4cd0000 msgr2=0x560ec4cb8c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.077+0000 7f7f6fc1a740 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x560ec4cd0000 0x560ec4cb8c00 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.077+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x560ec5862c00 msgr2=0x560ec49c6100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.077+0000 7f7f6fc1a740 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x560ec5862c00 0x560ec49c6100 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.077+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 >> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] conn(0x560ec4cb0c00 msgr2=0x560ec4cb7080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.077+0000 7f7f6fc1a740 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] conn(0x560ec4cb0c00 0x560ec4cb7080 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.077+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x560ec4cb1000 msgr2=0x560ec4cb7b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.077+0000 7f7f6fc1a740 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x560ec4cb1000 0x560ec4cb7b80 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.077+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x560ec56b7c00 msgr2=0x560ec4cb6b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.077+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 reap_dead start 2024-04-25T01:18:26.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.077+0000 7f7f6fc1a740 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x560ec56b7c00 0x560ec4cb6b00 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.077+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 reap_dead start 2024-04-25T01:18:26.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.077+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x560ec4cb0800 msgr2=0x560ec4cb6580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.077+0000 7f7f6fc1a740 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x560ec4cb0800 0x560ec4cb6580 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.077+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x560ec4cb1800 msgr2=0x560ec4cb8680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.077+0000 7f7f6fc1a740 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x560ec4cb1800 0x560ec4cb8680 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.077+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x560ec5863000 msgr2=0x560ec4cb6000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.077+0000 7f7f6fc1a740 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x560ec5863000 0x560ec4cb6000 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.078+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x560ec5862800 msgr2=0x560ec49c7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.078+0000 7f7f6fc1a740 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x560ec5862800 0x560ec49c7700 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x560ec56baed0 tx=0x560ec4c2cf00 comp rx=0 tx=0).stop 2024-04-25T01:18:26.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.078+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec57ff400 msgr2=0x560ec49c6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.078+0000 7f7f6fc1a740 1 --2- 172.21.15.90:0/1398212762 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec57ff400 0x560ec49c6c00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x560ec57fdcb0 tx=0x560ec56bcfc0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.078+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 reap_dead start 2024-04-25T01:18:26.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.078+0000 7f7f6d200640 1 -- 172.21.15.90:0/1398212762 reap_dead start 2024-04-25T01:18:26.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.078+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 shutdown_connections 2024-04-25T01:18:26.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.078+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 >> 172.21.15.90:0/1398212762 conn(0x560ec57fec00 msgr2=0x560ec4a0d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:26.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.079+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 shutdown_connections 2024-04-25T01:18:26.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.079+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1398212762 wait complete. 2024-04-25T01:18:26.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.079+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1422867676 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec4921c00 msgr2=0x560ec49c5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.079+0000 7f7f6fc1a740 1 --2- 172.21.15.90:0/1422867676 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec4921c00 0x560ec49c5080 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x560ec573d260 tx=0x560ec56bcde0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.080+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1422867676 shutdown_connections 2024-04-25T01:18:26.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.080+0000 7f7f6fc1a740 1 --2- 172.21.15.90:0/1422867676 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec4921c00 0x560ec49c5080 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.080+0000 7f7f6fc1a740 1 --2- 172.21.15.90:0/1422867676 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x560ec4921800 0x560ec49c4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.080+0000 7f7f6fc1a740 1 --2- 172.21.15.90:0/1422867676 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ec4920c00 0x560ec49c4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.080+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1422867676 >> 172.21.15.90:0/1422867676 conn(0x560ec4921000 msgr2=0x560ec4a0d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:26.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.080+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1422867676 shutdown_connections 2024-04-25T01:18:26.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.080+0000 7f7f6fc1a740 1 -- 172.21.15.90:0/1422867676 wait complete. 2024-04-25T01:18:26.074 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_links', '--filesystem', 'recovery_fs'] in time 0:00:00.219055, result: 2024-04-25T01:18:26.075 DEBUG:teuthology.orchestra.run.smithi090:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank recovery_fs:0 event recover_dentries list --alternate-pool recovery_fs_metadata 2024-04-25T01:18:26.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.199+0000 7f9e5f888280 1 -- start start 2024-04-25T01:18:26.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.199+0000 7f9e5f888280 1 Processor -- start 2024-04-25T01:18:26.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.200+0000 7f9e5f888280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562407220c00 0x56240728db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.200+0000 7f9e5f888280 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562407221800 0x56240728d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.200+0000 7f9e5f888280 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562407221c00 0x56240728d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.200+0000 7f9e5f888280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x562407f84180 con 0x562407220c00 2024-04-25T01:18:26.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.200+0000 7f9e5f888280 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562407282900 con 0x562407221c00 2024-04-25T01:18:26.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.200+0000 7f9e5f888280 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x562407282480 con 0x562407221800 2024-04-25T01:18:26.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.200+0000 7f9e5da00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562407220c00 0x56240728db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.200+0000 7f9e5da00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562407220c00 0x56240728db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47754/0 (socket says 172.21.15.90:47754) 2024-04-25T01:18:26.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.200+0000 7f9e5da00640 1 -- 172.21.15.90:0/4195492242 learned_addr learned my addr 172.21.15.90:0/4195492242 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:26.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.200+0000 7f9e5e400640 1 --2- 172.21.15.90:0/4195492242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562407221c00 0x56240728d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.200+0000 7f9e5da00640 1 -- 172.21.15.90:0/4195492242 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562407221800 msgr2=0x56240728d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:26.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.200+0000 7f9e5d000640 1 --2- 172.21.15.90:0/4195492242 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562407221800 0x56240728d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.200+0000 7f9e5da00640 1 --2- 172.21.15.90:0/4195492242 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562407221800 0x56240728d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.200+0000 7f9e5da00640 1 -- 172.21.15.90:0/4195492242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562407221c00 msgr2=0x56240728d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:26.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.200+0000 7f9e5da00640 1 --2- 172.21.15.90:0/4195492242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562407221c00 0x56240728d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.200+0000 7f9e5da00640 1 -- 172.21.15.90:0/4195492242 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562407254780 con 0x562407220c00 2024-04-25T01:18:26.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.201+0000 7f9e5da00640 1 --2- 172.21.15.90:0/4195492242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562407220c00 0x56240728db80 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x562408045830 tx=0x562407fb4d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d531bc92141ea6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:26.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.201+0000 7f9e5a800640 1 -- 172.21.15.90:0/4195492242 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x562407246680 con 0x562407220c00 2024-04-25T01:18:26.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.201+0000 7f9e5a800640 1 -- 172.21.15.90:0/4195492242 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x56240724c700 con 0x562407220c00 2024-04-25T01:18:26.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.201+0000 7f9e5a800640 1 -- 172.21.15.90:0/4195492242 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x562407246b60 con 0x562407220c00 2024-04-25T01:18:26.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.201+0000 7f9e5f888280 1 -- 172.21.15.90:0/4195492242 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624072554a0 con 0x562407220c00 2024-04-25T01:18:26.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.201+0000 7f9e5f888280 1 -- 172.21.15.90:0/4195492242 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) -- 0x562407254000 con 0x562407220c00 2024-04-25T01:18:26.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.201+0000 7f9e5f888280 4 waiting for MDS map... 2024-04-25T01:18:26.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.201+0000 7f9e5a800640 1 -- 172.21.15.90:0/4195492242 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x5624080f0600 con 0x562407220c00 2024-04-25T01:18:26.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.202+0000 7f9e5a800640 1 -- 172.21.15.90:0/4195492242 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 23) ==== 4135+0+0 (secure 0 0 0) 0x56240727b180 con 0x562407220c00 2024-04-25T01:18:26.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.202+0000 7f9e5f888280 4 Got MDS map 23 2024-04-25T01:18:26.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.202+0000 7f9e5f888280 10 main: JournalTool::main 2024-04-25T01:18:26.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.202+0000 7f9e5f888280 4 main: JournalTool: connecting to RADOS... 2024-04-25T01:18:26.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.202+0000 7f9e5f888280 1 Processor -- start 2024-04-25T01:18:26.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.203+0000 7f9e5f888280 1 -- start start 2024-04-25T01:18:26.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.203+0000 7f9e5f888280 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624080f5c00 0x56240728cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.204+0000 7f9e5f888280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56240815a000 0x56240728c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.204+0000 7f9e5f888280 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56240815a400 0x56240728e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.204+0000 7f9e5f888280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x562407282180 con 0x56240815a000 2024-04-25T01:18:26.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.204+0000 7f9e5f888280 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562407f85500 con 0x5624080f5c00 2024-04-25T01:18:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.204+0000 7f9e5f888280 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x562407f85380 con 0x56240815a400 2024-04-25T01:18:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.204+0000 7f9e5da00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56240815a000 0x56240728c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.204+0000 7f9e5da00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56240815a000 0x56240728c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47756/0 (socket says 172.21.15.90:47756) 2024-04-25T01:18:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.204+0000 7f9e5da00640 1 -- 172.21.15.90:0/2707442223 learned_addr learned my addr 172.21.15.90:0/2707442223 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.204+0000 7f9e5e400640 1 --2- 172.21.15.90:0/2707442223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624080f5c00 0x56240728cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.204+0000 7f9e5da00640 1 -- 172.21.15.90:0/2707442223 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56240815a400 msgr2=0x56240728e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.204+0000 7f9e5da00640 1 --2- 172.21.15.90:0/2707442223 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56240815a400 0x56240728e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.204+0000 7f9e5da00640 1 -- 172.21.15.90:0/2707442223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624080f5c00 msgr2=0x56240728cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.204+0000 7f9e5da00640 1 --2- 172.21.15.90:0/2707442223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624080f5c00 0x56240728cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.204+0000 7f9e5da00640 1 -- 172.21.15.90:0/2707442223 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624072543c0 con 0x56240815a000 2024-04-25T01:18:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.205+0000 7f9e5da00640 1 --2- 172.21.15.90:0/2707442223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56240815a000 0x56240728c580 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x562408045ef0 tx=0x562407fb40c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bad6ce2fed9cf6b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.205+0000 7f9e58000640 1 -- 172.21.15.90:0/2707442223 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x562407246820 con 0x56240815a000 2024-04-25T01:18:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.205+0000 7f9e58000640 1 -- 172.21.15.90:0/2707442223 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x56240724cfc0 con 0x56240815a000 2024-04-25T01:18:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.205+0000 7f9e58000640 1 -- 172.21.15.90:0/2707442223 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x562407246680 con 0x56240815a000 2024-04-25T01:18:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.205+0000 7f9e5f888280 1 -- 172.21.15.90:0/2707442223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56240815a000 msgr2=0x56240728c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.205+0000 7f9e5f888280 1 --2- 172.21.15.90:0/2707442223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56240815a000 0x56240728c580 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x562408045ef0 tx=0x562407fb40c0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.205+0000 7f9e5f888280 1 -- 172.21.15.90:0/2707442223 shutdown_connections 2024-04-25T01:18:26.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.205+0000 7f9e5f888280 1 --2- 172.21.15.90:0/2707442223 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56240815a400 0x56240728e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.205+0000 7f9e5f888280 1 --2- 172.21.15.90:0/2707442223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56240815a000 0x56240728c580 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.205+0000 7f9e5f888280 1 --2- 172.21.15.90:0/2707442223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624080f5c00 0x56240728cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.205+0000 7f9e5f888280 1 -- 172.21.15.90:0/2707442223 >> 172.21.15.90:0/2707442223 conn(0x5624080f5800 msgr2=0x56240732d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:26.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.206+0000 7f9e5f888280 1 -- 172.21.15.90:0/2707442223 shutdown_connections 2024-04-25T01:18:26.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.206+0000 7f9e5f888280 1 -- 172.21.15.90:0/2707442223 wait complete. 2024-04-25T01:18:26.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.206+0000 7f9e5f888280 1 Processor -- start 2024-04-25T01:18:26.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.206+0000 7f9e5f888280 1 -- start start 2024-04-25T01:18:26.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.207+0000 7f9e5f888280 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624080f5800 0x56240728cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.207+0000 7f9e5f888280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56240815b000 0x56240728c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.207+0000 7f9e5f888280 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56240815ac00 0x56240728e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.207+0000 7f9e5f888280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x562407f84780 con 0x56240815b000 2024-04-25T01:18:26.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.207+0000 7f9e5f888280 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562407f84900 con 0x5624080f5800 2024-04-25T01:18:26.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.207+0000 7f9e5f888280 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x562407f84600 con 0x56240815ac00 2024-04-25T01:18:26.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.207+0000 7f9e5da00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56240815b000 0x56240728c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.208+0000 7f9e5da00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56240815b000 0x56240728c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47760/0 (socket says 172.21.15.90:47760) 2024-04-25T01:18:26.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.208+0000 7f9e5da00640 1 -- 172.21.15.90:0/1456635541 learned_addr learned my addr 172.21.15.90:0/1456635541 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:26.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.208+0000 7f9e5d000640 1 --2- 172.21.15.90:0/1456635541 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56240815ac00 0x56240728e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.208+0000 7f9e5da00640 1 -- 172.21.15.90:0/1456635541 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56240815ac00 msgr2=0x56240728e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.208+0000 7f9e5da00640 1 --2- 172.21.15.90:0/1456635541 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x56240815ac00 0x56240728e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.208+0000 7f9e5da00640 1 -- 172.21.15.90:0/1456635541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624080f5800 msgr2=0x56240728cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.208+0000 7f9e5da00640 1 --2- 172.21.15.90:0/1456635541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624080f5800 0x56240728cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.208+0000 7f9e5da00640 1 -- 172.21.15.90:0/1456635541 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562407254780 con 0x56240815b000 2024-04-25T01:18:26.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.208+0000 7f9e5da00640 1 --2- 172.21.15.90:0/1456635541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56240815b000 0x56240728c580 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x5624081067b0 tx=0x562408109800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:26.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.208+0000 7f9e56200640 1 -- 172.21.15.90:0/1456635541 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x562407246b60 con 0x56240815b000 2024-04-25T01:18:26.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.208+0000 7f9e56200640 1 -- 172.21.15.90:0/1456635541 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x56240724ce00 con 0x56240815b000 2024-04-25T01:18:26.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.208+0000 7f9e5f888280 1 -- 172.21.15.90:0/1456635541 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5624072554a0 con 0x56240815b000 2024-04-25T01:18:26.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.208+0000 7f9e56200640 1 -- 172.21.15.90:0/1456635541 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5624072464e0 con 0x56240815b000 2024-04-25T01:18:26.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.209+0000 7f9e5f888280 1 -- 172.21.15.90:0/1456635541 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562407254960 con 0x56240815b000 2024-04-25T01:18:26.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.209+0000 7f9e5f888280 4 main: JournalTool: resolving pool 7 2024-04-25T01:18:26.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.210+0000 7f9e56200640 1 -- 172.21.15.90:0/1456635541 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x562408148900 con 0x56240815b000 2024-04-25T01:18:26.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.211+0000 7f9e56200640 1 --2- 172.21.15.90:0/1456635541 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x56240759c000 0x5624075a2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.211+0000 7f9e56200640 1 -- 172.21.15.90:0/1456635541 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x562408164600 con 0x56240815b000 2024-04-25T01:18:26.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.211+0000 7f9e5f888280 4 main: JournalTool: creating IoCtx.. 2024-04-25T01:18:26.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.211+0000 7f9e5f888280 4 main: Executing for rank 0 2024-04-25T01:18:26.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.211+0000 7f9e5f888280 1 main_event: Using alternate pool recovery_fs_metadata 2024-04-25T01:18:26.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.211+0000 7f9e5f888280 1 --2- 172.21.15.90:0/1456635541 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x56240815a800 0x56240728e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.211+0000 7f9e5f888280 1 -- 172.21.15.90:0/1456635541 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50) -- 0x56240815a400 con 0x56240815a800 2024-04-25T01:18:26.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.211+0000 7f9e5e400640 1 --2- 172.21.15.90:0/1456635541 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x56240759c000 0x5624075a2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.212+0000 7f9e5e400640 1 --2- 172.21.15.90:0/1456635541 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x56240759c000 0x5624075a2000 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x562407fa2240 tx=0x562407fb4de0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:26.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.212+0000 7f9e5d000640 1 --2- 172.21.15.90:0/1456635541 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x56240815a800 0x56240728e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.213+0000 7f9e5d000640 1 --2- 172.21.15.90:0/1456635541 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x56240815a800 0x56240728e680 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:26.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.213+0000 7f9e5d000640 1 -- 172.21.15.90:0/1456635541 <== osd.7 v2:172.21.15.103:6800/1145679154 1 ==== osd_op_reply(1 400.00000000 [read 0~2147483647] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x56240731c000 con 0x56240815a800 2024-04-25T01:18:26.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.214+0000 7f9e5f888280 -1 Pointer 400.00000000 is absent 2024-04-25T01:18:26.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.214+0000 7f9e5f888280 4 main_event: consumed 0 inodes 2024-04-25T01:18:26.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.214+0000 7f9e5f888280 1 --2- 172.21.15.90:0/1456635541 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x56240815a000 0x56240728ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.214+0000 7f9e5f888280 1 -- 172.21.15.90:0/1456635541 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- osd_op(unknown.0.0:2 7.3 7:c0ce80d4:::4.00000000:head [omap-rm-keys in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) -- 0x56240759c400 con 0x56240815a000 2024-04-25T01:18:26.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.214+0000 7f9e5e400640 1 --2- 172.21.15.90:0/1456635541 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x56240815a000 0x56240728ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.215+0000 7f9e5e400640 1 --2- 172.21.15.90:0/1456635541 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x56240815a000 0x56240728ec00 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:26.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.218+0000 7f9e5e400640 1 -- 172.21.15.90:0/1456635541 <== osd.5 v2:172.21.15.103:6801/1348846168 1 ==== osd_op_reply(2 4.00000000 [omap-rm-keys] v50'1 uv0 ondisk = -2 ((2) No such file or directory)) ==== 154+0+0 (crc 0 0 0) 0x56240731c500 con 0x56240815a000 2024-04-25T01:18:26.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.218+0000 7f9e5f888280 4 0 events 2024-04-25T01:18:26.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.218+0000 7f9e5f888280 1 -- 172.21.15.90:0/1456635541 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x56240815a000 msgr2=0x56240728ec00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.218+0000 7f9e5f888280 1 --2- 172.21.15.90:0/1456635541 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x56240815a000 0x56240728ec00 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.218+0000 7f9e5f888280 1 -- 172.21.15.90:0/1456635541 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x56240815a800 msgr2=0x56240728e680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.218+0000 7f9e5f888280 1 --2- 172.21.15.90:0/1456635541 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x56240815a800 0x56240728e680 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.218+0000 7f9e5f888280 1 -- 172.21.15.90:0/1456635541 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x56240759c000 msgr2=0x5624075a2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.218+0000 7f9e5f888280 1 --2- 172.21.15.90:0/1456635541 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x56240759c000 0x5624075a2000 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x562407fa2240 tx=0x562407fb4de0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.218+0000 7f9e5f888280 1 -- 172.21.15.90:0/1456635541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56240815b000 msgr2=0x56240728c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.218+0000 7f9e5f888280 1 --2- 172.21.15.90:0/1456635541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56240815b000 0x56240728c580 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x5624081067b0 tx=0x562408109800 comp rx=0 tx=0).stop 2024-04-25T01:18:26.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.218+0000 7f9e5d000640 1 -- 172.21.15.90:0/1456635541 reap_dead start 2024-04-25T01:18:26.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.218+0000 7f9e5f888280 1 -- 172.21.15.90:0/1456635541 shutdown_connections 2024-04-25T01:18:26.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.218+0000 7f9e5f888280 1 --2- 172.21.15.90:0/1456635541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56240815b000 0x56240728c580 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.218+0000 7f9e5f888280 1 -- 172.21.15.90:0/1456635541 >> 172.21.15.90:0/1456635541 conn(0x5624080f5c00 msgr2=0x56240732d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:26.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.218+0000 7f9e5f888280 1 -- 172.21.15.90:0/1456635541 shutdown_connections 2024-04-25T01:18:26.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.218+0000 7f9e5f888280 1 -- 172.21.15.90:0/1456635541 wait complete. 2024-04-25T01:18:26.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.219+0000 7f9e5f888280 1 -- 172.21.15.90:0/4195492242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562407220c00 msgr2=0x56240728db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.219+0000 7f9e5f888280 1 --2- 172.21.15.90:0/4195492242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562407220c00 0x56240728db80 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x562408045830 tx=0x562407fb4d80 comp rx=0 tx=0).stop 2024-04-25T01:18:26.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.219+0000 7f9e5f888280 1 -- 172.21.15.90:0/4195492242 shutdown_connections 2024-04-25T01:18:26.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.219+0000 7f9e5f888280 1 --2- 172.21.15.90:0/4195492242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562407221c00 0x56240728d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.219+0000 7f9e5f888280 1 --2- 172.21.15.90:0/4195492242 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x562407221800 0x56240728d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.219+0000 7f9e5f888280 1 --2- 172.21.15.90:0/4195492242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562407220c00 0x56240728db80 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.219+0000 7f9e5f888280 1 -- 172.21.15.90:0/4195492242 >> 172.21.15.90:0/4195492242 conn(0x562407221000 msgr2=0x56240732c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:26.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.219+0000 7f9e5f888280 1 -- 172.21.15.90:0/4195492242 shutdown_connections 2024-04-25T01:18:26.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.219+0000 7f9e5f888280 1 -- 172.21.15.90:0/4195492242 wait complete. 2024-04-25T01:18:26.208 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'recovery_fs:0', 'event', 'recover_dentries', 'list', '--alternate-pool', 'recovery_fs_metadata'] in time 0:00:00.133943, result: 2024-04-25T01:18:26.209 DEBUG:teuthology.orchestra.run.smithi090:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank recovery_fs:0 journal reset --force 2024-04-25T01:18:26.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.309+0000 7f8ce7e24280 1 -- start start 2024-04-25T01:18:26.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.309+0000 7f8ce7e24280 1 Processor -- start 2024-04-25T01:18:26.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.310+0000 7f8ce7e24280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da0677cc00 0x55da067e8580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.310+0000 7f8ce7e24280 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55da0677d800 0x55da067e8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.310+0000 7f8ce7e24280 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da0677dc00 0x55da067e9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.310+0000 7f8ce7e24280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55da07560180 con 0x55da0677cc00 2024-04-25T01:18:26.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.310+0000 7f8ce7e24280 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55da07560300 con 0x55da0677dc00 2024-04-25T01:18:26.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.310+0000 7f8ce7e24280 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55da07560480 con 0x55da0677d800 2024-04-25T01:18:26.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.310+0000 7f8ce5200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da0677cc00 0x55da067e8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.310+0000 7f8ce5200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da0677cc00 0x55da067e8580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47774/0 (socket says 172.21.15.90:47774) 2024-04-25T01:18:26.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.310+0000 7f8ce5200640 1 -- 172.21.15.90:0/883934650 learned_addr learned my addr 172.21.15.90:0/883934650 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:26.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.310+0000 7f8ce5200640 1 -- 172.21.15.90:0/883934650 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55da0677d800 msgr2=0x55da067e8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:26.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.310+0000 7f8ce5200640 1 --2- 172.21.15.90:0/883934650 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55da0677d800 0x55da067e8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.310+0000 7f8ce5200640 1 -- 172.21.15.90:0/883934650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da0677dc00 msgr2=0x55da067e9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:26.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.310+0000 7f8ce5200640 1 --2- 172.21.15.90:0/883934650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da0677dc00 0x55da067e9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.310+0000 7f8ce5200640 1 -- 172.21.15.90:0/883934650 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da067b0780 con 0x55da0677cc00 2024-04-25T01:18:26.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.311+0000 7f8ce5200640 1 --2- 172.21.15.90:0/883934650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da0677cc00 0x55da067e8580 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x55da075a33e0 tx=0x55da0750e660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=580590bf4aaf36be server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:26.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.311+0000 7f8ce2000640 1 -- 172.21.15.90:0/883934650 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55da067a2680 con 0x55da0677cc00 2024-04-25T01:18:26.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.311+0000 7f8ce2000640 1 -- 172.21.15.90:0/883934650 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55da067a8700 con 0x55da0677cc00 2024-04-25T01:18:26.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.311+0000 7f8ce2000640 1 -- 172.21.15.90:0/883934650 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55da067a2b60 con 0x55da0677cc00 2024-04-25T01:18:26.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.311+0000 7f8ce7e24280 1 -- 172.21.15.90:0/883934650 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da067b14a0 con 0x55da0677cc00 2024-04-25T01:18:26.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.311+0000 7f8ce7e24280 1 -- 172.21.15.90:0/883934650 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) -- 0x55da067b0000 con 0x55da0677cc00 2024-04-25T01:18:26.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.311+0000 7f8ce7e24280 4 waiting for MDS map... 2024-04-25T01:18:26.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.311+0000 7f8ce2000640 1 -- 172.21.15.90:0/883934650 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x55da0763e600 con 0x55da0677cc00 2024-04-25T01:18:26.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.311+0000 7f8ce2000640 1 -- 172.21.15.90:0/883934650 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 23) ==== 4135+0+0 (secure 0 0 0) 0x55da067d7180 con 0x55da0677cc00 2024-04-25T01:18:26.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.311+0000 7f8ce7e24280 4 Got MDS map 23 2024-04-25T01:18:26.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.312+0000 7f8ce7e24280 10 main: JournalTool::main 2024-04-25T01:18:26.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.312+0000 7f8ce7e24280 4 main: JournalTool: connecting to RADOS... 2024-04-25T01:18:26.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.312+0000 7f8ce7e24280 1 Processor -- start 2024-04-25T01:18:26.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.313+0000 7f8ce7e24280 1 -- start start 2024-04-25T01:18:26.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.313+0000 7f8ce7e24280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da07641c00 0x55da067e9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.313+0000 7f8ce7e24280 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55da076a2000 0x55da067e9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.313+0000 7f8ce7e24280 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da076a2400 0x55da067ea100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.313+0000 7f8ce7e24280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55da07560600 con 0x55da07641c00 2024-04-25T01:18:26.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.313+0000 7f8ce7e24280 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55da07560780 con 0x55da076a2400 2024-04-25T01:18:26.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.313+0000 7f8ce7e24280 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55da07560900 con 0x55da076a2000 2024-04-25T01:18:26.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.314+0000 7f8ce5c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da07641c00 0x55da067e9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.314+0000 7f8ce5c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da07641c00 0x55da067e9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47782/0 (socket says 172.21.15.90:47782) 2024-04-25T01:18:26.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.314+0000 7f8ce5c00640 1 -- 172.21.15.90:0/2651234609 learned_addr learned my addr 172.21.15.90:0/2651234609 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:26.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.314+0000 7f8ce5200640 1 --2- 172.21.15.90:0/2651234609 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55da076a2000 0x55da067e9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.314+0000 7f8ce5c00640 1 -- 172.21.15.90:0/2651234609 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55da076a2000 msgr2=0x55da067e9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:26.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.314+0000 7f8ce5c00640 1 --2- 172.21.15.90:0/2651234609 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55da076a2000 0x55da067e9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.314+0000 7f8ce5c00640 1 -- 172.21.15.90:0/2651234609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da076a2400 msgr2=0x55da067ea100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:26.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.314+0000 7f8ce4800640 1 --2- 172.21.15.90:0/2651234609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da076a2400 0x55da067ea100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.314+0000 7f8ce5c00640 1 --2- 172.21.15.90:0/2651234609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da076a2400 0x55da067ea100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.314+0000 7f8ce5c00640 1 -- 172.21.15.90:0/2651234609 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da067b03c0 con 0x55da07641c00 2024-04-25T01:18:26.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.314+0000 7f8ce4800640 1 --2- 172.21.15.90:0/2651234609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da076a2400 0x55da067ea100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:26.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.314+0000 7f8ce5c00640 1 --2- 172.21.15.90:0/2651234609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da07641c00 0x55da067e9600 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x55da076527e0 tx=0x55da07656d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=436fdae2bb5878be server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:26.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.315+0000 7f8cdf800640 1 -- 172.21.15.90:0/2651234609 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55da067a29c0 con 0x55da07641c00 2024-04-25T01:18:26.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.315+0000 7f8cdf800640 1 -- 172.21.15.90:0/2651234609 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55da067a8fc0 con 0x55da07641c00 2024-04-25T01:18:26.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.315+0000 7f8cdf800640 1 -- 172.21.15.90:0/2651234609 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55da067a2680 con 0x55da07641c00 2024-04-25T01:18:26.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.315+0000 7f8ce7e24280 1 -- 172.21.15.90:0/2651234609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da07641c00 msgr2=0x55da067e9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.315+0000 7f8ce7e24280 1 --2- 172.21.15.90:0/2651234609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da07641c00 0x55da067e9600 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x55da076527e0 tx=0x55da07656d20 comp rx=0 tx=0).stop 2024-04-25T01:18:26.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.315+0000 7f8ce7e24280 1 -- 172.21.15.90:0/2651234609 shutdown_connections 2024-04-25T01:18:26.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.315+0000 7f8ce7e24280 1 --2- 172.21.15.90:0/2651234609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da076a2400 0x55da067ea100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.315+0000 7f8ce7e24280 1 --2- 172.21.15.90:0/2651234609 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55da076a2000 0x55da067e9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.315+0000 7f8ce7e24280 1 --2- 172.21.15.90:0/2651234609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da07641c00 0x55da067e9600 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.315+0000 7f8ce7e24280 1 -- 172.21.15.90:0/2651234609 >> 172.21.15.90:0/2651234609 conn(0x55da07641800 msgr2=0x55da06861000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:26.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.315+0000 7f8ce7e24280 1 -- 172.21.15.90:0/2651234609 shutdown_connections 2024-04-25T01:18:26.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.315+0000 7f8ce7e24280 1 -- 172.21.15.90:0/2651234609 wait complete. 2024-04-25T01:18:26.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.316+0000 7f8ce7e24280 1 Processor -- start 2024-04-25T01:18:26.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.316+0000 7f8ce7e24280 1 -- start start 2024-04-25T01:18:26.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.317+0000 7f8ce7e24280 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da07641800 0x55da067eb180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.317+0000 7f8ce7e24280 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55da076ac000 0x55da067eac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.317+0000 7f8ce7e24280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da076ac400 0x55da067ea680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.317+0000 7f8ce7e24280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55da067de180 con 0x55da076ac400 2024-04-25T01:18:26.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.317+0000 7f8ce7e24280 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55da07561e00 con 0x55da07641800 2024-04-25T01:18:26.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.317+0000 7f8ce7e24280 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55da07560d80 con 0x55da076ac000 2024-04-25T01:18:26.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.317+0000 7f8ce4800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da076ac400 0x55da067ea680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.317+0000 7f8ce4800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da076ac400 0x55da067ea680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47788/0 (socket says 172.21.15.90:47788) 2024-04-25T01:18:26.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.317+0000 7f8ce4800640 1 -- 172.21.15.90:0/1366678781 learned_addr learned my addr 172.21.15.90:0/1366678781 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:26.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.317+0000 7f8ce5c00640 1 --2- 172.21.15.90:0/1366678781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da07641800 0x55da067eb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.318+0000 7f8ce5200640 1 --2- 172.21.15.90:0/1366678781 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55da076ac000 0x55da067eac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.318+0000 7f8ce4800640 1 -- 172.21.15.90:0/1366678781 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55da076ac000 msgr2=0x55da067eac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.318+0000 7f8ce4800640 1 --2- 172.21.15.90:0/1366678781 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55da076ac000 0x55da067eac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.318+0000 7f8ce4800640 1 -- 172.21.15.90:0/1366678781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da07641800 msgr2=0x55da067eb180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.318+0000 7f8ce4800640 1 --2- 172.21.15.90:0/1366678781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da07641800 0x55da067eb180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.318+0000 7f8ce4800640 1 -- 172.21.15.90:0/1366678781 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da067b0780 con 0x55da076ac400 2024-04-25T01:18:26.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.318+0000 7f8ce5200640 1 --2- 172.21.15.90:0/1366678781 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55da076ac000 0x55da067eac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:26.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.318+0000 7f8ce4800640 1 --2- 172.21.15.90:0/1366678781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da076ac400 0x55da067ea680 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x55da076535f0 tx=0x55da07657020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:26.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.318+0000 7f8cdda00640 1 -- 172.21.15.90:0/1366678781 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55da067a29c0 con 0x55da076ac400 2024-04-25T01:18:26.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.318+0000 7f8cdda00640 1 -- 172.21.15.90:0/1366678781 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55da067a8e00 con 0x55da076ac400 2024-04-25T01:18:26.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.318+0000 7f8ce7e24280 1 -- 172.21.15.90:0/1366678781 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55da067b14a0 con 0x55da076ac400 2024-04-25T01:18:26.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.318+0000 7f8cdda00640 1 -- 172.21.15.90:0/1366678781 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55da067a2680 con 0x55da076ac400 2024-04-25T01:18:26.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.318+0000 7f8ce7e24280 1 -- 172.21.15.90:0/1366678781 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da067b0960 con 0x55da076ac400 2024-04-25T01:18:26.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.318+0000 7f8ce7e24280 4 main: JournalTool: resolving pool 7 2024-04-25T01:18:26.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.320+0000 7f8cdda00640 1 -- 172.21.15.90:0/1366678781 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x55da07696900 con 0x55da076ac400 2024-04-25T01:18:26.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.320+0000 7f8cdda00640 1 --2- 172.21.15.90:0/1366678781 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x55da0757cc00 0x55da067eb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.321+0000 7f8cdda00640 1 -- 172.21.15.90:0/1366678781 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x55da076aea00 con 0x55da076ac400 2024-04-25T01:18:26.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.321+0000 7f8ce7e24280 4 main: JournalTool: creating IoCtx.. 2024-04-25T01:18:26.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.321+0000 7f8ce7e24280 4 main: Executing for rank 0 2024-04-25T01:18:26.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.321+0000 7f8ce5c00640 1 --2- 172.21.15.90:0/1366678781 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x55da0757cc00 0x55da067eb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.321+0000 7f8ce5c00640 1 --2- 172.21.15.90:0/1366678781 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x55da0757cc00 0x55da067eb700 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x55da076527e0 tx=0x55da07656c00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:26.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.322+0000 7f8ce7e24280 1 -- start start 2024-04-25T01:18:26.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.322+0000 7f8ce7e24280 1 Processor -- start 2024-04-25T01:18:26.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.323+0000 7f8ce7e24280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da076acc00 0x55da067ea100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.323+0000 7f8ce7e24280 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55da076ad000 0x55da067e9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.323+0000 7f8ce7e24280 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da076ad400 0x55da067e9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.323+0000 7f8ce7e24280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55da07561080 con 0x55da076acc00 2024-04-25T01:18:26.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.323+0000 7f8ce7e24280 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55da07561b00 con 0x55da076ad400 2024-04-25T01:18:26.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.323+0000 7f8ce7e24280 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55da07561c80 con 0x55da076ad000 2024-04-25T01:18:26.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.323+0000 7f8ce5c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da076acc00 0x55da067ea100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.323+0000 7f8ce5c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da076acc00 0x55da067ea100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47804/0 (socket says 172.21.15.90:47804) 2024-04-25T01:18:26.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.323+0000 7f8ce5c00640 1 -- 172.21.15.90:0/736504668 learned_addr learned my addr 172.21.15.90:0/736504668 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:26.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.323+0000 7f8ce5200640 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55da076ad000 0x55da067e9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.323+0000 7f8ce5c00640 1 -- 172.21.15.90:0/736504668 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55da076ad000 msgr2=0x55da067e9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:26.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.323+0000 7f8ce5c00640 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55da076ad000 0x55da067e9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.323+0000 7f8ce5c00640 1 -- 172.21.15.90:0/736504668 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da076ad400 msgr2=0x55da067e9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:26.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.323+0000 7f8ce5c00640 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da076ad400 0x55da067e9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.323+0000 7f8ce5c00640 1 -- 172.21.15.90:0/736504668 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da067b1680 con 0x55da076acc00 2024-04-25T01:18:26.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.324+0000 7f8ce5c00640 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da076acc00 0x55da067ea100 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x55da06a9b8c0 tx=0x55da07656de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dde8abe1353987cc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:26.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.324+0000 7f8cd9e00640 1 -- 172.21.15.90:0/736504668 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55da067a2b60 con 0x55da076acc00 2024-04-25T01:18:26.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.324+0000 7f8cd9e00640 1 -- 172.21.15.90:0/736504668 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55da067a8c40 con 0x55da076acc00 2024-04-25T01:18:26.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.324+0000 7f8cd9e00640 1 -- 172.21.15.90:0/736504668 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55da067a24e0 con 0x55da076acc00 2024-04-25T01:18:26.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.324+0000 7f8ce7e24280 1 -- 172.21.15.90:0/736504668 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da067b0780 con 0x55da076acc00 2024-04-25T01:18:26.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.324+0000 7f8ce7e24280 1 -- 172.21.15.90:0/736504668 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) -- 0x55da067b14a0 con 0x55da076acc00 2024-04-25T01:18:26.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.324+0000 7f8ce7e24280 4 waiting for MDS map... 2024-04-25T01:18:26.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.324+0000 7f8cd9e00640 1 -- 172.21.15.90:0/736504668 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x55da06b3e200 con 0x55da076acc00 2024-04-25T01:18:26.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.324+0000 7f8cd9e00640 1 -- 172.21.15.90:0/736504668 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 23) ==== 4135+0+0 (secure 0 0 0) 0x55da067d7500 con 0x55da076acc00 2024-04-25T01:18:26.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.324+0000 7f8ce7e24280 4 Got MDS map 23 2024-04-25T01:18:26.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.325+0000 7f8ce7e24280 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x55da076ad800 0x55da06b56000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.325+0000 7f8ce7e24280 1 -- 172.21.15.90:0/736504668 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [writefull 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) -- 0x55da076adc00 con 0x55da076ad800 2024-04-25T01:18:26.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.325+0000 7f8ce5200640 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x55da076ad800 0x55da06b56000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.326+0000 7f8ce5200640 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x55da076ad800 0x55da06b56000 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=26ceb1677c87e986 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:26.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.330+0000 7f8ce5200640 1 -- 172.21.15.90:0/736504668 <== osd.7 v2:172.21.15.103:6800/1145679154 1 ==== osd_op_reply(1 400.00000000 [writefull 0~22] v50'1 uv1 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x55da06858000 con 0x55da076ad800 2024-04-25T01:18:26.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.330+0000 7f8ce7e24280 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x55da06b4b000 0x55da06b56580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.330+0000 7f8ce7e24280 1 -- 172.21.15.90:0/736504668 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- osd_op(unknown.0.0:2 7.14 7:292cf221:::200.00000000:head [writefull 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) -- 0x55da06b4b400 con 0x55da06b4b000 2024-04-25T01:18:26.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.331+0000 7f8ce4800640 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x55da06b4b000 0x55da06b56580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.332+0000 7f8ce4800640 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x55da06b4b000 0x55da06b56580 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=2d6783541820a427 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:26.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.336+0000 7f8ce4800640 1 -- 172.21.15.90:0/736504668 <== osd.5 v2:172.21.15.103:6801/1348846168 1 ==== osd_op_reply(2 200.00000000 [writefull 0~90] v50'2 uv2 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x55da06858500 con 0x55da06b4b000 2024-04-25T01:18:26.320 INFO:teuthology.orchestra.run.smithi090.stdout:writing EResetJournal entry 2024-04-25T01:18:26.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.336+0000 7f8ce7e24280 1 -- 172.21.15.90:0/736504668 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- osd_op(unknown.0.0:3 7.14 7:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) -- 0x55da06b4b800 con 0x55da06b4b000 2024-04-25T01:18:26.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.336+0000 7f8ce7e24280 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x55da06b4bc00 0x55da06b56b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.336+0000 7f8ce7e24280 1 -- 172.21.15.90:0/736504668 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- osd_op(unknown.0.0:4 7.1b 7:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) -- 0x55da06b72000 con 0x55da06b4bc00 2024-04-25T01:18:26.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.336+0000 7f8ce7e24280 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x55da06b72400 0x55da06b57080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.336+0000 7f8ce7e24280 1 -- 172.21.15.90:0/736504668 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- osd_op(unknown.0.0:5 7.b 7:d5c7a900:::200.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) -- 0x55da06b72800 con 0x55da06b72400 2024-04-25T01:18:26.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.336+0000 7f8ce7e24280 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x55da06b72c00 0x55da06b57600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.336+0000 7f8ce7e24280 1 -- 172.21.15.90:0/736504668 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- osd_op(unknown.0.0:6 7.1f 7:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) -- 0x55da06b73000 con 0x55da06b72c00 2024-04-25T01:18:26.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.336+0000 7f8ce7e24280 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x55da06b73400 0x55da06b57b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.336+0000 7f8ce7e24280 1 -- 172.21.15.90:0/736504668 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- osd_op(unknown.0.0:7 7.1a 7:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) -- 0x55da06b73800 con 0x55da06b73400 2024-04-25T01:18:26.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.336+0000 7f8ce7e24280 1 -- 172.21.15.90:0/736504668 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- osd_op(unknown.0.0:8 7.1e 7:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) -- 0x55da06b73c00 con 0x55da076ad800 2024-04-25T01:18:26.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.336+0000 7f8ce4800640 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x55da06b72c00 0x55da06b57600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.336+0000 7f8ce5c00640 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x55da06b73400 0x55da06b57b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.337+0000 7f8ce5c00640 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x55da06b4bc00 0x55da06b56b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.337+0000 7f8ce5200640 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x55da06b72400 0x55da06b57080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.337+0000 7f8ce4800640 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x55da06b72c00 0x55da06b57600 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=6d0d38d7c9ea6ce5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:26.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.337+0000 7f8ce5c00640 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x55da06b73400 0x55da06b57b80 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=672b039c55499129 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:26.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.338+0000 7f8ce5c00640 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x55da06b4bc00 0x55da06b56b00 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=ae265cdc789e6d10 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:26.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.338+0000 7f8ce5200640 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x55da06b72400 0x55da06b57080 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=f638b7bd326ee3fa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:26.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.339+0000 7f8ce5c00640 1 -- 172.21.15.90:0/736504668 <== osd.3 v2:172.21.15.90:6812/8683104 1 ==== osd_op_reply(7 200.00000005 [delete] v50'1 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55da06858780 con 0x55da06b73400 2024-04-25T01:18:26.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.340+0000 7f8ce4800640 1 -- 172.21.15.90:0/736504668 <== osd.0 v2:172.21.15.90:6816/2625481430 1 ==== osd_op_reply(6 200.00000004 [delete] v50'5 uv4 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55da06858500 con 0x55da06b72c00 2024-04-25T01:18:26.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.341+0000 7f8ce5200640 1 -- 172.21.15.90:0/736504668 <== osd.7 v2:172.21.15.103:6800/1145679154 2 ==== osd_op_reply(8 200.00000006 [delete] v50'1 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55da06858000 con 0x55da076ad800 2024-04-25T01:18:26.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.342+0000 7f8ce5200640 1 -- 172.21.15.90:0/736504668 <== osd.6 v2:172.21.15.103:6816/2008388906 1 ==== osd_op_reply(5 200.00000003 [delete] v50'2 uv1 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55da06858000 con 0x55da06b72400 2024-04-25T01:18:26.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.343+0000 7f8ce4800640 1 -- 172.21.15.90:0/736504668 <== osd.5 v2:172.21.15.103:6801/1348846168 2 ==== osd_op_reply(3 200.00000001 [delete] v50'3 uv2 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55da06858500 con 0x55da06b4b000 2024-04-25T01:18:26.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.346+0000 7f8ce5c00640 1 -- 172.21.15.90:0/736504668 <== osd.4 v2:172.21.15.103:6821/1971061680 1 ==== osd_op_reply(4 200.00000002 [delete] v50'1 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55da06858780 con 0x55da06b4bc00 2024-04-25T01:18:26.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.346+0000 7f8cd8000640 1 -- 172.21.15.90:0/736504668 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- osd_op(unknown.0.0:9 7.14 7:2e2fa760:::200.00000001:head [write 0~48 in=48b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) -- 0x55da06b8c000 con 0x55da06b4b000 2024-04-25T01:18:26.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce4800640 1 -- 172.21.15.90:0/736504668 <== osd.5 v2:172.21.15.103:6801/1348846168 3 ==== osd_op_reply(9 200.00000001 [write 0~48] v50'4 uv4 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x55da06858500 con 0x55da06b4b000 2024-04-25T01:18:26.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 4 Successfully wrote new journal pointer and header for rank 3:0 2024-04-25T01:18:26.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 1 -- 172.21.15.90:0/736504668 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x55da06b72c00 msgr2=0x55da06b57600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x55da06b72c00 0x55da06b57600 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 1 -- 172.21.15.90:0/736504668 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x55da06b73400 msgr2=0x55da06b57b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x55da06b73400 0x55da06b57b80 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 1 -- 172.21.15.90:0/736504668 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x55da06b4bc00 msgr2=0x55da06b56b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x55da06b4bc00 0x55da06b56b00 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 1 -- 172.21.15.90:0/736504668 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x55da06b4b000 msgr2=0x55da06b56580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x55da06b4b000 0x55da06b56580 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 1 -- 172.21.15.90:0/736504668 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x55da06b72400 msgr2=0x55da06b57080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x55da06b72400 0x55da06b57080 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 1 -- 172.21.15.90:0/736504668 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x55da076ad800 msgr2=0x55da06b56000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x55da076ad800 0x55da06b56000 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce5200640 1 -- 172.21.15.90:0/736504668 reap_dead start 2024-04-25T01:18:26.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce5200640 1 -- 172.21.15.90:0/736504668 reap_dead start 2024-04-25T01:18:26.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 1 -- 172.21.15.90:0/736504668 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da076acc00 msgr2=0x55da067ea100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da076acc00 0x55da067ea100 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x55da06a9b8c0 tx=0x55da07656de0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 1 -- 172.21.15.90:0/736504668 shutdown_connections 2024-04-25T01:18:26.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x55da076ad800 0x55da06b56000 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 1 --2- 172.21.15.90:0/736504668 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da076acc00 0x55da067ea100 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 1 -- 172.21.15.90:0/736504668 >> 172.21.15.90:0/736504668 conn(0x55da076ac800 msgr2=0x55da06861800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:26.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 1 -- 172.21.15.90:0/736504668 shutdown_connections 2024-04-25T01:18:26.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 1 -- 172.21.15.90:0/736504668 wait complete. 2024-04-25T01:18:26.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 1 -- 172.21.15.90:0/1366678781 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x55da0757cc00 msgr2=0x55da067eb700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 1 --2- 172.21.15.90:0/1366678781 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x55da0757cc00 0x55da067eb700 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x55da076527e0 tx=0x55da07656c00 comp rx=0 tx=0).stop 2024-04-25T01:18:26.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 1 -- 172.21.15.90:0/1366678781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da076ac400 msgr2=0x55da067ea680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 1 --2- 172.21.15.90:0/1366678781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da076ac400 0x55da067ea680 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x55da076535f0 tx=0x55da07657020 comp rx=0 tx=0).stop 2024-04-25T01:18:26.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 1 -- 172.21.15.90:0/1366678781 shutdown_connections 2024-04-25T01:18:26.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 1 --2- 172.21.15.90:0/1366678781 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x55da0757cc00 0x55da067eb700 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 1 --2- 172.21.15.90:0/1366678781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da076ac400 0x55da067ea680 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 1 --2- 172.21.15.90:0/1366678781 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55da076ac000 0x55da067eac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 1 --2- 172.21.15.90:0/1366678781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da07641800 0x55da067eb180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.351+0000 7f8ce7e24280 1 -- 172.21.15.90:0/1366678781 >> 172.21.15.90:0/1366678781 conn(0x55da07641c00 msgr2=0x55da06861000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:26.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.352+0000 7f8ce7e24280 1 -- 172.21.15.90:0/1366678781 shutdown_connections 2024-04-25T01:18:26.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.352+0000 7f8ce7e24280 1 -- 172.21.15.90:0/1366678781 wait complete. 2024-04-25T01:18:26.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.352+0000 7f8ce7e24280 1 -- 172.21.15.90:0/883934650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da0677cc00 msgr2=0x55da067e8580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.352+0000 7f8ce7e24280 1 --2- 172.21.15.90:0/883934650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da0677cc00 0x55da067e8580 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x55da075a33e0 tx=0x55da0750e660 comp rx=0 tx=0).stop 2024-04-25T01:18:26.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.352+0000 7f8ce7e24280 1 -- 172.21.15.90:0/883934650 shutdown_connections 2024-04-25T01:18:26.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.352+0000 7f8ce7e24280 1 --2- 172.21.15.90:0/883934650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da0677dc00 0x55da067e9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.352+0000 7f8ce7e24280 1 --2- 172.21.15.90:0/883934650 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55da0677d800 0x55da067e8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.352+0000 7f8ce7e24280 1 --2- 172.21.15.90:0/883934650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da0677cc00 0x55da067e8580 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.352+0000 7f8ce7e24280 1 -- 172.21.15.90:0/883934650 >> 172.21.15.90:0/883934650 conn(0x55da0677d000 msgr2=0x55da06860800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:26.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.352+0000 7f8ce7e24280 1 -- 172.21.15.90:0/883934650 shutdown_connections 2024-04-25T01:18:26.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.353+0000 7f8ce7e24280 1 -- 172.21.15.90:0/883934650 wait complete. 2024-04-25T01:18:26.343 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'recovery_fs:0', 'journal', 'reset', '--force'] in time 0:00:00.134385, result: writing EResetJournal entry 2024-04-25T01:18:26.343 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set recovery_fs joinable true 2024-04-25T01:18:26.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.563+0000 7f91bfe00640 1 Processor -- start 2024-04-25T01:18:26.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.563+0000 7f91bfe00640 1 -- start start 2024-04-25T01:18:26.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.565+0000 7f91bfe00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f91b8072420 0x7f91b8166d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.565+0000 7f91bfe00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91b80ced40 con 0x7f91b8141c80 2024-04-25T01:18:26.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.565+0000 7f91bfe00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91b80cf020 con 0x7f91b813ef30 2024-04-25T01:18:26.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.565+0000 7f91bfe00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f91b81423d0 con 0x7f91b8072420 2024-04-25T01:18:26.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.565+0000 7f91bca00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f91b8141c80 0x7f91b80ce060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43728/0 (socket says 172.21.15.90:43728) 2024-04-25T01:18:26.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.565+0000 7f91bca00640 1 -- 172.21.15.90:0/4249893872 learned_addr learned my addr 172.21.15.90:0/4249893872 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:26.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.566+0000 7f91bde00640 1 --2- 172.21.15.90:0/4249893872 >> v2:172.21.15.103:3300/0 conn(0x7f91b8072420 0x7f91b8166d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.566+0000 7f91b7e00640 1 -- 172.21.15.90:0/4249893872 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1154851582 0 0) 0x7f91b80ced40 con 0x7f91b8141c80 2024-04-25T01:18:26.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.566+0000 7f91b7e00640 1 -- 172.21.15.90:0/4249893872 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91a0003520 con 0x7f91b8141c80 2024-04-25T01:18:26.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.566+0000 7f91b7e00640 1 -- 172.21.15.90:0/4249893872 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 154520559 0 0) 0x7f91a0003520 con 0x7f91b8141c80 2024-04-25T01:18:26.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.566+0000 7f91b7e00640 1 -- 172.21.15.90:0/4249893872 >> v2:172.21.15.103:3300/0 conn(0x7f91b8072420 msgr2=0x7f91b8166d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:26.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.566+0000 7f91b7e00640 1 --2- 172.21.15.90:0/4249893872 >> v2:172.21.15.103:3300/0 conn(0x7f91b8072420 0x7f91b8166d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.566+0000 7f91b7e00640 1 -- 172.21.15.90:0/4249893872 >> v1:172.21.15.103:6789/0 conn(0x7f91b813ef30 legacy=0x7f91b8141570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.566+0000 7f91b7e00640 1 -- 172.21.15.90:0/4249893872 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91b80cf300 con 0x7f91b8141c80 2024-04-25T01:18:26.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.566+0000 7f91b7e00640 1 -- 172.21.15.90:0/4249893872 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f91a4001670 con 0x7f91b8141c80 2024-04-25T01:18:26.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.566+0000 7f91b7e00640 1 -- 172.21.15.90:0/4249893872 >> v1:172.21.15.90:6789/0 conn(0x7f91b8141c80 legacy=0x7f91b80ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.566+0000 7f91b7e00640 1 --2- 172.21.15.90:0/4249893872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f91a00039b0 0x7f91a0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.566+0000 7f91b7e00640 1 --2- 172.21.15.90:0/4249893872 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91a0004800 0x7f91a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.566+0000 7f91b7e00640 1 --2- 172.21.15.90:0/4249893872 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91a00051c0 0x7f91a000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.566+0000 7f91b7e00640 1 -- 172.21.15.90:0/4249893872 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f91a000c030 con 0x7f91a0004800 2024-04-25T01:18:26.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.566+0000 7f91b7e00640 1 -- 172.21.15.90:0/4249893872 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f91a000c1e0 con 0x7f91a00039b0 2024-04-25T01:18:26.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.566+0000 7f91b7e00640 1 -- 172.21.15.90:0/4249893872 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f91a000c3e0 con 0x7f91a00051c0 2024-04-25T01:18:26.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.567+0000 7f91bca00640 1 --2- 172.21.15.90:0/4249893872 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91a0004800 0x7f91a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.567+0000 7f91bd400640 1 --2- 172.21.15.90:0/4249893872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f91a00039b0 0x7f91a0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.567+0000 7f91bca00640 1 -- 172.21.15.90:0/4249893872 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91a00051c0 msgr2=0x7f91a000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:26.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.567+0000 7f91bca00640 1 --2- 172.21.15.90:0/4249893872 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91a00051c0 0x7f91a000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.567+0000 7f91bca00640 1 -- 172.21.15.90:0/4249893872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f91a00039b0 msgr2=0x7f91a0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:26.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.567+0000 7f91bca00640 1 --2- 172.21.15.90:0/4249893872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f91a00039b0 0x7f91a0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.567+0000 7f91bca00640 1 -- 172.21.15.90:0/4249893872 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91a0003520 con 0x7f91a0004800 2024-04-25T01:18:26.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.567+0000 7f91bde00640 1 -- 172.21.15.90:0/4249893872 reap_dead start 2024-04-25T01:18:26.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.568+0000 7f91bca00640 1 --2- 172.21.15.90:0/4249893872 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91a0004800 0x7f91a0004c80 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f91a400c290 tx=0x7f91a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9006cfe31e88806 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:26.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.568+0000 7f91b7e00640 1 -- 172.21.15.90:0/4249893872 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a4007d30 con 0x7f91a0004800 2024-04-25T01:18:26.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.568+0000 7f91b7e00640 1 -- 172.21.15.90:0/4249893872 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91a4005af0 con 0x7f91a0004800 2024-04-25T01:18:26.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.568+0000 7f91b7e00640 1 -- 172.21.15.90:0/4249893872 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a4007070 con 0x7f91a0004800 2024-04-25T01:18:26.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.569+0000 7f91bfe00640 1 -- 172.21.15.90:0/4249893872 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91a0004800 msgr2=0x7f91a0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.569+0000 7f91bfe00640 1 --2- 172.21.15.90:0/4249893872 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91a0004800 0x7f91a0004c80 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f91a400c290 tx=0x7f91a0005830 comp rx=0 tx=0).stop 2024-04-25T01:18:26.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.569+0000 7f91bfe00640 1 -- 172.21.15.90:0/4249893872 shutdown_connections 2024-04-25T01:18:26.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.569+0000 7f91bfe00640 1 --2- 172.21.15.90:0/4249893872 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91a0004800 0x7f91a0004c80 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.569+0000 7f91bfe00640 1 -- 172.21.15.90:0/4249893872 >> 172.21.15.90:0/4249893872 conn(0x7f91b80b17b0 msgr2=0x7f91b80b1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:26.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.569+0000 7f91bfe00640 1 -- 172.21.15.90:0/4249893872 shutdown_connections 2024-04-25T01:18:26.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.569+0000 7f91bfe00640 1 -- 172.21.15.90:0/4249893872 wait complete. 2024-04-25T01:18:26.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.571+0000 7f91bfe00640 1 Processor -- start 2024-04-25T01:18:26.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.571+0000 7f91bfe00640 1 -- start start 2024-04-25T01:18:26.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.572+0000 7f91bfe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f91a0004800 0x7f91b8152e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.572+0000 7f91bfe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91b81f2f30 0x7f91b80aef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.572+0000 7f91bfe00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91b813cd70 0x7f91b813d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.572+0000 7f91bfe00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f91b8141570 con 0x7f91b81f2f30 2024-04-25T01:18:26.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.572+0000 7f91bfe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f91b8070f80 con 0x7f91a0004800 2024-04-25T01:18:26.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.572+0000 7f91bfe00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f91b81416f0 con 0x7f91b813cd70 2024-04-25T01:18:26.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.572+0000 7f91bca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91b81f2f30 0x7f91b80aef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.572+0000 7f91bca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91b81f2f30 0x7f91b80aef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47812/0 (socket says 172.21.15.90:47812) 2024-04-25T01:18:26.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.572+0000 7f91bca00640 1 -- 172.21.15.90:0/339345470 learned_addr learned my addr 172.21.15.90:0/339345470 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:26.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.573+0000 7f91bd400640 1 --2- 172.21.15.90:0/339345470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f91a0004800 0x7f91b8152e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.573+0000 7f91bca00640 1 -- 172.21.15.90:0/339345470 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91b813cd70 msgr2=0x7f91b813d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.573+0000 7f91bca00640 1 --2- 172.21.15.90:0/339345470 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91b813cd70 0x7f91b813d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.573+0000 7f91bca00640 1 -- 172.21.15.90:0/339345470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f91a0004800 msgr2=0x7f91b8152e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:26.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.573+0000 7f91bca00640 1 --2- 172.21.15.90:0/339345470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f91a0004800 0x7f91b8152e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:26.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.573+0000 7f91bca00640 1 -- 172.21.15.90:0/339345470 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91b80af4b0 con 0x7f91b81f2f30 2024-04-25T01:18:26.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.573+0000 7f91bca00640 1 --2- 172.21.15.90:0/339345470 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91b81f2f30 0x7f91b80aef70 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f91a402add0 tx=0x7f91a4000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:26.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.573+0000 7f91b5600640 1 -- 172.21.15.90:0/339345470 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a4037040 con 0x7f91b81f2f30 2024-04-25T01:18:26.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.573+0000 7f91b5600640 1 -- 172.21.15.90:0/339345470 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91a4041070 con 0x7f91b81f2f30 2024-04-25T01:18:26.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.573+0000 7f91b5600640 1 -- 172.21.15.90:0/339345470 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a40047e0 con 0x7f91b81f2f30 2024-04-25T01:18:26.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.573+0000 7f91bfe00640 1 -- 172.21.15.90:0/339345470 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91b81f33b0 con 0x7f91b81f2f30 2024-04-25T01:18:26.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.573+0000 7f91bfe00640 1 -- 172.21.15.90:0/339345470 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91b81f5e60 con 0x7f91b81f2f30 2024-04-25T01:18:26.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.574+0000 7f91bfe00640 1 -- 172.21.15.90:0/339345470 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91b80ced40 con 0x7f91b81f2f30 2024-04-25T01:18:26.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.575+0000 7f91b5600640 1 -- 172.21.15.90:0/339345470 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f91a4004980 con 0x7f91b81f2f30 2024-04-25T01:18:26.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.576+0000 7f91b5600640 1 --2- 172.21.15.90:0/339345470 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f919806be60 0x7f919806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:26.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.576+0000 7f91b5600640 1 -- 172.21.15.90:0/339345470 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f91a40aafe0 con 0x7f91b81f2f30 2024-04-25T01:18:26.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.577+0000 7f91bd400640 1 --2- 172.21.15.90:0/339345470 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f919806be60 0x7f919806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:26.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.577+0000 7f91bd400640 1 --2- 172.21.15.90:0/339345470 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f919806be60 0x7f919806e320 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f91ac001510 tx=0x7f91ac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:26.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.581+0000 7f91b5600640 1 -- 172.21.15.90:0/339345470 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f91b80ced40 con 0x7f91b81f2f30 2024-04-25T01:18:26.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:26.819+0000 7f91bfe00640 1 -- 172.21.15.90:0/339345470 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "recovery_fs", "var": "joinable", "val": "true"} v 0) -- 0x7f91b80679d0 con 0x7f91b81f2f30 2024-04-25T01:18:27.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.495+0000 7f91b5600640 1 -- 172.21.15.90:0/339345470 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "recovery_fs", "var": "joinable", "val": "true"}]=0 recovery_fs marked joinable; MDS may join as newly active. v24) ==== 172+0+0 (secure 0 0 0) 0x7f91b80679d0 con 0x7f91b81f2f30 2024-04-25T01:18:27.479 INFO:teuthology.orchestra.run.smithi090.stderr:recovery_fs marked joinable; MDS may join as newly active. 2024-04-25T01:18:27.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.498+0000 7f91bfe00640 1 -- 172.21.15.90:0/339345470 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f919806be60 msgr2=0x7f919806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:27.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.498+0000 7f91bfe00640 1 --2- 172.21.15.90:0/339345470 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f919806be60 0x7f919806e320 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f91ac001510 tx=0x7f91ac009040 comp rx=0 tx=0).stop 2024-04-25T01:18:27.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.498+0000 7f91bfe00640 1 -- 172.21.15.90:0/339345470 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91b81f2f30 msgr2=0x7f91b80aef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:27.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.498+0000 7f91bfe00640 1 --2- 172.21.15.90:0/339345470 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91b81f2f30 0x7f91b80aef70 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f91a402add0 tx=0x7f91a4000fe0 comp rx=0 tx=0).stop 2024-04-25T01:18:27.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.498+0000 7f91bfe00640 1 -- 172.21.15.90:0/339345470 shutdown_connections 2024-04-25T01:18:27.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.498+0000 7f91bfe00640 1 --2- 172.21.15.90:0/339345470 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f919806be60 0x7f919806e320 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:27.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.498+0000 7f91bfe00640 1 --2- 172.21.15.90:0/339345470 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91b813cd70 0x7f91b813d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:27.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.498+0000 7f91bfe00640 1 --2- 172.21.15.90:0/339345470 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91b81f2f30 0x7f91b80aef70 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:27.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.498+0000 7f91bfe00640 1 --2- 172.21.15.90:0/339345470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f91a0004800 0x7f91b8152e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:27.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.498+0000 7f91bfe00640 1 -- 172.21.15.90:0/339345470 >> 172.21.15.90:0/339345470 conn(0x7f91b80b17b0 msgr2=0x7f91b813f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:27.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.499+0000 7f91bfe00640 1 -- 172.21.15.90:0/339345470 shutdown_connections 2024-04-25T01:18:27.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.499+0000 7f91bfe00640 1 -- 172.21.15.90:0/339345470 wait complete. 2024-04-25T01:18:27.496 INFO:tasks.ceph.mds.d.smithi103.stderr:2024-04-25T01:18:27.512+0000 7f4f9e600640 -1 mds.0.openfiles _load_finish got (2) No such file or directory 2024-04-25T01:18:27.504 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:18:27.571 INFO:tasks.ceph.mon.a.smithi090.stderr:2024-04-25T01:18:27.587+0000 7f0aa8600640 -1 log_channel(cluster) log [ERR] : Health check update: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T01:18:27.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.715+0000 7f2e37200640 1 Processor -- start 2024-04-25T01:18:27.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.715+0000 7f2e37200640 1 -- start start 2024-04-25T01:18:27.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.716+0000 7f2e37200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2e3815ddd0 0x7f2e38070b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:27.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.716+0000 7f2e37200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e380d0240 0x7f2e380b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:27.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.717+0000 7f2e37200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e3815e1b0 con 0x7f2e38072920 2024-04-25T01:18:27.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.717+0000 7f2e37200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f2e380713b0 con 0x7f2e3815ddd0 2024-04-25T01:18:27.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.717+0000 7f2e37200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2e38071690 con 0x7f2e380d0240 2024-04-25T01:18:27.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.717+0000 7f2e35e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2e3815ddd0 0x7f2e38070b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:27.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.717+0000 7f2e2d400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f2e38072920 0x7f2e380cfb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43742/0 (socket says 172.21.15.90:43742) 2024-04-25T01:18:27.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.717+0000 7f2e2d400640 1 -- 172.21.15.90:0/954112972 learned_addr learned my addr 172.21.15.90:0/954112972 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:27.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.717+0000 7f2e36800640 1 --2- 172.21.15.90:0/954112972 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e380d0240 0x7f2e380b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:27.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.717+0000 7f2e35e00640 1 -- 172.21.15.90:0/954112972 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e380d0240 msgr2=0x7f2e380b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:27.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.717+0000 7f2e35e00640 1 --2- 172.21.15.90:0/954112972 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e380d0240 0x7f2e380b2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:27.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.717+0000 7f2e35e00640 1 -- 172.21.15.90:0/954112972 >> v1:172.21.15.90:6789/0 conn(0x7f2e38072920 legacy=0x7f2e380cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:27.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.717+0000 7f2e35e00640 1 -- 172.21.15.90:0/954112972 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e38071970 con 0x7f2e3815ddd0 2024-04-25T01:18:27.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.717+0000 7f2e36800640 1 --2- 172.21.15.90:0/954112972 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e380d0240 0x7f2e380b2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:27.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.718+0000 7f2e35e00640 1 --2- 172.21.15.90:0/954112972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e3815ddd0 0x7f2e38070b60 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f2e280097a0 tx=0x7f2e2802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a65093bdcee76510 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:27.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.719+0000 7f2e35400640 1 -- 172.21.15.90:0/954112972 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e28037070 con 0x7f2e3815ddd0 2024-04-25T01:18:27.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.719+0000 7f2e35400640 1 -- 172.21.15.90:0/954112972 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e28032440 con 0x7f2e3815ddd0 2024-04-25T01:18:27.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.719+0000 7f2e35400640 1 -- 172.21.15.90:0/954112972 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e28032810 con 0x7f2e3815ddd0 2024-04-25T01:18:27.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.719+0000 7f2e37200640 1 -- 172.21.15.90:0/954112972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e3815ddd0 msgr2=0x7f2e38070b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:27.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.719+0000 7f2e37200640 1 --2- 172.21.15.90:0/954112972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e3815ddd0 0x7f2e38070b60 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f2e280097a0 tx=0x7f2e2802a180 comp rx=0 tx=0).stop 2024-04-25T01:18:27.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.720+0000 7f2e37200640 1 -- 172.21.15.90:0/954112972 shutdown_connections 2024-04-25T01:18:27.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.720+0000 7f2e37200640 1 --2- 172.21.15.90:0/954112972 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e380d0240 0x7f2e380b2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:27.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.720+0000 7f2e37200640 1 --2- 172.21.15.90:0/954112972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e3815ddd0 0x7f2e38070b60 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:27.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.720+0000 7f2e37200640 1 -- 172.21.15.90:0/954112972 >> 172.21.15.90:0/954112972 conn(0x7f2e3806b320 msgr2=0x7f2e3806b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:27.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.720+0000 7f2e37200640 1 -- 172.21.15.90:0/954112972 shutdown_connections 2024-04-25T01:18:27.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.720+0000 7f2e37200640 1 -- 172.21.15.90:0/954112972 wait complete. 2024-04-25T01:18:27.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.721+0000 7f2e37200640 1 Processor -- start 2024-04-25T01:18:27.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.721+0000 7f2e37200640 1 -- start start 2024-04-25T01:18:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.722+0000 7f2e37200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e38072920 0x7f2e381453e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.722+0000 7f2e37200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e380d0240 0x7f2e3815fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.722+0000 7f2e37200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e3815ddd0 0x7f2e38155740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.722+0000 7f2e37200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2e380b3140 con 0x7f2e3815ddd0 2024-04-25T01:18:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.722+0000 7f2e37200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2e380710a0 con 0x7f2e38072920 2024-04-25T01:18:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.722+0000 7f2e37200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2e380b32c0 con 0x7f2e380d0240 2024-04-25T01:18:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.723+0000 7f2e36800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e3815ddd0 0x7f2e38155740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.723+0000 7f2e36800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e3815ddd0 0x7f2e38155740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47844/0 (socket says 172.21.15.90:47844) 2024-04-25T01:18:27.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.723+0000 7f2e36800640 1 -- 172.21.15.90:0/182297274 learned_addr learned my addr 172.21.15.90:0/182297274 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:27.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.723+0000 7f2e36800640 1 -- 172.21.15.90:0/182297274 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e380d0240 msgr2=0x7f2e3815fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:27.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.723+0000 7f2e36800640 1 --2- 172.21.15.90:0/182297274 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e380d0240 0x7f2e3815fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:27.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.723+0000 7f2e36800640 1 -- 172.21.15.90:0/182297274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e38072920 msgr2=0x7f2e381453e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:27.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.723+0000 7f2e36800640 1 --2- 172.21.15.90:0/182297274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e38072920 0x7f2e381453e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:27.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.723+0000 7f2e36800640 1 -- 172.21.15.90:0/182297274 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e38160780 con 0x7f2e3815ddd0 2024-04-25T01:18:27.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.723+0000 7f2e36800640 1 --2- 172.21.15.90:0/182297274 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e3815ddd0 0x7f2e38155740 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f2e3000cf20 tx=0x7f2e3000a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:27.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.724+0000 7f2e2ea00640 1 -- 172.21.15.90:0/182297274 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3001a070 con 0x7f2e3815ddd0 2024-04-25T01:18:27.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.724+0000 7f2e2ea00640 1 -- 172.21.15.90:0/182297274 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e3000d840 con 0x7f2e3815ddd0 2024-04-25T01:18:27.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.724+0000 7f2e2ea00640 1 -- 172.21.15.90:0/182297274 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3000dc30 con 0x7f2e3815ddd0 2024-04-25T01:18:27.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.724+0000 7f2e37200640 1 -- 172.21.15.90:0/182297274 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e38160a60 con 0x7f2e3815ddd0 2024-04-25T01:18:27.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.724+0000 7f2e37200640 1 -- 172.21.15.90:0/182297274 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e38160300 con 0x7f2e3815ddd0 2024-04-25T01:18:27.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.725+0000 7f2e37200640 1 -- 172.21.15.90:0/182297274 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e3804f500 con 0x7f2e3815ddd0 2024-04-25T01:18:27.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.726+0000 7f2e2ea00640 1 -- 172.21.15.90:0/182297274 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2e3001d070 con 0x7f2e3815ddd0 2024-04-25T01:18:27.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.727+0000 7f2e2ea00640 1 --2- 172.21.15.90:0/182297274 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2e1406be30 0x7f2e1406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:27.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.727+0000 7f2e2ea00640 1 -- 172.21.15.90:0/182297274 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f2e3008bb60 con 0x7f2e3815ddd0 2024-04-25T01:18:27.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.727+0000 7f2e35e00640 1 --2- 172.21.15.90:0/182297274 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2e1406be30 0x7f2e1406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:27.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.728+0000 7f2e35e00640 1 --2- 172.21.15.90:0/182297274 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2e1406be30 0x7f2e1406e2f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2e28034000 tx=0x7f2e2802c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:27.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.732+0000 7f2e2ea00640 1 -- 172.21.15.90:0/182297274 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2e3005a090 con 0x7f2e3815ddd0 2024-04-25T01:18:27.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.967+0000 7f2e37200640 1 -- 172.21.15.90:0/182297274 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e380b0f60 con 0x7f2e3815ddd0 2024-04-25T01:18:27.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.969+0000 7f2e2ea00640 1 -- 172.21.15.90:0/182297274 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6485 (secure 0 0 0) 0x7f2e30059990 con 0x7f2e3815ddd0 2024-04-25T01:18:27.953 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:27.953 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1873497946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6833","nonce":1873497946},{"type":"v1","addr":"172.21.15.103:6835","nonce":1873497946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/3206865830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":3206865830},{"type":"v1","addr":"172.21.15.90:6835","nonce":3206865830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5598,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2208594231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2208594231},{"type":"v1","addr":"172.21.15.90:6837","nonce":2208594231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:18:02.701652+0000","modified":"2024-04-25T01:18:15.603132+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2},{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:18:19.371011+0000","modified":"2024-04-25T01:18:27.501490+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5601":{"gid":5601,"name":"d","rank":0,"incarnation":25,"state":"up:replay","state_seq":1,"addr":"172.21.15.103:6834/2850373997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":2850373997},{"type":"v1","addr":"172.21.15.103:6834","nonce":2850373997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":7,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3}]} 2024-04-25T01:18:27.953 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 25 2024-04-25T01:18:27.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.972+0000 7f2e37200640 1 -- 172.21.15.90:0/182297274 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2e1406be30 msgr2=0x7f2e1406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:27.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.972+0000 7f2e37200640 1 --2- 172.21.15.90:0/182297274 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2e1406be30 0x7f2e1406e2f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2e28034000 tx=0x7f2e2802c000 comp rx=0 tx=0).stop 2024-04-25T01:18:27.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.972+0000 7f2e37200640 1 -- 172.21.15.90:0/182297274 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e3815ddd0 msgr2=0x7f2e38155740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:27.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.972+0000 7f2e37200640 1 --2- 172.21.15.90:0/182297274 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e3815ddd0 0x7f2e38155740 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f2e3000cf20 tx=0x7f2e3000a7d0 comp rx=0 tx=0).stop 2024-04-25T01:18:27.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.972+0000 7f2e37200640 1 -- 172.21.15.90:0/182297274 shutdown_connections 2024-04-25T01:18:27.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.972+0000 7f2e37200640 1 --2- 172.21.15.90:0/182297274 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2e1406be30 0x7f2e1406e2f0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:27.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.972+0000 7f2e37200640 1 --2- 172.21.15.90:0/182297274 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e3815ddd0 0x7f2e38155740 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:27.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.972+0000 7f2e37200640 1 --2- 172.21.15.90:0/182297274 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2e380d0240 0x7f2e3815fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:27.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.972+0000 7f2e37200640 1 --2- 172.21.15.90:0/182297274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e38072920 0x7f2e381453e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:27.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.972+0000 7f2e37200640 1 -- 172.21.15.90:0/182297274 >> 172.21.15.90:0/182297274 conn(0x7f2e3806b320 msgr2=0x7f2e3815ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:27.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.972+0000 7f2e37200640 1 -- 172.21.15.90:0/182297274 shutdown_connections 2024-04-25T01:18:27.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:27.972+0000 7f2e37200640 1 -- 172.21.15.90:0/182297274 wait complete. 2024-04-25T01:18:28.379 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T01:18:19.371011+0000', 'modified': '2024-04-25T01:18:27.501490+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5601}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5601': {'gid': 5601, 'name': 'd', 'rank': 0, 'incarnation': 25, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.103:6834/2850373997', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6832', 'nonce': 2850373997}, {'type': 'v1', 'addr': '172.21.15.103:6834', 'nonce': 2850373997}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'recovery_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-25T01:18:28.379 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5601:up:replay 2024-04-25T01:18:29.380 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:18:29.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.608+0000 7f93edc00640 1 Processor -- start 2024-04-25T01:18:29.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.609+0000 7f93edc00640 1 -- start start 2024-04-25T01:18:29.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.611+0000 7f93edc00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f93e80d0240 0x7f93e8168970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:29.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.611+0000 7f93edc00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93e815e1b0 con 0x7f93e8072920 2024-04-25T01:18:29.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.611+0000 7f93edc00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93e80713b0 con 0x7f93e815ddd0 2024-04-25T01:18:29.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.611+0000 7f93edc00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f93e8071690 con 0x7f93e80d0240 2024-04-25T01:18:29.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.611+0000 7f93e7400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f93e8072920 0x7f93e80cfb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43754/0 (socket says 172.21.15.90:43754) 2024-04-25T01:18:29.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.611+0000 7f93e7400640 1 -- 172.21.15.90:0/895071468 learned_addr learned my addr 172.21.15.90:0/895071468 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:29.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.611+0000 7f93e6a00640 1 -- 172.21.15.90:0/895071468 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 460966529 0 0) 0x7f93e815e1b0 con 0x7f93e8072920 2024-04-25T01:18:29.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.612+0000 7f93ed200640 1 --2- 172.21.15.90:0/895071468 >> v2:172.21.15.103:3300/0 conn(0x7f93e80d0240 0x7f93e8168970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:29.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.612+0000 7f93e6a00640 1 -- 172.21.15.90:0/895071468 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93cc003520 con 0x7f93e8072920 2024-04-25T01:18:29.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.612+0000 7f93e6a00640 1 -- 172.21.15.90:0/895071468 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2900450361 0 0) 0x7f93cc003520 con 0x7f93e8072920 2024-04-25T01:18:29.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.612+0000 7f93e6a00640 1 -- 172.21.15.90:0/895071468 >> v2:172.21.15.103:3300/0 conn(0x7f93e80d0240 msgr2=0x7f93e8168970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:29.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.612+0000 7f93e6a00640 1 --2- 172.21.15.90:0/895071468 >> v2:172.21.15.103:3300/0 conn(0x7f93e80d0240 0x7f93e8168970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:29.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.612+0000 7f93e6a00640 1 -- 172.21.15.90:0/895071468 >> v1:172.21.15.103:6789/0 conn(0x7f93e815ddd0 legacy=0x7f93e8070b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:29.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.612+0000 7f93e6a00640 1 -- 172.21.15.90:0/895071468 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93e8071970 con 0x7f93e8072920 2024-04-25T01:18:29.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.612+0000 7f93e6a00640 1 -- 172.21.15.90:0/895071468 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f93d0001690 con 0x7f93e8072920 2024-04-25T01:18:29.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.612+0000 7f93ed200640 1 --2- 172.21.15.90:0/895071468 >> v2:172.21.15.103:3300/0 conn(0x7f93e80d0240 0x7f93e8168970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:18:29.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.612+0000 7f93e6a00640 1 -- 172.21.15.90:0/895071468 >> v1:172.21.15.90:6789/0 conn(0x7f93e8072920 legacy=0x7f93e80cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:29.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.612+0000 7f93e6a00640 1 --2- 172.21.15.90:0/895071468 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93cc0039b0 0x7f93cc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:29.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.612+0000 7f93e6a00640 1 --2- 172.21.15.90:0/895071468 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93cc004800 0x7f93cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:29.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.612+0000 7f93e6a00640 1 --2- 172.21.15.90:0/895071468 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93cc0051c0 0x7f93cc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:29.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.612+0000 7f93e6a00640 1 -- 172.21.15.90:0/895071468 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f93e8071690 con 0x7f93cc0051c0 2024-04-25T01:18:29.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.612+0000 7f93e6a00640 1 -- 172.21.15.90:0/895071468 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f93cc00c110 con 0x7f93cc0039b0 2024-04-25T01:18:29.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.612+0000 7f93e6a00640 1 -- 172.21.15.90:0/895071468 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f93cc00c310 con 0x7f93cc004800 2024-04-25T01:18:29.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.612+0000 7f93ed200640 1 --2- 172.21.15.90:0/895071468 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93cc0051c0 0x7f93cc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:29.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.613+0000 7f93e7e00640 1 --2- 172.21.15.90:0/895071468 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93cc0039b0 0x7f93cc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:29.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.613+0000 7f93e7400640 1 --2- 172.21.15.90:0/895071468 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93cc004800 0x7f93cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:29.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.613+0000 7f93ed200640 1 -- 172.21.15.90:0/895071468 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93cc004800 msgr2=0x7f93cc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:29.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.613+0000 7f93ed200640 1 --2- 172.21.15.90:0/895071468 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93cc004800 0x7f93cc004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:29.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.613+0000 7f93ed200640 1 -- 172.21.15.90:0/895071468 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93cc0039b0 msgr2=0x7f93cc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:29.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.613+0000 7f93ed200640 1 --2- 172.21.15.90:0/895071468 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93cc0039b0 0x7f93cc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:29.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.613+0000 7f93ed200640 1 -- 172.21.15.90:0/895071468 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93cc003520 con 0x7f93cc0051c0 2024-04-25T01:18:29.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.614+0000 7f93ed200640 1 -- 172.21.15.90:0/895071468 reap_dead start 2024-04-25T01:18:29.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.614+0000 7f93ed200640 1 --2- 172.21.15.90:0/895071468 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93cc0051c0 0x7f93cc00b9c0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f93e00073b0 tx=0x7f93cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fd2c7012ec7132 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:29.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.614+0000 7f93e6a00640 1 -- 172.21.15.90:0/895071468 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e0009c80 con 0x7f93cc0051c0 2024-04-25T01:18:29.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.614+0000 7f93e6a00640 1 -- 172.21.15.90:0/895071468 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93e0003b90 con 0x7f93cc0051c0 2024-04-25T01:18:29.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.614+0000 7f93e6a00640 1 -- 172.21.15.90:0/895071468 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e0004020 con 0x7f93cc0051c0 2024-04-25T01:18:29.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.615+0000 7f93edc00640 1 -- 172.21.15.90:0/895071468 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93cc0051c0 msgr2=0x7f93cc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:29.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.615+0000 7f93edc00640 1 --2- 172.21.15.90:0/895071468 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93cc0051c0 0x7f93cc00b9c0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f93e00073b0 tx=0x7f93cc005830 comp rx=0 tx=0).stop 2024-04-25T01:18:29.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.615+0000 7f93edc00640 1 -- 172.21.15.90:0/895071468 shutdown_connections 2024-04-25T01:18:29.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.615+0000 7f93edc00640 1 --2- 172.21.15.90:0/895071468 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93cc0051c0 0x7f93cc00b9c0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:29.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.615+0000 7f93edc00640 1 -- 172.21.15.90:0/895071468 >> 172.21.15.90:0/895071468 conn(0x7f93e806b320 msgr2=0x7f93e806b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:29.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.615+0000 7f93edc00640 1 -- 172.21.15.90:0/895071468 shutdown_connections 2024-04-25T01:18:29.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.615+0000 7f93edc00640 1 -- 172.21.15.90:0/895071468 wait complete. 2024-04-25T01:18:29.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.617+0000 7f93edc00640 1 Processor -- start 2024-04-25T01:18:29.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.617+0000 7f93edc00640 1 -- start start 2024-04-25T01:18:29.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.618+0000 7f93edc00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93cc0051c0 0x7f93e80ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:29.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.618+0000 7f93edc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93e8144cd0 0x7f93e8156ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:29.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.618+0000 7f93edc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93e80b0900 0x7f93e80b0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:29.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.618+0000 7f93edc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f93e80cfcb0 con 0x7f93e8144cd0 2024-04-25T01:18:29.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.618+0000 7f93edc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f93e80cfb30 con 0x7f93e80b0900 2024-04-25T01:18:29.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.618+0000 7f93edc00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f93e80cfe30 con 0x7f93cc0051c0 2024-04-25T01:18:29.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.618+0000 7f93e7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93e8144cd0 0x7f93e8156ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:29.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.618+0000 7f93e7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93e8144cd0 0x7f93e8156ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47872/0 (socket says 172.21.15.90:47872) 2024-04-25T01:18:29.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.618+0000 7f93e7400640 1 -- 172.21.15.90:0/2550555089 learned_addr learned my addr 172.21.15.90:0/2550555089 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:29.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.618+0000 7f93e7e00640 1 --2- 172.21.15.90:0/2550555089 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93cc0051c0 0x7f93e80ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:29.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.618+0000 7f93ed200640 1 --2- 172.21.15.90:0/2550555089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93e80b0900 0x7f93e80b0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:29.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.618+0000 7f93e7400640 1 -- 172.21.15.90:0/2550555089 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93cc0051c0 msgr2=0x7f93e80ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:29.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.618+0000 7f93e7400640 1 --2- 172.21.15.90:0/2550555089 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93cc0051c0 0x7f93e80ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:29.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.618+0000 7f93e7400640 1 -- 172.21.15.90:0/2550555089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93e80b0900 msgr2=0x7f93e80b0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:29.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.618+0000 7f93e7400640 1 --2- 172.21.15.90:0/2550555089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93e80b0900 0x7f93e80b0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:29.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.618+0000 7f93e7400640 1 -- 172.21.15.90:0/2550555089 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93e81584b0 con 0x7f93e8144cd0 2024-04-25T01:18:29.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.618+0000 7f93e7e00640 1 --2- 172.21.15.90:0/2550555089 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93cc0051c0 0x7f93e80ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:29.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.619+0000 7f93e7400640 1 --2- 172.21.15.90:0/2550555089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93e8144cd0 0x7f93e8156ac0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f93d000d7d0 tx=0x7f93d000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:29.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.619+0000 7f93dfe00640 1 -- 172.21.15.90:0/2550555089 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d0019070 con 0x7f93e8144cd0 2024-04-25T01:18:29.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.619+0000 7f93dfe00640 1 -- 172.21.15.90:0/2550555089 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93d00074e0 con 0x7f93e8144cd0 2024-04-25T01:18:29.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.619+0000 7f93dfe00640 1 -- 172.21.15.90:0/2550555089 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d0004a00 con 0x7f93e8144cd0 2024-04-25T01:18:29.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.619+0000 7f93edc00640 1 -- 172.21.15.90:0/2550555089 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93e8157410 con 0x7f93e8144cd0 2024-04-25T01:18:29.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.619+0000 7f93edc00640 1 -- 172.21.15.90:0/2550555089 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93e81579d0 con 0x7f93e8144cd0 2024-04-25T01:18:29.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.620+0000 7f93edc00640 1 -- 172.21.15.90:0/2550555089 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93e80b12c0 con 0x7f93e8144cd0 2024-04-25T01:18:29.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.621+0000 7f93dfe00640 1 -- 172.21.15.90:0/2550555089 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f93d0004ba0 con 0x7f93e8144cd0 2024-04-25T01:18:29.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.622+0000 7f93dfe00640 1 --2- 172.21.15.90:0/2550555089 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f93c406be30 0x7f93c406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:29.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.622+0000 7f93dfe00640 1 -- 172.21.15.90:0/2550555089 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f93d008acc0 con 0x7f93e8144cd0 2024-04-25T01:18:29.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.622+0000 7f93e7e00640 1 --2- 172.21.15.90:0/2550555089 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f93c406be30 0x7f93c406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:29.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.623+0000 7f93e7e00640 1 --2- 172.21.15.90:0/2550555089 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f93c406be30 0x7f93c406e2f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f93d80015b0 tx=0x7f93d8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:29.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.626+0000 7f93dfe00640 1 -- 172.21.15.90:0/2550555089 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f93d0091080 con 0x7f93e8144cd0 2024-04-25T01:18:29.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.863+0000 7f93edc00640 1 -- 172.21.15.90:0/2550555089 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f93e80af8a0 con 0x7f93e8144cd0 2024-04-25T01:18:29.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.864+0000 7f93dfe00640 1 -- 172.21.15.90:0/2550555089 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6485 (secure 0 0 0) 0x7f93d0059160 con 0x7f93e8144cd0 2024-04-25T01:18:29.848 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:29.848 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1873497946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6833","nonce":1873497946},{"type":"v1","addr":"172.21.15.103:6835","nonce":1873497946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/3206865830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":3206865830},{"type":"v1","addr":"172.21.15.90:6835","nonce":3206865830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5598,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2208594231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2208594231},{"type":"v1","addr":"172.21.15.90:6837","nonce":2208594231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:18:02.701652+0000","modified":"2024-04-25T01:18:15.603132+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2},{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:18:19.371011+0000","modified":"2024-04-25T01:18:28.522277+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5601":{"gid":5601,"name":"d","rank":0,"incarnation":25,"state":"up:rejoin","state_seq":5,"addr":"172.21.15.103:6834/2850373997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":2850373997},{"type":"v1","addr":"172.21.15.103:6834","nonce":2850373997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":7,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3}]} 2024-04-25T01:18:29.849 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-25T01:18:29.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.867+0000 7f93edc00640 1 -- 172.21.15.90:0/2550555089 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f93c406be30 msgr2=0x7f93c406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:29.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.867+0000 7f93edc00640 1 --2- 172.21.15.90:0/2550555089 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f93c406be30 0x7f93c406e2f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f93d80015b0 tx=0x7f93d8009040 comp rx=0 tx=0).stop 2024-04-25T01:18:29.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.867+0000 7f93edc00640 1 -- 172.21.15.90:0/2550555089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93e8144cd0 msgr2=0x7f93e8156ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:29.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.867+0000 7f93edc00640 1 --2- 172.21.15.90:0/2550555089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93e8144cd0 0x7f93e8156ac0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f93d000d7d0 tx=0x7f93d000dc90 comp rx=0 tx=0).stop 2024-04-25T01:18:29.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.867+0000 7f93edc00640 1 -- 172.21.15.90:0/2550555089 shutdown_connections 2024-04-25T01:18:29.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.867+0000 7f93edc00640 1 --2- 172.21.15.90:0/2550555089 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f93c406be30 0x7f93c406e2f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:29.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.867+0000 7f93edc00640 1 --2- 172.21.15.90:0/2550555089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93e80b0900 0x7f93e80b0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:29.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.867+0000 7f93edc00640 1 --2- 172.21.15.90:0/2550555089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93e8144cd0 0x7f93e8156ac0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:29.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.867+0000 7f93edc00640 1 --2- 172.21.15.90:0/2550555089 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f93cc0051c0 0x7f93e80ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:29.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.867+0000 7f93edc00640 1 -- 172.21.15.90:0/2550555089 >> 172.21.15.90:0/2550555089 conn(0x7f93e806b320 msgr2=0x7f93e80b2ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:29.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.867+0000 7f93edc00640 1 -- 172.21.15.90:0/2550555089 shutdown_connections 2024-04-25T01:18:29.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:29.867+0000 7f93edc00640 1 -- 172.21.15.90:0/2550555089 wait complete. 2024-04-25T01:18:30.273 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T01:18:19.371011+0000', 'modified': '2024-04-25T01:18:28.522277+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5601}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5601': {'gid': 5601, 'name': 'd', 'rank': 0, 'incarnation': 25, 'state': 'up:rejoin', 'state_seq': 5, 'addr': '172.21.15.103:6834/2850373997', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6832', 'nonce': 2850373997}, {'type': 'v1', 'addr': '172.21.15.103:6834', 'nonce': 2850373997}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'recovery_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-25T01:18:30.273 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5601:up:rejoin 2024-04-25T01:18:31.275 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:18:31.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.500+0000 7f062ac00640 1 Processor -- start 2024-04-25T01:18:31.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.500+0000 7f062ac00640 1 -- start start 2024-04-25T01:18:31.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.502+0000 7f062ac00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f062415a410 0x7f06240aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:31.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.502+0000 7f062ac00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f062406e120 con 0x7f062415e500 2024-04-25T01:18:31.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.502+0000 7f062ac00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f062406e400 con 0x7f062406dc50 2024-04-25T01:18:31.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.502+0000 7f062ac00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f062406e6e0 con 0x7f062415a410 2024-04-25T01:18:31.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.502+0000 7f062a200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f062415e500 0x7f06240b2f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43756/0 (socket says 172.21.15.90:43756) 2024-04-25T01:18:31.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.502+0000 7f062a200640 1 -- 172.21.15.90:0/1653009659 learned_addr learned my addr 172.21.15.90:0/1653009659 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:31.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.502+0000 7f0629800640 1 --2- 172.21.15.90:0/1653009659 >> v2:172.21.15.103:3300/0 conn(0x7f062415a410 0x7f06240aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:31.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.503+0000 7f0623e00640 1 -- 172.21.15.90:0/1653009659 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 112228691 0 0) 0x7f062406e120 con 0x7f062415e500 2024-04-25T01:18:31.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.503+0000 7f0623e00640 1 -- 172.21.15.90:0/1653009659 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f060c003520 con 0x7f062415e500 2024-04-25T01:18:31.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.503+0000 7f0623e00640 1 -- 172.21.15.90:0/1653009659 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2883493541 0 0) 0x7f060c003520 con 0x7f062415e500 2024-04-25T01:18:31.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.503+0000 7f0623e00640 1 -- 172.21.15.90:0/1653009659 >> v2:172.21.15.103:3300/0 conn(0x7f062415a410 msgr2=0x7f06240aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:31.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.503+0000 7f0623e00640 1 --2- 172.21.15.90:0/1653009659 >> v2:172.21.15.103:3300/0 conn(0x7f062415a410 0x7f06240aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:31.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.503+0000 7f0623e00640 1 -- 172.21.15.90:0/1653009659 >> v1:172.21.15.103:6789/0 conn(0x7f062406dc50 legacy=0x7f062415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:31.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.503+0000 7f0623e00640 1 -- 172.21.15.90:0/1653009659 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06240af130 con 0x7f062415e500 2024-04-25T01:18:31.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.503+0000 7f0623e00640 1 -- 172.21.15.90:0/1653009659 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f061c0019f0 con 0x7f062415e500 2024-04-25T01:18:31.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.503+0000 7f0623e00640 1 -- 172.21.15.90:0/1653009659 >> v1:172.21.15.90:6789/0 conn(0x7f062415e500 legacy=0x7f06240b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:31.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.503+0000 7f0623e00640 1 --2- 172.21.15.90:0/1653009659 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f060c003a00 0x7f060c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:31.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.503+0000 7f0623e00640 1 --2- 172.21.15.90:0/1653009659 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f060c004800 0x7f060c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:31.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.503+0000 7f0623e00640 1 --2- 172.21.15.90:0/1653009659 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f060c0051c0 0x7f060c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:31.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.503+0000 7f0623e00640 1 -- 172.21.15.90:0/1653009659 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f062406e6e0 con 0x7f060c003a00 2024-04-25T01:18:31.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.503+0000 7f0623e00640 1 -- 172.21.15.90:0/1653009659 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f060c00c0c0 con 0x7f060c0051c0 2024-04-25T01:18:31.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.503+0000 7f0623e00640 1 -- 172.21.15.90:0/1653009659 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f060c00c2c0 con 0x7f060c004800 2024-04-25T01:18:31.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.504+0000 7f0629800640 1 --2- 172.21.15.90:0/1653009659 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f060c003a00 0x7f060c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:31.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.504+0000 7f0629800640 1 -- 172.21.15.90:0/1653009659 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f060c004800 msgr2=0x7f060c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:31.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.504+0000 7f0629800640 1 --2- 172.21.15.90:0/1653009659 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f060c004800 0x7f060c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:31.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.504+0000 7f0629800640 1 -- 172.21.15.90:0/1653009659 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f060c0051c0 msgr2=0x7f060c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:31.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.504+0000 7f0629800640 1 --2- 172.21.15.90:0/1653009659 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f060c0051c0 0x7f060c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:31.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.504+0000 7f0629800640 1 -- 172.21.15.90:0/1653009659 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f060c003520 con 0x7f060c003a00 2024-04-25T01:18:31.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.504+0000 7f062a200640 1 -- 172.21.15.90:0/1653009659 reap_dead start 2024-04-25T01:18:31.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.505+0000 7f0629800640 1 --2- 172.21.15.90:0/1653009659 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f060c003a00 0x7f060c003e20 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f0618009ac0 tx=0x7f060c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1ce82f252b7594c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:31.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.505+0000 7f0623e00640 1 -- 172.21.15.90:0/1653009659 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0618036070 con 0x7f060c003a00 2024-04-25T01:18:31.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.505+0000 7f0623e00640 1 -- 172.21.15.90:0/1653009659 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0618031400 con 0x7f060c003a00 2024-04-25T01:18:31.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.505+0000 7f0623e00640 1 -- 172.21.15.90:0/1653009659 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0618009690 con 0x7f060c003a00 2024-04-25T01:18:31.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.506+0000 7f062ac00640 1 -- 172.21.15.90:0/1653009659 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f060c003a00 msgr2=0x7f060c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:31.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.506+0000 7f062ac00640 1 --2- 172.21.15.90:0/1653009659 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f060c003a00 0x7f060c003e20 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f0618009ac0 tx=0x7f060c005830 comp rx=0 tx=0).stop 2024-04-25T01:18:31.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.506+0000 7f062ac00640 1 -- 172.21.15.90:0/1653009659 shutdown_connections 2024-04-25T01:18:31.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.506+0000 7f062ac00640 1 --2- 172.21.15.90:0/1653009659 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f060c003a00 0x7f060c003e20 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:31.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.506+0000 7f062ac00640 1 -- 172.21.15.90:0/1653009659 >> 172.21.15.90:0/1653009659 conn(0x7f062406b320 msgr2=0x7f062406b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:31.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.506+0000 7f062ac00640 1 -- 172.21.15.90:0/1653009659 shutdown_connections 2024-04-25T01:18:31.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.506+0000 7f062ac00640 1 -- 172.21.15.90:0/1653009659 wait complete. 2024-04-25T01:18:31.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.508+0000 7f062ac00640 1 Processor -- start 2024-04-25T01:18:31.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.508+0000 7f062ac00640 1 -- start start 2024-04-25T01:18:31.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.509+0000 7f062ac00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f060c003a00 0x7f06241509e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:31.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.509+0000 7f062ac00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06241588f0 0x7f0624160a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:31.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.509+0000 7f062ac00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06241591a0 0x7f0624160f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:31.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.509+0000 7f062ac00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f06240aef80 con 0x7f060c003a00 2024-04-25T01:18:31.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.509+0000 7f062ac00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f06240aee00 con 0x7f06241588f0 2024-04-25T01:18:31.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.509+0000 7f062ac00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f062415ddf0 con 0x7f06241591a0 2024-04-25T01:18:31.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.509+0000 7f0629800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f060c003a00 0x7f06241509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:31.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.509+0000 7f0629800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f060c003a00 0x7f06241509e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47882/0 (socket says 172.21.15.90:47882) 2024-04-25T01:18:31.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.509+0000 7f0629800640 1 -- 172.21.15.90:0/3535009722 learned_addr learned my addr 172.21.15.90:0/3535009722 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:31.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.510+0000 7f0629800640 1 -- 172.21.15.90:0/3535009722 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06241591a0 msgr2=0x7f0624160f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:31.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.510+0000 7f0629800640 1 --2- 172.21.15.90:0/3535009722 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06241591a0 0x7f0624160f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:31.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.510+0000 7f0629800640 1 -- 172.21.15.90:0/3535009722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06241588f0 msgr2=0x7f0624160a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:31.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.510+0000 7f0629800640 1 --2- 172.21.15.90:0/3535009722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06241588f0 0x7f0624160a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:31.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.510+0000 7f0629800640 1 -- 172.21.15.90:0/3535009722 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0624158d80 con 0x7f060c003a00 2024-04-25T01:18:31.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.510+0000 7f0629800640 1 --2- 172.21.15.90:0/3535009722 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f060c003a00 0x7f06241509e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f0618034040 tx=0x7f0618004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:31.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.510+0000 7f0621600640 1 -- 172.21.15.90:0/3535009722 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f061803f020 con 0x7f060c003a00 2024-04-25T01:18:31.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.510+0000 7f0621600640 1 -- 172.21.15.90:0/3535009722 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0618037040 con 0x7f060c003a00 2024-04-25T01:18:31.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.510+0000 7f0621600640 1 -- 172.21.15.90:0/3535009722 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0618036070 con 0x7f060c003a00 2024-04-25T01:18:31.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.510+0000 7f062ac00640 1 -- 172.21.15.90:0/3535009722 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0624156d40 con 0x7f060c003a00 2024-04-25T01:18:31.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.510+0000 7f062ac00640 1 -- 172.21.15.90:0/3535009722 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0624157300 con 0x7f060c003a00 2024-04-25T01:18:31.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.512+0000 7f062ac00640 1 -- 172.21.15.90:0/3535009722 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f062415db00 con 0x7f060c003a00 2024-04-25T01:18:31.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.513+0000 7f0621600640 1 -- 172.21.15.90:0/3535009722 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0618042020 con 0x7f060c003a00 2024-04-25T01:18:31.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.514+0000 7f0621600640 1 --2- 172.21.15.90:0/3535009722 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f060406be60 0x7f060406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:31.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.514+0000 7f0621600640 1 -- 172.21.15.90:0/3535009722 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f06180ab2c0 con 0x7f060c003a00 2024-04-25T01:18:31.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.514+0000 7f0628e00640 1 --2- 172.21.15.90:0/3535009722 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f060406be60 0x7f060406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:31.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.515+0000 7f0628e00640 1 --2- 172.21.15.90:0/3535009722 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f060406be60 0x7f060406e320 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f06240b0240 tx=0x7f0610007990 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:31.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.519+0000 7f0621600640 1 -- 172.21.15.90:0/3535009722 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f062415db00 con 0x7f060c003a00 2024-04-25T01:18:31.610 INFO:tasks.ceph.mds.d.smithi103.stderr:2024-04-25T01:18:31.626+0000 7f4fa3600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T01:18:31.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.759+0000 7f062ac00640 1 -- 172.21.15.90:0/3535009722 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f062404f500 con 0x7f060c003a00 2024-04-25T01:18:31.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.760+0000 7f0621600640 1 -- 172.21.15.90:0/3535009722 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6492 (secure 0 0 0) 0x7f0618049030 con 0x7f060c003a00 2024-04-25T01:18:31.749 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:31.749 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1873497946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6833","nonce":1873497946},{"type":"v1","addr":"172.21.15.103:6835","nonce":1873497946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/3206865830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":3206865830},{"type":"v1","addr":"172.21.15.90:6835","nonce":3206865830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5598,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2208594231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2208594231},{"type":"v1","addr":"172.21.15.90:6837","nonce":2208594231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:18:02.701652+0000","modified":"2024-04-25T01:18:15.603132+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2},{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:18:19.371011+0000","modified":"2024-04-25T01:18:30.533899+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5601":{"gid":5601,"name":"d","rank":0,"incarnation":25,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6834/2850373997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":2850373997},{"type":"v1","addr":"172.21.15.103:6834","nonce":2850373997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":7,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5601,"qdb_cluster":[5601]},"id":3}]} 2024-04-25T01:18:31.750 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 28 2024-04-25T01:18:31.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.762+0000 7f062ac00640 1 -- 172.21.15.90:0/3535009722 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f060406be60 msgr2=0x7f060406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:31.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.762+0000 7f062ac00640 1 --2- 172.21.15.90:0/3535009722 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f060406be60 0x7f060406e320 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f06240b0240 tx=0x7f0610007990 comp rx=0 tx=0).stop 2024-04-25T01:18:31.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.763+0000 7f062ac00640 1 -- 172.21.15.90:0/3535009722 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f060c003a00 msgr2=0x7f06241509e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:31.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.763+0000 7f062ac00640 1 --2- 172.21.15.90:0/3535009722 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f060c003a00 0x7f06241509e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f0618034040 tx=0x7f0618004730 comp rx=0 tx=0).stop 2024-04-25T01:18:31.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.763+0000 7f062ac00640 1 -- 172.21.15.90:0/3535009722 shutdown_connections 2024-04-25T01:18:31.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.763+0000 7f062ac00640 1 --2- 172.21.15.90:0/3535009722 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f060406be60 0x7f060406e320 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:31.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.763+0000 7f062ac00640 1 --2- 172.21.15.90:0/3535009722 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f06241591a0 0x7f0624160f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:31.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.763+0000 7f062ac00640 1 --2- 172.21.15.90:0/3535009722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06241588f0 0x7f0624160a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:31.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.763+0000 7f062ac00640 1 --2- 172.21.15.90:0/3535009722 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f060c003a00 0x7f06241509e0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:31.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.763+0000 7f062ac00640 1 -- 172.21.15.90:0/3535009722 >> 172.21.15.90:0/3535009722 conn(0x7f062406b320 msgr2=0x7f06240cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:31.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.763+0000 7f062ac00640 1 -- 172.21.15.90:0/3535009722 shutdown_connections 2024-04-25T01:18:31.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:31.763+0000 7f062ac00640 1 -- 172.21.15.90:0/3535009722 wait complete. 2024-04-25T01:18:32.168 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T01:18:19.371011+0000', 'modified': '2024-04-25T01:18:30.533899+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5601}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5601': {'gid': 5601, 'name': 'd', 'rank': 0, 'incarnation': 25, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.103:6834/2850373997', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6832', 'nonce': 2850373997}, {'type': 'v1', 'addr': '172.21.15.103:6834', 'nonce': 2850373997}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'recovery_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5601, 'qdb_cluster': [5601]} 2024-04-25T01:18:32.168 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T01:18:32.169 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-25T01:18:32.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.401+0000 7f9497e00640 1 Processor -- start 2024-04-25T01:18:32.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.401+0000 7f9497e00640 1 -- start start 2024-04-25T01:18:32.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.402+0000 7f9497e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f94900aee30 0x7f9490072680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:32.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.402+0000 7f9497e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94900b3550 0x7f9490071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:32.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.403+0000 7f9497e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94900af210 con 0x7f9490072060 2024-04-25T01:18:32.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.403+0000 7f9497e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f9490072c60 con 0x7f94900aee30 2024-04-25T01:18:32.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.403+0000 7f9497e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9490071040 con 0x7f94900b3550 2024-04-25T01:18:32.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.403+0000 7f9495400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f94900aee30 0x7f9490072680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:32.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.403+0000 7f9495400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f94900aee30 0x7f9490072680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47896/0 (socket says 172.21.15.90:47896) 2024-04-25T01:18:32.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.403+0000 7f9495400640 1 -- 172.21.15.90:0/3335612652 learned_addr learned my addr 172.21.15.90:0/3335612652 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:32.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.403+0000 7f9494a00640 1 --2- 172.21.15.90:0/3335612652 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94900b3550 0x7f9490071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:32.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.404+0000 7f9495400640 1 -- 172.21.15.90:0/3335612652 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94900b3550 msgr2=0x7f9490071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:32.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.404+0000 7f9495400640 1 --2- 172.21.15.90:0/3335612652 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94900b3550 0x7f9490071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.404+0000 7f9495400640 1 -- 172.21.15.90:0/3335612652 >> v1:172.21.15.90:6789/0 conn(0x7f9490072060 legacy=0x7f9490161cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:32.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.404+0000 7f9495400640 1 -- 172.21.15.90:0/3335612652 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9490071320 con 0x7f94900aee30 2024-04-25T01:18:32.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.405+0000 7f9495400640 1 --2- 172.21.15.90:0/3335612652 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94900aee30 0x7f9490072680 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f9484009870 tx=0x7f948402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=452a10e484cf1d18 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:32.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.405+0000 7f948fe00640 1 -- 172.21.15.90:0/3335612652 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9484037070 con 0x7f94900aee30 2024-04-25T01:18:32.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.405+0000 7f948fe00640 1 -- 172.21.15.90:0/3335612652 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9484032440 con 0x7f94900aee30 2024-04-25T01:18:32.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.405+0000 7f948fe00640 1 -- 172.21.15.90:0/3335612652 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9484032810 con 0x7f94900aee30 2024-04-25T01:18:32.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.406+0000 7f9497e00640 1 -- 172.21.15.90:0/3335612652 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94900aee30 msgr2=0x7f9490072680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:32.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.406+0000 7f9497e00640 1 --2- 172.21.15.90:0/3335612652 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94900aee30 0x7f9490072680 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f9484009870 tx=0x7f948402a1c0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.406+0000 7f9497e00640 1 -- 172.21.15.90:0/3335612652 shutdown_connections 2024-04-25T01:18:32.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.406+0000 7f9497e00640 1 --2- 172.21.15.90:0/3335612652 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f94900b3550 0x7f9490071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.406+0000 7f9497e00640 1 --2- 172.21.15.90:0/3335612652 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94900aee30 0x7f9490072680 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.406+0000 7f9497e00640 1 -- 172.21.15.90:0/3335612652 >> 172.21.15.90:0/3335612652 conn(0x7f949006b140 msgr2=0x7f949006b550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:32.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.406+0000 7f9497e00640 1 -- 172.21.15.90:0/3335612652 shutdown_connections 2024-04-25T01:18:32.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.406+0000 7f9497e00640 1 -- 172.21.15.90:0/3335612652 wait complete. 2024-04-25T01:18:32.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.408+0000 7f9497e00640 1 Processor -- start 2024-04-25T01:18:32.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.408+0000 7f9497e00640 1 -- start start 2024-04-25T01:18:32.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.409+0000 7f9497e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9490072060 0x7f9490157210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:32.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.409+0000 7f9497e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94900aee30 0x7f9490151dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:32.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.409+0000 7f9497e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94900b3550 0x7f9490158f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:32.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.409+0000 7f9497e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f94900b3b10 con 0x7f94900aee30 2024-04-25T01:18:32.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.409+0000 7f9497e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f94900b3990 con 0x7f94900b3550 2024-04-25T01:18:32.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.409+0000 7f9497e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f94901623c0 con 0x7f9490072060 2024-04-25T01:18:32.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.409+0000 7f9494a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94900aee30 0x7f9490151dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:32.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.409+0000 7f9494a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94900aee30 0x7f9490151dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47910/0 (socket says 172.21.15.90:47910) 2024-04-25T01:18:32.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.409+0000 7f9494a00640 1 -- 172.21.15.90:0/2787883484 learned_addr learned my addr 172.21.15.90:0/2787883484 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:32.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.409+0000 7f9494a00640 1 -- 172.21.15.90:0/2787883484 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9490072060 msgr2=0x7f9490157210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:32.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.409+0000 7f9494a00640 1 --2- 172.21.15.90:0/2787883484 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9490072060 0x7f9490157210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.409+0000 7f9494a00640 1 -- 172.21.15.90:0/2787883484 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94900b3550 msgr2=0x7f9490158f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:32.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.409+0000 7f9495e00640 1 --2- 172.21.15.90:0/2787883484 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94900b3550 0x7f9490158f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:32.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.409+0000 7f9494a00640 1 --2- 172.21.15.90:0/2787883484 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94900b3550 0x7f9490158f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.409+0000 7f9494a00640 1 -- 172.21.15.90:0/2787883484 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9490154940 con 0x7f94900aee30 2024-04-25T01:18:32.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.410+0000 7f9494a00640 1 --2- 172.21.15.90:0/2787883484 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94900aee30 0x7f9490151dd0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f948000d800 tx=0x7f948000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:32.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.410+0000 7f948d600640 1 -- 172.21.15.90:0/2787883484 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948000b6c0 con 0x7f94900aee30 2024-04-25T01:18:32.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.410+0000 7f948d600640 1 -- 172.21.15.90:0/2787883484 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9480019070 con 0x7f94900aee30 2024-04-25T01:18:32.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.410+0000 7f9497e00640 1 -- 172.21.15.90:0/2787883484 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f949015beb0 con 0x7f94900aee30 2024-04-25T01:18:32.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.410+0000 7f948d600640 1 -- 172.21.15.90:0/2787883484 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948000ba80 con 0x7f94900aee30 2024-04-25T01:18:32.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.410+0000 7f9497e00640 1 -- 172.21.15.90:0/2787883484 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f949015c470 con 0x7f94900aee30 2024-04-25T01:18:32.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.412+0000 7f948d600640 1 -- 172.21.15.90:0/2787883484 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f948000bc20 con 0x7f94900aee30 2024-04-25T01:18:32.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.413+0000 7f948d600640 1 --2- 172.21.15.90:0/2787883484 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f947406be30 0x7f947406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:32.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.414+0000 7f948d600640 1 -- 172.21.15.90:0/2787883484 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f948008b7b0 con 0x7f94900aee30 2024-04-25T01:18:32.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.414+0000 7f9495400640 1 --2- 172.21.15.90:0/2787883484 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f947406be30 0x7f947406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:32.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.414+0000 7f949ac70740 1 Processor -- start 2024-04-25T01:18:32.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.414+0000 7f949ac70740 1 -- start start 2024-04-25T01:18:32.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.414+0000 7f9495400640 1 --2- 172.21.15.90:0/2787883484 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f947406be30 0x7f947406e2f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9484034000 tx=0x7f948402c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:32.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.415+0000 7f949ac70740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae6148b510 0x55ae6148b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:32.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.415+0000 7f949ac70740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ae614a8db0 0x55ae614a9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:32.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.415+0000 7f949ac70740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ae614611f0 0x55ae614a7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:32.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.415+0000 7f949ac70740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ae61360ed0 con 0x55ae614611f0 2024-04-25T01:18:32.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.415+0000 7f949ac70740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ae613c3df0 con 0x55ae6148b510 2024-04-25T01:18:32.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.415+0000 7f949ac70740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55ae6136c8d0 con 0x55ae614a8db0 2024-04-25T01:18:32.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.415+0000 7f9495e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ae614611f0 0x55ae614a7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:32.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.416+0000 7f9495e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ae614611f0 0x55ae614a7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47918/0 (socket says 172.21.15.90:47918) 2024-04-25T01:18:32.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.416+0000 7f9495e00640 1 -- 172.21.15.90:0/3413636332 learned_addr learned my addr 172.21.15.90:0/3413636332 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:32.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.416+0000 7f9495400640 1 --2- 172.21.15.90:0/3413636332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae6148b510 0x55ae6148b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:32.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.416+0000 7f9494a00640 1 --2- 172.21.15.90:0/3413636332 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ae614a8db0 0x55ae614a9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:32.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.416+0000 7f9495e00640 1 -- 172.21.15.90:0/3413636332 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ae614a8db0 msgr2=0x55ae614a9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:32.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.416+0000 7f9495e00640 1 --2- 172.21.15.90:0/3413636332 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ae614a8db0 0x55ae614a9300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.416+0000 7f9495e00640 1 -- 172.21.15.90:0/3413636332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae6148b510 msgr2=0x55ae6148b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:32.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.416+0000 7f9495e00640 1 --2- 172.21.15.90:0/3413636332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae6148b510 0x55ae6148b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.416+0000 7f9495e00640 1 -- 172.21.15.90:0/3413636332 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae610d82d0 con 0x55ae614611f0 2024-04-25T01:18:32.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.416+0000 7f9495400640 1 --2- 172.21.15.90:0/3413636332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae6148b510 0x55ae6148b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:32.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.416+0000 7f9495e00640 1 --2- 172.21.15.90:0/3413636332 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ae614611f0 0x55ae614a7470 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f9488001c60 tx=0x7f9488002120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2e7a54fa0902bc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:32.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.416+0000 7f947dc00640 1 -- 172.21.15.90:0/3413636332 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9488016020 con 0x55ae614611f0 2024-04-25T01:18:32.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.417+0000 7f947dc00640 1 -- 172.21.15.90:0/3413636332 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f948800dc60 con 0x55ae614611f0 2024-04-25T01:18:32.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.417+0000 7f947dc00640 1 -- 172.21.15.90:0/3413636332 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94880125f0 con 0x55ae614611f0 2024-04-25T01:18:32.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.417+0000 7f949ac70740 1 -- 172.21.15.90:0/3413636332 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ae614611f0 msgr2=0x55ae614a7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:32.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.417+0000 7f949ac70740 1 --2- 172.21.15.90:0/3413636332 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ae614611f0 0x55ae614a7470 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f9488001c60 tx=0x7f9488002120 comp rx=0 tx=0).stop 2024-04-25T01:18:32.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.417+0000 7f949ac70740 1 -- 172.21.15.90:0/3413636332 shutdown_connections 2024-04-25T01:18:32.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.417+0000 7f949ac70740 1 --2- 172.21.15.90:0/3413636332 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ae614611f0 0x55ae614a7470 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.417+0000 7f949ac70740 1 --2- 172.21.15.90:0/3413636332 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ae614a8db0 0x55ae614a9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.417+0000 7f949ac70740 1 --2- 172.21.15.90:0/3413636332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae6148b510 0x55ae6148b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.417+0000 7f949ac70740 1 -- 172.21.15.90:0/3413636332 >> 172.21.15.90:0/3413636332 conn(0x55ae6145db20 msgr2=0x55ae614a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:32.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.417+0000 7f949ac70740 1 -- 172.21.15.90:0/3413636332 shutdown_connections 2024-04-25T01:18:32.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.417+0000 7f949ac70740 1 -- 172.21.15.90:0/3413636332 wait complete. 2024-04-25T01:18:32.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.418+0000 7f949ac70740 1 -- start start 2024-04-25T01:18:32.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.418+0000 7f949ac70740 1 Processor -- start 2024-04-25T01:18:32.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.419+0000 7f949ac70740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ae6148b510 0x55ae6149eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:32.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.419+0000 7f949ac70740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae614a8db0 0x55ae614a98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:32.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.419+0000 7f949ac70740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ae6149e510 0x55ae614a9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:32.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.419+0000 7f949ac70740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ae6137fd20 con 0x55ae6148b510 2024-04-25T01:18:32.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.419+0000 7f949ac70740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ae613bfdb0 con 0x55ae614a8db0 2024-04-25T01:18:32.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.419+0000 7f949ac70740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55ae61461f90 con 0x55ae6149e510 2024-04-25T01:18:32.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.419+0000 7f949ac70740 10 client.0 _ll_register_callbacks cb 0x55ae613baf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:18:32.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.420+0000 7f9495400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ae6148b510 0x55ae6149eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:32.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.420+0000 7f9495400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ae6148b510 0x55ae6149eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47924/0 (socket says 172.21.15.90:47924) 2024-04-25T01:18:32.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.420+0000 7f9495400640 1 -- 172.21.15.90:0/972141887 learned_addr learned my addr 172.21.15.90:0/972141887 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:32.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.420+0000 7f9494a00640 1 --2- 172.21.15.90:0/972141887 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae614a8db0 0x55ae614a98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:32.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.420+0000 7f9495e00640 1 --2- 172.21.15.90:0/972141887 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ae6149e510 0x55ae614a9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:32.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.420+0000 7f9495400640 1 -- 172.21.15.90:0/972141887 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ae6149e510 msgr2=0x55ae614a9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:32.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.420+0000 7f9495400640 1 --2- 172.21.15.90:0/972141887 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ae6149e510 0x55ae614a9de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.420+0000 7f9495400640 1 -- 172.21.15.90:0/972141887 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae614a8db0 msgr2=0x55ae614a98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:32.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.420+0000 7f9495400640 1 --2- 172.21.15.90:0/972141887 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae614a8db0 0x55ae614a98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.420+0000 7f9495400640 1 -- 172.21.15.90:0/972141887 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae611a26b0 con 0x55ae6148b510 2024-04-25T01:18:32.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.420+0000 7f9495400640 1 --2- 172.21.15.90:0/972141887 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ae6148b510 0x55ae6149eff0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f9484009630 tx=0x7f948402c070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d67965301de23c5c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:32.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.420+0000 7f9472000640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T01:18:32.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.421+0000 7f9472000640 1 -- 172.21.15.90:0/972141887 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9484037040 con 0x55ae6148b510 2024-04-25T01:18:32.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.421+0000 7f9472000640 1 -- 172.21.15.90:0/972141887 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f948403d440 con 0x55ae6148b510 2024-04-25T01:18:32.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.421+0000 7f949ac70740 1 -- 172.21.15.90:0/972141887 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f948800dc60 con 0x55ae6148b510 2024-04-25T01:18:32.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.421+0000 7f9472000640 1 -- 172.21.15.90:0/972141887 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f948403d640 con 0x55ae6148b510 2024-04-25T01:18:32.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.421+0000 7f949ac70740 10 client.5862 fetch_fsmap learned FSMap version 28 2024-04-25T01:18:32.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.421+0000 7f949ac70740 1 -- 172.21.15.90:0/972141887 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) -- 0x55ae6149dd10 con 0x55ae6148b510 2024-04-25T01:18:32.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.421+0000 7f9472000640 1 -- 172.21.15.90:0/972141887 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f948800dc60 con 0x55ae6148b510 2024-04-25T01:18:32.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.421+0000 7f9472000640 1 -- 172.21.15.90:0/972141887 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 4139+0+0 (secure 0 0 0) 0x7f948403a030 con 0x55ae6148b510 2024-04-25T01:18:32.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.421+0000 7f949ac70740 10 client.5862 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:18:32.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.421+0000 7f949ac70740 10 client.5862 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-25T01:18:32.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.421+0000 7f949ac70740 20 client.5862 populate_metadata read hostname 'smithi090' 2024-04-25T01:18:32.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.422+0000 7f949ac70740 1 --2- 172.21.15.90:0/972141887 >> [v2:172.21.15.103:6832/2850373997,v1:172.21.15.103:6834/2850373997] conn(0x55ae6148db70 0x55ae614b6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:32.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.422+0000 7f949ac70740 4 client.5862 mds_command: new command op to 5601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:18:32.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.422+0000 7f949ac70740 1 -- 172.21.15.90:0/972141887 --> [v2:172.21.15.103:6832/2850373997,v1:172.21.15.103:6834/2850373997] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae613d1330 con 0x55ae6148db70 2024-04-25T01:18:32.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.422+0000 7f9494a00640 1 --2- 172.21.15.90:0/972141887 >> [v2:172.21.15.103:6832/2850373997,v1:172.21.15.103:6834/2850373997] conn(0x55ae6148db70 0x55ae614b6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:32.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.423+0000 7f9494a00640 1 --2- 172.21.15.90:0/972141887 >> [v2:172.21.15.103:6832/2850373997,v1:172.21.15.103:6834/2850373997] conn(0x55ae6148db70 0x55ae614b6a60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a866f878deec77e8 server_cookie=c976f1f01a331e84 in_seq=0 out_seq=0 2024-04-25T01:18:32.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.423+0000 7f9472000640 10 client.5862 ms_handle_connect on v2:172.21.15.103:6832/2850373997 2024-04-25T01:18:32.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.426+0000 7f9472000640 1 -- 172.21.15.90:0/972141887 <== mds.0 v2:172.21.15.103:6832/2850373997 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ae613d1330 con 0x55ae6148db70 2024-04-25T01:18:32.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.426+0000 7f9472000640 10 client.5862 handle_command_reply: tid=0 2024-04-25T01:18:32.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.426+0000 7f949ac70740 1 client.5862 shutdown 2024-04-25T01:18:32.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.426+0000 7f949ac70740 1 -- 172.21.15.90:0/972141887 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ae6148b510 msgr2=0x55ae6149eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:32.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.426+0000 7f949ac70740 1 --2- 172.21.15.90:0/972141887 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ae6148b510 0x55ae6149eff0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f9484009630 tx=0x7f948402c070 comp rx=0 tx=0).stop 2024-04-25T01:18:32.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.426+0000 7f949ac70740 1 -- 172.21.15.90:0/972141887 shutdown_connections 2024-04-25T01:18:32.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.426+0000 7f949ac70740 1 --2- 172.21.15.90:0/972141887 >> [v2:172.21.15.103:6832/2850373997,v1:172.21.15.103:6834/2850373997] conn(0x55ae6148db70 0x55ae614b6a60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.426+0000 7f949ac70740 1 --2- 172.21.15.90:0/972141887 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ae6149e510 0x55ae614a9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.426+0000 7f949ac70740 1 --2- 172.21.15.90:0/972141887 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae614a8db0 0x55ae614a98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.426+0000 7f949ac70740 1 --2- 172.21.15.90:0/972141887 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ae6148b510 0x55ae6149eff0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.426+0000 7f9472000640 0 client.5862 ms_handle_reset on v2:172.21.15.103:6832/2850373997 2024-04-25T01:18:32.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.426+0000 7f949ac70740 1 -- 172.21.15.90:0/972141887 >> 172.21.15.90:0/972141887 conn(0x55ae614611f0 msgr2=0x55ae6145afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:32.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.426+0000 7f949ac70740 1 -- 172.21.15.90:0/972141887 shutdown_connections 2024-04-25T01:18:32.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.426+0000 7f949ac70740 1 -- 172.21.15.90:0/972141887 wait complete. 2024-04-25T01:18:32.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.427+0000 7f949ac70740 20 client.5862 trim_cache size 0 max 16384 2024-04-25T01:18:32.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.446+0000 7f949ac70740 1 Processor -- start 2024-04-25T01:18:32.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.446+0000 7f949ac70740 1 -- start start 2024-04-25T01:18:32.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.447+0000 7f949ac70740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ae6148b510 0x55ae6147c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:32.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.447+0000 7f949ac70740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ae6149e510 0x55ae6148b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:32.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.447+0000 7f949ac70740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae614a8db0 0x55ae614a6740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:32.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.447+0000 7f949ac70740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9490200e60 con 0x55ae6148b510 2024-04-25T01:18:32.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.447+0000 7f949ac70740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f949015dfb0 con 0x55ae614a8db0 2024-04-25T01:18:32.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.447+0000 7f949ac70740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9490200810 con 0x55ae6149e510 2024-04-25T01:18:32.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.447+0000 7f9495400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ae6148b510 0x55ae6147c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:32.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.447+0000 7f9495400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ae6148b510 0x55ae6147c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47928/0 (socket says 172.21.15.90:47928) 2024-04-25T01:18:32.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.447+0000 7f9495400640 1 -- 172.21.15.90:0/4129725730 learned_addr learned my addr 172.21.15.90:0/4129725730 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:32.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.448+0000 7f9495e00640 1 --2- 172.21.15.90:0/4129725730 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae614a8db0 0x55ae614a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:32.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.448+0000 7f9495400640 1 -- 172.21.15.90:0/4129725730 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ae6149e510 msgr2=0x55ae6148b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:32.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.448+0000 7f9495400640 1 --2- 172.21.15.90:0/4129725730 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ae6149e510 0x55ae6148b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.448+0000 7f9495400640 1 -- 172.21.15.90:0/4129725730 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae614a8db0 msgr2=0x55ae614a6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:32.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.448+0000 7f9495400640 1 --2- 172.21.15.90:0/4129725730 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae614a8db0 0x55ae614a6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.448+0000 7f9495400640 1 -- 172.21.15.90:0/4129725730 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae614a8be0 con 0x55ae6148b510 2024-04-25T01:18:32.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.448+0000 7f9495400640 1 --2- 172.21.15.90:0/4129725730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ae6148b510 0x55ae6147c530 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f9484032c30 tx=0x7f948403c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=493a3dcff8789fd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:32.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.449+0000 7f947d200640 1 -- 172.21.15.90:0/4129725730 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9484037040 con 0x55ae6148b510 2024-04-25T01:18:32.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.449+0000 7f947d200640 1 -- 172.21.15.90:0/4129725730 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9484004030 con 0x55ae6148b510 2024-04-25T01:18:32.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.449+0000 7f947d200640 1 -- 172.21.15.90:0/4129725730 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94840043e0 con 0x55ae6148b510 2024-04-25T01:18:32.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.449+0000 7f949ac70740 1 -- 172.21.15.90:0/4129725730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ae6148b510 msgr2=0x55ae6147c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:32.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.449+0000 7f949ac70740 1 --2- 172.21.15.90:0/4129725730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ae6148b510 0x55ae6147c530 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f9484032c30 tx=0x7f948403c000 comp rx=0 tx=0).stop 2024-04-25T01:18:32.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.449+0000 7f949ac70740 1 -- 172.21.15.90:0/4129725730 shutdown_connections 2024-04-25T01:18:32.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.449+0000 7f949ac70740 1 --2- 172.21.15.90:0/4129725730 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae614a8db0 0x55ae614a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.449+0000 7f949ac70740 1 --2- 172.21.15.90:0/4129725730 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ae6149e510 0x55ae6148b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.449+0000 7f949ac70740 1 --2- 172.21.15.90:0/4129725730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ae6148b510 0x55ae6147c530 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.449+0000 7f949ac70740 1 -- 172.21.15.90:0/4129725730 >> 172.21.15.90:0/4129725730 conn(0x55ae614611f0 msgr2=0x55ae614a5150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:32.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.449+0000 7f949ac70740 1 -- 172.21.15.90:0/4129725730 shutdown_connections 2024-04-25T01:18:32.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.449+0000 7f949ac70740 1 -- 172.21.15.90:0/4129725730 wait complete. 2024-04-25T01:18:32.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.450+0000 7f949ac70740 1 -- start start 2024-04-25T01:18:32.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.450+0000 7f949ac70740 1 Processor -- start 2024-04-25T01:18:32.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.451+0000 7f949ac70740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ae6148b510 0x55ae614b89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:32.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.451+0000 7f949ac70740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ae6149e510 0x55ae614615d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:32.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.451+0000 7f949ac70740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae614a8db0 0x55ae61461b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:32.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.451+0000 7f949ac70740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ae6148d9f0 con 0x55ae6149e510 2024-04-25T01:18:32.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.451+0000 7f949ac70740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ae6148d550 con 0x55ae614a8db0 2024-04-25T01:18:32.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.451+0000 7f949ac70740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x55ae6148df80 con 0x55ae6148b510 2024-04-25T01:18:32.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.451+0000 7f949ac70740 10 client.0 _ll_register_callbacks cb 0x7f9484035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:18:32.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.451+0000 7f9494a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ae6149e510 0x55ae614615d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:32.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.451+0000 7f9494a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ae6149e510 0x55ae614615d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47942/0 (socket says 172.21.15.90:47942) 2024-04-25T01:18:32.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.451+0000 7f9494a00640 1 -- 172.21.15.90:0/1560157472 learned_addr learned my addr 172.21.15.90:0/1560157472 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:32.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.452+0000 7f9494a00640 1 -- 172.21.15.90:0/1560157472 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ae6148b510 msgr2=0x55ae614b89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:32.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.452+0000 7f9494a00640 1 --2- 172.21.15.90:0/1560157472 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ae6148b510 0x55ae614b89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.452+0000 7f9494a00640 1 -- 172.21.15.90:0/1560157472 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae614a8db0 msgr2=0x55ae61461b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:32.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.452+0000 7f9494a00640 1 --2- 172.21.15.90:0/1560157472 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae614a8db0 0x55ae61461b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.452+0000 7f9494a00640 1 -- 172.21.15.90:0/1560157472 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae614a6740 con 0x55ae6149e510 2024-04-25T01:18:32.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.452+0000 7f9494a00640 1 --2- 172.21.15.90:0/1560157472 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ae6149e510 0x55ae614615d0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f9480055a40 tx=0x7f9480055880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d0a75575f12634b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:32.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.452+0000 7f9472000640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T01:18:32.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.452+0000 7f9472000640 1 -- 172.21.15.90:0/1560157472 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9480071070 con 0x55ae6149e510 2024-04-25T01:18:32.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.452+0000 7f9472000640 1 -- 172.21.15.90:0/1560157472 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f9480055c50 con 0x55ae6149e510 2024-04-25T01:18:32.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.452+0000 7f949ac70740 1 -- 172.21.15.90:0/1560157472 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9484004030 con 0x55ae6149e510 2024-04-25T01:18:32.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.452+0000 7f9472000640 1 -- 172.21.15.90:0/1560157472 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f94800621c0 con 0x55ae6149e510 2024-04-25T01:18:32.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.452+0000 7f949ac70740 10 client.5868 fetch_fsmap learned FSMap version 28 2024-04-25T01:18:32.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.452+0000 7f949ac70740 1 -- 172.21.15.90:0/1560157472 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) -- 0x55ae614a6ca0 con 0x55ae6149e510 2024-04-25T01:18:32.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.453+0000 7f9472000640 1 -- 172.21.15.90:0/1560157472 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9484004030 con 0x55ae6149e510 2024-04-25T01:18:32.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.453+0000 7f9472000640 1 -- 172.21.15.90:0/1560157472 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 4139+0+0 (secure 0 0 0) 0x7f9480013070 con 0x55ae6149e510 2024-04-25T01:18:32.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.453+0000 7f949ac70740 10 client.5868 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:18:32.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.453+0000 7f949ac70740 10 client.5868 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-25T01:18:32.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.453+0000 7f949ac70740 20 client.5868 populate_metadata read hostname 'smithi090' 2024-04-25T01:18:32.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.453+0000 7f949ac70740 1 --2- 172.21.15.90:0/1560157472 >> [v2:172.21.15.103:6832/2850373997,v1:172.21.15.103:6834/2850373997] conn(0x55ae6149cb10 0x55ae6145f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:32.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.453+0000 7f949ac70740 4 client.5868 mds_command: new command op to 5601 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:18:32.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.453+0000 7f949ac70740 1 -- 172.21.15.90:0/1560157472 --> [v2:172.21.15.103:6832/2850373997,v1:172.21.15.103:6834/2850373997] -- command(tid 0: {"prefix": "status"}) -- 0x55ae613cb680 con 0x55ae6149cb10 2024-04-25T01:18:32.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.454+0000 7f9495400640 1 --2- 172.21.15.90:0/1560157472 >> [v2:172.21.15.103:6832/2850373997,v1:172.21.15.103:6834/2850373997] conn(0x55ae6149cb10 0x55ae6145f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:32.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.454+0000 7f9495400640 1 --2- 172.21.15.90:0/1560157472 >> [v2:172.21.15.103:6832/2850373997,v1:172.21.15.103:6834/2850373997] conn(0x55ae6149cb10 0x55ae6145f0c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5f9e4bc3a4bfff7 server_cookie=1ec1024027f20958 in_seq=0 out_seq=0 2024-04-25T01:18:32.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.454+0000 7f9472000640 10 client.5868 ms_handle_connect on v2:172.21.15.103:6832/2850373997 2024-04-25T01:18:32.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.455+0000 7f9472000640 1 -- 172.21.15.90:0/1560157472 <== mds.0 v2:172.21.15.103:6832/2850373997 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x55ae613cb680 con 0x55ae6149cb10 2024-04-25T01:18:32.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.455+0000 7f9472000640 10 client.5868 handle_command_reply: tid=0 2024-04-25T01:18:32.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.455+0000 7f949ac70740 1 client.5868 shutdown 2024-04-25T01:18:32.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.455+0000 7f949ac70740 1 -- 172.21.15.90:0/1560157472 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ae6149e510 msgr2=0x55ae614615d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:32.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.455+0000 7f949ac70740 1 --2- 172.21.15.90:0/1560157472 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ae6149e510 0x55ae614615d0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f9480055a40 tx=0x7f9480055880 comp rx=0 tx=0).stop 2024-04-25T01:18:32.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.455+0000 7f949ac70740 1 -- 172.21.15.90:0/1560157472 shutdown_connections 2024-04-25T01:18:32.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.455+0000 7f949ac70740 1 --2- 172.21.15.90:0/1560157472 >> [v2:172.21.15.103:6832/2850373997,v1:172.21.15.103:6834/2850373997] conn(0x55ae6149cb10 0x55ae6145f0c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.455+0000 7f949ac70740 1 --2- 172.21.15.90:0/1560157472 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae614a8db0 0x55ae61461b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.455+0000 7f949ac70740 1 --2- 172.21.15.90:0/1560157472 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ae6149e510 0x55ae614615d0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.455+0000 7f949ac70740 1 --2- 172.21.15.90:0/1560157472 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x55ae6148b510 0x55ae614b89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.455+0000 7f9472000640 0 client.5868 ms_handle_reset on v2:172.21.15.103:6832/2850373997 2024-04-25T01:18:32.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.455+0000 7f949ac70740 1 -- 172.21.15.90:0/1560157472 >> 172.21.15.90:0/1560157472 conn(0x55ae614611f0 msgr2=0x55ae6145a6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:32.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.456+0000 7f949ac70740 1 -- 172.21.15.90:0/1560157472 shutdown_connections 2024-04-25T01:18:32.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.456+0000 7f949ac70740 1 -- 172.21.15.90:0/1560157472 wait complete. 2024-04-25T01:18:32.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.456+0000 7f949ac70740 20 client.5868 trim_cache size 0 max 16384 2024-04-25T01:18:32.443 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T01:18:32.443 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21", 2024-04-25T01:18:32.443 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 0, 2024-04-25T01:18:32.443 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 5601, 2024-04-25T01:18:32.443 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-04-25T01:18:32.443 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-04-25T01:18:32.443 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "recovery_fs", 2024-04-25T01:18:32.443 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 4.9501424629999997, 2024-04-25T01:18:32.444 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 28, 2024-04-25T01:18:32.444 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 50, 2024-04-25T01:18:32.444 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 50, 2024-04-25T01:18:32.444 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 16.235205117 2024-04-25T01:18:32.444 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T01:18:32.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.456+0000 7f9497e00640 1 -- 172.21.15.90:0/2787883484 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f947406be30 msgr2=0x7f947406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:32.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.456+0000 7f9497e00640 1 --2- 172.21.15.90:0/2787883484 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f947406be30 0x7f947406e2f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9484034000 tx=0x7f948402c000 comp rx=0 tx=0).stop 2024-04-25T01:18:32.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.456+0000 7f9497e00640 1 -- 172.21.15.90:0/2787883484 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94900aee30 msgr2=0x7f9490151dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:32.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.456+0000 7f9497e00640 1 --2- 172.21.15.90:0/2787883484 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94900aee30 0x7f9490151dd0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f948000d800 tx=0x7f948000dcc0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.456+0000 7f9497e00640 1 -- 172.21.15.90:0/2787883484 shutdown_connections 2024-04-25T01:18:32.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.456+0000 7f9497e00640 1 --2- 172.21.15.90:0/2787883484 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f947406be30 0x7f947406e2f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.456+0000 7f9497e00640 1 --2- 172.21.15.90:0/2787883484 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94900b3550 0x7f9490158f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.456+0000 7f9497e00640 1 --2- 172.21.15.90:0/2787883484 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94900aee30 0x7f9490151dd0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.456+0000 7f9497e00640 1 --2- 172.21.15.90:0/2787883484 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9490072060 0x7f9490157210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.456+0000 7f9497e00640 1 -- 172.21.15.90:0/2787883484 >> 172.21.15.90:0/2787883484 conn(0x7f949006b140 msgr2=0x7f949015a310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:32.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.457+0000 7f9497e00640 1 -- 172.21.15.90:0/2787883484 shutdown_connections 2024-04-25T01:18:32.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.457+0000 7f9497e00640 1 -- 172.21.15.90:0/2787883484 wait complete. 2024-04-25T01:18:32.458 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds debug_mds 20 2024-04-25T01:18:32.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.681+0000 7f68bf200640 1 Processor -- start 2024-04-25T01:18:32.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.681+0000 7f68bf200640 1 -- start start 2024-04-25T01:18:32.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.683+0000 7f68bf200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f68c015c840 0x7f68c00b4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:32.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.683+0000 7f68bf200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68c01627f0 0x7f68c01664a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:32.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.683+0000 7f68bf200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f68c01669e0 0x7f68c0070e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:32.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.683+0000 7f68bf200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f68c0166010 con 0x7f68c01669e0 2024-04-25T01:18:32.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.683+0000 7f68bf200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f68c006e3f0 con 0x7f68c015c840 2024-04-25T01:18:32.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.683+0000 7f68bf200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f68c015cc20 con 0x7f68c01627f0 2024-04-25T01:18:32.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.684+0000 7f68be800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f68c01669e0 0x7f68c0070e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:32.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.684+0000 7f68be800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f68c01669e0 0x7f68c0070e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47954/0 (socket says 172.21.15.90:47954) 2024-04-25T01:18:32.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.684+0000 7f68be800640 1 -- 172.21.15.90:0/1036576389 learned_addr learned my addr 172.21.15.90:0/1036576389 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:32.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.684+0000 7f68bd400640 1 --2- 172.21.15.90:0/1036576389 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68c01627f0 0x7f68c01664a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:32.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.684+0000 7f68be800640 1 -- 172.21.15.90:0/1036576389 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68c01627f0 msgr2=0x7f68c01664a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:32.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.684+0000 7f68be800640 1 --2- 172.21.15.90:0/1036576389 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68c01627f0 0x7f68c01664a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.684+0000 7f68be800640 1 -- 172.21.15.90:0/1036576389 >> v2:172.21.15.103:3300/0 conn(0x7f68c015c840 msgr2=0x7f68c00b4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:32.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.684+0000 7f68be800640 1 --2- 172.21.15.90:0/1036576389 >> v2:172.21.15.103:3300/0 conn(0x7f68c015c840 0x7f68c00b4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.684+0000 7f68be800640 1 -- 172.21.15.90:0/1036576389 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68c0162f60 con 0x7f68c01669e0 2024-04-25T01:18:32.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.685+0000 7f68be800640 1 --2- 172.21.15.90:0/1036576389 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68c01669e0 0x7f68c0070e60 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f68c006dc80 tx=0x7f68b802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51dcc266a14bffb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:32.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.686+0000 7f68bca00640 1 -- 172.21.15.90:0/1036576389 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b8036070 con 0x7f68c01669e0 2024-04-25T01:18:32.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.686+0000 7f68bca00640 1 -- 172.21.15.90:0/1036576389 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68b8031600 con 0x7f68c01669e0 2024-04-25T01:18:32.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.686+0000 7f68bca00640 1 -- 172.21.15.90:0/1036576389 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b80319d0 con 0x7f68c01669e0 2024-04-25T01:18:32.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.686+0000 7f68bf200640 1 -- 172.21.15.90:0/1036576389 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68c01669e0 msgr2=0x7f68c0070e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:32.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.686+0000 7f68bf200640 1 --2- 172.21.15.90:0/1036576389 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68c01669e0 0x7f68c0070e60 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f68c006dc80 tx=0x7f68b802a6a0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.687+0000 7f68bf200640 1 -- 172.21.15.90:0/1036576389 shutdown_connections 2024-04-25T01:18:32.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.687+0000 7f68bf200640 1 --2- 172.21.15.90:0/1036576389 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68c01627f0 0x7f68c01664a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.687+0000 7f68bf200640 1 --2- 172.21.15.90:0/1036576389 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68c01669e0 0x7f68c0070e60 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.687+0000 7f68bf200640 1 --2- 172.21.15.90:0/1036576389 >> v2:172.21.15.103:3300/0 conn(0x7f68c015c840 0x7f68c00b4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.687+0000 7f68bf200640 1 -- 172.21.15.90:0/1036576389 >> 172.21.15.90:0/1036576389 conn(0x7f68c006b560 msgr2=0x7f68c006b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:32.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.687+0000 7f68bf200640 1 -- 172.21.15.90:0/1036576389 shutdown_connections 2024-04-25T01:18:32.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.687+0000 7f68bf200640 1 -- 172.21.15.90:0/1036576389 wait complete. 2024-04-25T01:18:32.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.688+0000 7f68bf200640 1 Processor -- start 2024-04-25T01:18:32.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.689+0000 7f68bf200640 1 -- start start 2024-04-25T01:18:32.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.689+0000 7f68bf200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68c015c840 0x7f68c01591f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:32.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.690+0000 7f68bf200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68c01627f0 0x7f68c0143a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:32.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.690+0000 7f68bf200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68c01669e0 0x7f68c00b1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:32.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.690+0000 7f68bf200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f68c0162c70 con 0x7f68c015c840 2024-04-25T01:18:32.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.690+0000 7f68bf200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f68c0163520 con 0x7f68c01669e0 2024-04-25T01:18:32.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.690+0000 7f68bf200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f68c00713a0 con 0x7f68c01627f0 2024-04-25T01:18:32.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.690+0000 7f68bde00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68c015c840 0x7f68c01591f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:32.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.690+0000 7f68bde00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68c015c840 0x7f68c01591f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47966/0 (socket says 172.21.15.90:47966) 2024-04-25T01:18:32.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.690+0000 7f68bde00640 1 -- 172.21.15.90:0/1266287782 learned_addr learned my addr 172.21.15.90:0/1266287782 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:32.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.690+0000 7f68bd400640 1 --2- 172.21.15.90:0/1266287782 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68c01627f0 0x7f68c0143a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:32.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.690+0000 7f68be800640 1 --2- 172.21.15.90:0/1266287782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68c01669e0 0x7f68c00b1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:32.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.690+0000 7f68bde00640 1 -- 172.21.15.90:0/1266287782 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68c01627f0 msgr2=0x7f68c0143a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:32.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.690+0000 7f68bde00640 1 --2- 172.21.15.90:0/1266287782 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68c01627f0 0x7f68c0143a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.690+0000 7f68bde00640 1 -- 172.21.15.90:0/1266287782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68c01669e0 msgr2=0x7f68c00b1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:32.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.690+0000 7f68bde00640 1 --2- 172.21.15.90:0/1266287782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68c01669e0 0x7f68c00b1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.690+0000 7f68bde00640 1 -- 172.21.15.90:0/1266287782 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68c0143fb0 con 0x7f68c015c840 2024-04-25T01:18:32.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.691+0000 7f68bde00640 1 --2- 172.21.15.90:0/1266287782 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68c015c840 0x7f68c01591f0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f68b000b4a0 tx=0x7f68b000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:32.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.691+0000 7f68b6000640 1 -- 172.21.15.90:0/1266287782 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b0014070 con 0x7f68c015c840 2024-04-25T01:18:32.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.691+0000 7f68b6000640 1 -- 172.21.15.90:0/1266287782 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68b0010610 con 0x7f68c015c840 2024-04-25T01:18:32.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.691+0000 7f68b6000640 1 -- 172.21.15.90:0/1266287782 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b0010a00 con 0x7f68c015c840 2024-04-25T01:18:32.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.691+0000 7f68bf200640 1 -- 172.21.15.90:0/1266287782 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68c00b2500 con 0x7f68c015c840 2024-04-25T01:18:32.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.691+0000 7f68bf200640 1 -- 172.21.15.90:0/1266287782 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68c0158d50 con 0x7f68c015c840 2024-04-25T01:18:32.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.692+0000 7f68bf200640 1 -- 172.21.15.90:0/1266287782 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68c004f500 con 0x7f68c015c840 2024-04-25T01:18:32.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.693+0000 7f68b6000640 1 -- 172.21.15.90:0/1266287782 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f68b0017070 con 0x7f68c015c840 2024-04-25T01:18:32.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.694+0000 7f68b6000640 1 --2- 172.21.15.90:0/1266287782 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f689c06be60 0x7f689c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:32.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.694+0000 7f68b6000640 1 -- 172.21.15.90:0/1266287782 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f68b008ad40 con 0x7f68c015c840 2024-04-25T01:18:32.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.694+0000 7f68bd400640 1 --2- 172.21.15.90:0/1266287782 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f689c06be60 0x7f689c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:32.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.695+0000 7f68bd400640 1 --2- 172.21.15.90:0/1266287782 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f689c06be60 0x7f689c06e320 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f68c00710b0 tx=0x7f68a8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:32.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.699+0000 7f68b6000640 1 -- 172.21.15.90:0/1266287782 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f68c0143fb0 con 0x7f68c015c840 2024-04-25T01:18:32.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.897+0000 7f68bf200640 1 -- 172.21.15.90:0/1266287782 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7f68c00b4240 con 0x7f68c015c840 2024-04-25T01:18:32.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.904+0000 7f68b6000640 1 -- 172.21.15.90:0/1266287782 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v11)=0 v11) ==== 108+0+0 (secure 0 0 0) 0x7f68b0059270 con 0x7f68c015c840 2024-04-25T01:18:32.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.907+0000 7f68bf200640 1 -- 172.21.15.90:0/1266287782 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f689c06be60 msgr2=0x7f689c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:32.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.907+0000 7f68bf200640 1 --2- 172.21.15.90:0/1266287782 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f689c06be60 0x7f689c06e320 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f68c00710b0 tx=0x7f68a8008040 comp rx=0 tx=0).stop 2024-04-25T01:18:32.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.907+0000 7f68bf200640 1 -- 172.21.15.90:0/1266287782 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68c015c840 msgr2=0x7f68c01591f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:32.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.907+0000 7f68bf200640 1 --2- 172.21.15.90:0/1266287782 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68c015c840 0x7f68c01591f0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f68b000b4a0 tx=0x7f68b000b960 comp rx=0 tx=0).stop 2024-04-25T01:18:32.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.907+0000 7f68bf200640 1 -- 172.21.15.90:0/1266287782 shutdown_connections 2024-04-25T01:18:32.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.907+0000 7f68bf200640 1 --2- 172.21.15.90:0/1266287782 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f689c06be60 0x7f689c06e320 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.907+0000 7f68bf200640 1 --2- 172.21.15.90:0/1266287782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68c01669e0 0x7f68c00b1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.907+0000 7f68bf200640 1 --2- 172.21.15.90:0/1266287782 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f68c01627f0 0x7f68c0143a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.907+0000 7f68bf200640 1 --2- 172.21.15.90:0/1266287782 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68c015c840 0x7f68c01591f0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:32.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.907+0000 7f68bf200640 1 -- 172.21.15.90:0/1266287782 >> 172.21.15.90:0/1266287782 conn(0x7f68c006b560 msgr2=0x7f68c01651e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:32.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.907+0000 7f68bf200640 1 -- 172.21.15.90:0/1266287782 shutdown_connections 2024-04-25T01:18:32.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:32.907+0000 7f68bf200640 1 -- 172.21.15.90:0/1266287782 wait complete. 2024-04-25T01:18:32.910 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub start / force,recursive,repair 2024-04-25T01:18:33.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.130+0000 7f0854a00640 1 Processor -- start 2024-04-25T01:18:33.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.130+0000 7f0854a00640 1 -- start start 2024-04-25T01:18:33.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.132+0000 7f0854a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08500cc200 0x7f085006db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.132+0000 7f0854a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f085006e230 0x7f085006e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.132+0000 7f0854a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0850160420 con 0x7f0850071f10 2024-04-25T01:18:33.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.132+0000 7f0854a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f085015d2a0 con 0x7f085006e230 2024-04-25T01:18:33.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.132+0000 7f0854a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f085006e070 con 0x7f08500cc200 2024-04-25T01:18:33.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.132+0000 7f084f200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f085006e230 0x7f085006e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:33.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.133+0000 7f084f200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f085006e230 0x7f085006e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47976/0 (socket says 172.21.15.90:47976) 2024-04-25T01:18:33.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.133+0000 7f084e800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f0850071f10 0x7f085015cb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43762/0 (socket says 172.21.15.90:43762) 2024-04-25T01:18:33.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.133+0000 7f084f200640 1 -- 172.21.15.90:0/2432309479 learned_addr learned my addr 172.21.15.90:0/2432309479 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:33.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.133+0000 7f0847e00640 1 --2- 172.21.15.90:0/2432309479 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08500cc200 0x7f085006db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:33.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.133+0000 7f084f200640 1 -- 172.21.15.90:0/2432309479 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08500cc200 msgr2=0x7f085006db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:33.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.133+0000 7f084f200640 1 --2- 172.21.15.90:0/2432309479 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08500cc200 0x7f085006db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.133+0000 7f084f200640 1 -- 172.21.15.90:0/2432309479 >> v1:172.21.15.90:6789/0 conn(0x7f0850071f10 legacy=0x7f085015cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:33.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.133+0000 7f084f200640 1 -- 172.21.15.90:0/2432309479 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f085015c390 con 0x7f085006e230 2024-04-25T01:18:33.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.133+0000 7f0847e00640 1 --2- 172.21.15.90:0/2432309479 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08500cc200 0x7f085006db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:33.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.134+0000 7f084f200640 1 --2- 172.21.15.90:0/2432309479 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f085006e230 0x7f085006e680 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f0848009e50 tx=0x7f084802a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eb145c215f3200c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:33.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.134+0000 7f084de00640 1 -- 172.21.15.90:0/2432309479 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0848036070 con 0x7f085006e230 2024-04-25T01:18:33.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.134+0000 7f084de00640 1 -- 172.21.15.90:0/2432309479 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0848031600 con 0x7f085006e230 2024-04-25T01:18:33.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.134+0000 7f084de00640 1 -- 172.21.15.90:0/2432309479 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08480319d0 con 0x7f085006e230 2024-04-25T01:18:33.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.135+0000 7f0854a00640 1 -- 172.21.15.90:0/2432309479 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f085006e230 msgr2=0x7f085006e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:33.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.135+0000 7f0854a00640 1 --2- 172.21.15.90:0/2432309479 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f085006e230 0x7f085006e680 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f0848009e50 tx=0x7f084802a710 comp rx=0 tx=0).stop 2024-04-25T01:18:33.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.135+0000 7f0854a00640 1 -- 172.21.15.90:0/2432309479 shutdown_connections 2024-04-25T01:18:33.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.135+0000 7f0854a00640 1 --2- 172.21.15.90:0/2432309479 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f085006e230 0x7f085006e680 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.135+0000 7f0854a00640 1 --2- 172.21.15.90:0/2432309479 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f08500cc200 0x7f085006db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.135+0000 7f0854a00640 1 -- 172.21.15.90:0/2432309479 >> 172.21.15.90:0/2432309479 conn(0x7f08500b1460 msgr2=0x7f08500b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:33.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.135+0000 7f0854a00640 1 -- 172.21.15.90:0/2432309479 shutdown_connections 2024-04-25T01:18:33.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.135+0000 7f0854a00640 1 -- 172.21.15.90:0/2432309479 wait complete. 2024-04-25T01:18:33.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.137+0000 7f0854a00640 1 Processor -- start 2024-04-25T01:18:33.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.137+0000 7f0854a00640 1 -- start start 2024-04-25T01:18:33.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.138+0000 7f0854a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f085006e230 0x7f08501e6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.138+0000 7f0854a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0850071f10 0x7f08501f4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.138+0000 7f0854a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08500cc200 0x7f08501482b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.138+0000 7f0854a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f08500722f0 con 0x7f08500cc200 2024-04-25T01:18:33.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.138+0000 7f0854a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f085006ebc0 con 0x7f0850071f10 2024-04-25T01:18:33.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.138+0000 7f0854a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f085015c700 con 0x7f085006e230 2024-04-25T01:18:33.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.138+0000 7f084f200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08500cc200 0x7f08501482b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:33.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.138+0000 7f084f200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08500cc200 0x7f08501482b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47978/0 (socket says 172.21.15.90:47978) 2024-04-25T01:18:33.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.138+0000 7f084f200640 1 -- 172.21.15.90:0/3200962980 learned_addr learned my addr 172.21.15.90:0/3200962980 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:33.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.139+0000 7f084f200640 1 -- 172.21.15.90:0/3200962980 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f085006e230 msgr2=0x7f08501e6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:33.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.139+0000 7f084f200640 1 --2- 172.21.15.90:0/3200962980 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f085006e230 0x7f08501e6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.139+0000 7f084f200640 1 -- 172.21.15.90:0/3200962980 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0850071f10 msgr2=0x7f08501f4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:33.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.139+0000 7f084f200640 1 --2- 172.21.15.90:0/3200962980 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0850071f10 0x7f08501f4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.139+0000 7f084f200640 1 -- 172.21.15.90:0/3200962980 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f085015a3c0 con 0x7f08500cc200 2024-04-25T01:18:33.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.139+0000 7f084f200640 1 --2- 172.21.15.90:0/3200962980 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08500cc200 0x7f08501482b0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f0848034040 tx=0x7f0848004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:33.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.139+0000 7f0846a00640 1 -- 172.21.15.90:0/3200962980 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0848036050 con 0x7f08500cc200 2024-04-25T01:18:33.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.139+0000 7f0846a00640 1 -- 172.21.15.90:0/3200962980 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0848042070 con 0x7f08500cc200 2024-04-25T01:18:33.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.139+0000 7f0846a00640 1 -- 172.21.15.90:0/3200962980 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0848009510 con 0x7f08500cc200 2024-04-25T01:18:33.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.139+0000 7f0854a00640 1 -- 172.21.15.90:0/3200962980 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08501e6670 con 0x7f08500cc200 2024-04-25T01:18:33.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.140+0000 7f0854a00640 1 -- 172.21.15.90:0/3200962980 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f085015a910 con 0x7f08500cc200 2024-04-25T01:18:33.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.142+0000 7f0846a00640 1 -- 172.21.15.90:0/3200962980 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0848007bd0 con 0x7f08500cc200 2024-04-25T01:18:33.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.143+0000 7f0846a00640 1 --2- 172.21.15.90:0/3200962980 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f082c06be30 0x7f082c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.143+0000 7f0846a00640 1 -- 172.21.15.90:0/3200962980 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f08480ac220 con 0x7f08500cc200 2024-04-25T01:18:33.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.144+0000 7f084e800640 1 --2- 172.21.15.90:0/3200962980 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f082c06be30 0x7f082c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:33.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.144+0000 7f0858366740 1 Processor -- start 2024-04-25T01:18:33.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.144+0000 7f084e800640 1 --2- 172.21.15.90:0/3200962980 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f082c06be30 0x7f082c06e2f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0840000f60 tx=0x7f0840007a00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:33.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.144+0000 7f0858366740 1 -- start start 2024-04-25T01:18:33.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.145+0000 7f0858366740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b6a315120 0x561b6a2c7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.145+0000 7f0858366740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6a2f8240 0x561b6a2f75c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.145+0000 7f0858366740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b6a2f7b00 0x561b6a2cc910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.145+0000 7f0858366740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561b6a2301c0 con 0x561b6a315120 2024-04-25T01:18:33.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.145+0000 7f0858366740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561b6a1d8a70 con 0x561b6a2f7b00 2024-04-25T01:18:33.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.145+0000 7f0858366740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x561b6a22c180 con 0x561b6a2f8240 2024-04-25T01:18:33.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.145+0000 7f084e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b6a315120 0x561b6a2c7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:33.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.146+0000 7f084e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b6a315120 0x561b6a2c7ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47984/0 (socket says 172.21.15.90:47984) 2024-04-25T01:18:33.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.146+0000 7f084e800640 1 -- 172.21.15.90:0/757847068 learned_addr learned my addr 172.21.15.90:0/757847068 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:33.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.146+0000 7f084e800640 1 -- 172.21.15.90:0/757847068 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6a2f8240 msgr2=0x561b6a2f75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:33.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.146+0000 7f084e800640 1 --2- 172.21.15.90:0/757847068 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6a2f8240 0x561b6a2f75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.146+0000 7f084e800640 1 -- 172.21.15.90:0/757847068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b6a2f7b00 msgr2=0x561b6a2cc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:33.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.146+0000 7f084e800640 1 --2- 172.21.15.90:0/757847068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b6a2f7b00 0x561b6a2cc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.146+0000 7f084e800640 1 -- 172.21.15.90:0/757847068 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b6a1e01e0 con 0x561b6a315120 2024-04-25T01:18:33.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.146+0000 7f084e800640 1 --2- 172.21.15.90:0/757847068 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b6a315120 0x561b6a2c7ab0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f0840007550 tx=0x7f084000f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e05efe4177a5a89 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:33.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.147+0000 7f083f000640 1 -- 172.21.15.90:0/757847068 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0840007640 con 0x561b6a315120 2024-04-25T01:18:33.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.147+0000 7f083f000640 1 -- 172.21.15.90:0/757847068 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0840005c80 con 0x561b6a315120 2024-04-25T01:18:33.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.147+0000 7f083f000640 1 -- 172.21.15.90:0/757847068 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0840016b90 con 0x561b6a315120 2024-04-25T01:18:33.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.147+0000 7f0858366740 1 -- 172.21.15.90:0/757847068 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b6a315120 msgr2=0x561b6a2c7ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:33.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.147+0000 7f0858366740 1 --2- 172.21.15.90:0/757847068 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b6a315120 0x561b6a2c7ab0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f0840007550 tx=0x7f084000f000 comp rx=0 tx=0).stop 2024-04-25T01:18:33.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.147+0000 7f0858366740 1 -- 172.21.15.90:0/757847068 shutdown_connections 2024-04-25T01:18:33.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.147+0000 7f0858366740 1 --2- 172.21.15.90:0/757847068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b6a2f7b00 0x561b6a2cc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.147+0000 7f0858366740 1 --2- 172.21.15.90:0/757847068 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6a2f8240 0x561b6a2f75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.147+0000 7f0858366740 1 --2- 172.21.15.90:0/757847068 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b6a315120 0x561b6a2c7ab0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.147+0000 7f0858366740 1 -- 172.21.15.90:0/757847068 >> 172.21.15.90:0/757847068 conn(0x561b6a2c9d20 msgr2=0x561b6a3125c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:33.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.147+0000 7f0858366740 1 -- 172.21.15.90:0/757847068 shutdown_connections 2024-04-25T01:18:33.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.148+0000 7f0858366740 1 -- 172.21.15.90:0/757847068 wait complete. 2024-04-25T01:18:33.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.148+0000 7f0858366740 1 -- start start 2024-04-25T01:18:33.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.148+0000 7f0858366740 1 Processor -- start 2024-04-25T01:18:33.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.149+0000 7f0858366740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6a2f8240 0x561b6a2e4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.149+0000 7f0858366740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b6a315120 0x561b6a2cd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.149+0000 7f0858366740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b6a2f8730 0x561b6a2cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.149+0000 7f0858366740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561b6a2cbed0 con 0x561b6a2f8730 2024-04-25T01:18:33.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.149+0000 7f0858366740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561b6a1ebec0 con 0x561b6a315120 2024-04-25T01:18:33.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.149+0000 7f0858366740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x561b6a2cb2c0 con 0x561b6a2f8240 2024-04-25T01:18:33.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.149+0000 7f0858366740 10 client.0 _ll_register_callbacks cb 0x561b6a1f09b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:18:33.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.150+0000 7f084f200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b6a2f8730 0x561b6a2cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:33.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.150+0000 7f084f200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b6a2f8730 0x561b6a2cd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47994/0 (socket says 172.21.15.90:47994) 2024-04-25T01:18:33.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.150+0000 7f084f200640 1 -- 172.21.15.90:0/981548904 learned_addr learned my addr 172.21.15.90:0/981548904 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:33.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.150+0000 7f084f200640 1 -- 172.21.15.90:0/981548904 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6a2f8240 msgr2=0x561b6a2e4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:33.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.150+0000 7f084f200640 1 --2- 172.21.15.90:0/981548904 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6a2f8240 0x561b6a2e4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.150+0000 7f084f200640 1 -- 172.21.15.90:0/981548904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b6a315120 msgr2=0x561b6a2cd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:33.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.150+0000 7f084f200640 1 --2- 172.21.15.90:0/981548904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b6a315120 0x561b6a2cd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.150+0000 7f084f200640 1 -- 172.21.15.90:0/981548904 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b6a010940 con 0x561b6a2f8730 2024-04-25T01:18:33.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.150+0000 7f084f200640 1 --2- 172.21.15.90:0/981548904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b6a2f8730 0x561b6a2cd8f0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f0848009e50 tx=0x7f0848079b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ef336d4a75d9c71 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:33.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.150+0000 7f0833400640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T01:18:33.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.150+0000 7f0833400640 1 -- 172.21.15.90:0/981548904 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0848092020 con 0x561b6a2f8730 2024-04-25T01:18:33.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.150+0000 7f0833400640 1 -- 172.21.15.90:0/981548904 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f0848079530 con 0x561b6a2f8730 2024-04-25T01:18:33.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.150+0000 7f0858366740 1 -- 172.21.15.90:0/981548904 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0840005c80 con 0x561b6a2f8730 2024-04-25T01:18:33.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.151+0000 7f0833400640 1 -- 172.21.15.90:0/981548904 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f08480aa0d0 con 0x561b6a2f8730 2024-04-25T01:18:33.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.151+0000 7f0858366740 10 client.5889 fetch_fsmap learned FSMap version 28 2024-04-25T01:18:33.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.151+0000 7f0858366740 1 -- 172.21.15.90:0/981548904 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) -- 0x561b6a0710d0 con 0x561b6a2f8730 2024-04-25T01:18:33.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.151+0000 7f0833400640 1 -- 172.21.15.90:0/981548904 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0840005c80 con 0x561b6a2f8730 2024-04-25T01:18:33.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.151+0000 7f0833400640 1 -- 172.21.15.90:0/981548904 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 4139+0+0 (secure 0 0 0) 0x7f0848075e30 con 0x561b6a2f8730 2024-04-25T01:18:33.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.151+0000 7f0858366740 10 client.5889 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:18:33.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.151+0000 7f0858366740 10 client.5889 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-25T01:18:33.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.151+0000 7f0858366740 20 client.5889 populate_metadata read hostname 'smithi090' 2024-04-25T01:18:33.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.151+0000 7f0858366740 1 --2- 172.21.15.90:0/981548904 >> [v2:172.21.15.103:6832/2850373997,v1:172.21.15.103:6834/2850373997] conn(0x561b6a309750 0x561b6a309b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.151+0000 7f0858366740 4 client.5889 mds_command: new command op to 5601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:18:33.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.151+0000 7f0858366740 1 -- 172.21.15.90:0/981548904 --> [v2:172.21.15.103:6832/2850373997,v1:172.21.15.103:6834/2850373997] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b6a23d700 con 0x561b6a309750 2024-04-25T01:18:33.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.152+0000 7f084e800640 1 --2- 172.21.15.90:0/981548904 >> [v2:172.21.15.103:6832/2850373997,v1:172.21.15.103:6834/2850373997] conn(0x561b6a309750 0x561b6a309b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:33.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.153+0000 7f084e800640 1 --2- 172.21.15.90:0/981548904 >> [v2:172.21.15.103:6832/2850373997,v1:172.21.15.103:6834/2850373997] conn(0x561b6a309750 0x561b6a309b60 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18954bbb80e93191 server_cookie=92d2783e2460de11 in_seq=0 out_seq=0 2024-04-25T01:18:33.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.153+0000 7f0833400640 10 client.5889 ms_handle_connect on v2:172.21.15.103:6832/2850373997 2024-04-25T01:18:33.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.155+0000 7f0833400640 1 -- 172.21.15.90:0/981548904 <== mds.0 v2:172.21.15.103:6832/2850373997 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561b6a23d700 con 0x561b6a309750 2024-04-25T01:18:33.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.155+0000 7f0833400640 10 client.5889 handle_command_reply: tid=0 2024-04-25T01:18:33.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.155+0000 7f0858366740 1 client.5889 shutdown 2024-04-25T01:18:33.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.155+0000 7f0858366740 1 -- 172.21.15.90:0/981548904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b6a2f8730 msgr2=0x561b6a2cd8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:33.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.155+0000 7f0858366740 1 --2- 172.21.15.90:0/981548904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b6a2f8730 0x561b6a2cd8f0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f0848009e50 tx=0x7f0848079b00 comp rx=0 tx=0).stop 2024-04-25T01:18:33.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.155+0000 7f0858366740 1 -- 172.21.15.90:0/981548904 shutdown_connections 2024-04-25T01:18:33.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.155+0000 7f0858366740 1 --2- 172.21.15.90:0/981548904 >> [v2:172.21.15.103:6832/2850373997,v1:172.21.15.103:6834/2850373997] conn(0x561b6a309750 0x561b6a309b60 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.155+0000 7f0858366740 1 --2- 172.21.15.90:0/981548904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b6a2f8730 0x561b6a2cd8f0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.155+0000 7f0858366740 1 --2- 172.21.15.90:0/981548904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b6a315120 0x561b6a2cd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.155+0000 7f0858366740 1 --2- 172.21.15.90:0/981548904 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6a2f8240 0x561b6a2e4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.155+0000 7f0833400640 0 client.5889 ms_handle_reset on v2:172.21.15.103:6832/2850373997 2024-04-25T01:18:33.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.155+0000 7f0858366740 1 -- 172.21.15.90:0/981548904 >> 172.21.15.90:0/981548904 conn(0x561b6a2f7b00 msgr2=0x561b6a3136f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:33.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.155+0000 7f0858366740 1 -- 172.21.15.90:0/981548904 shutdown_connections 2024-04-25T01:18:33.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.155+0000 7f0858366740 1 -- 172.21.15.90:0/981548904 wait complete. 2024-04-25T01:18:33.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.156+0000 7f0858366740 20 client.5889 trim_cache size 0 max 16384 2024-04-25T01:18:33.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.175+0000 7f0858366740 1 Processor -- start 2024-04-25T01:18:33.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.175+0000 7f0858366740 1 -- start start 2024-04-25T01:18:33.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.176+0000 7f0858366740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6a2f8240 0x561b6a313bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.176+0000 7f0858366740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b6a2f8730 0x561b6a308140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.176+0000 7f0858366740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b6a315120 0x561b6a2ccfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.176+0000 7f0858366740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0850202430 con 0x561b6a2f8730 2024-04-25T01:18:33.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.176+0000 7f0858366740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0850200af0 con 0x561b6a315120 2024-04-25T01:18:33.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.176+0000 7f0858366740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0850158b90 con 0x561b6a2f8240 2024-04-25T01:18:33.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.176+0000 7f0847e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b6a2f8730 0x561b6a308140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:33.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.176+0000 7f0847e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b6a2f8730 0x561b6a308140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48006/0 (socket says 172.21.15.90:48006) 2024-04-25T01:18:33.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.176+0000 7f0847e00640 1 -- 172.21.15.90:0/121663202 learned_addr learned my addr 172.21.15.90:0/121663202 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:33.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.176+0000 7f0847e00640 1 -- 172.21.15.90:0/121663202 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6a2f8240 msgr2=0x561b6a313bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:33.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.176+0000 7f0847e00640 1 --2- 172.21.15.90:0/121663202 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6a2f8240 0x561b6a313bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.176+0000 7f0847e00640 1 -- 172.21.15.90:0/121663202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b6a315120 msgr2=0x561b6a2ccfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:33.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.177+0000 7f0847e00640 1 --2- 172.21.15.90:0/121663202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b6a315120 0x561b6a2ccfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.177+0000 7f0847e00640 1 -- 172.21.15.90:0/121663202 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b69f04230 con 0x561b6a2f8730 2024-04-25T01:18:33.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.177+0000 7f0847e00640 1 --2- 172.21.15.90:0/121663202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b6a2f8730 0x561b6a308140 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f083800a750 tx=0x7f083800ac10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=548917ba96c6a8b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:33.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.177+0000 7f083e600640 1 -- 172.21.15.90:0/121663202 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0838015070 con 0x561b6a2f8730 2024-04-25T01:18:33.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.177+0000 7f083e600640 1 -- 172.21.15.90:0/121663202 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f083800cb60 con 0x561b6a2f8730 2024-04-25T01:18:33.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.177+0000 7f083e600640 1 -- 172.21.15.90:0/121663202 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f083800ae60 con 0x561b6a2f8730 2024-04-25T01:18:33.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.178+0000 7f0858366740 1 -- 172.21.15.90:0/121663202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b6a2f8730 msgr2=0x561b6a308140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:33.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.178+0000 7f0858366740 1 --2- 172.21.15.90:0/121663202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b6a2f8730 0x561b6a308140 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f083800a750 tx=0x7f083800ac10 comp rx=0 tx=0).stop 2024-04-25T01:18:33.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.178+0000 7f0858366740 1 -- 172.21.15.90:0/121663202 shutdown_connections 2024-04-25T01:18:33.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.178+0000 7f0858366740 1 --2- 172.21.15.90:0/121663202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b6a315120 0x561b6a2ccfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.178+0000 7f0858366740 1 --2- 172.21.15.90:0/121663202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b6a2f8730 0x561b6a308140 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.178+0000 7f0858366740 1 --2- 172.21.15.90:0/121663202 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6a2f8240 0x561b6a313bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.178+0000 7f0858366740 1 -- 172.21.15.90:0/121663202 >> 172.21.15.90:0/121663202 conn(0x561b6a2f7b00 msgr2=0x561b6a312570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:33.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.178+0000 7f0858366740 1 -- 172.21.15.90:0/121663202 shutdown_connections 2024-04-25T01:18:33.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.178+0000 7f0858366740 1 -- 172.21.15.90:0/121663202 wait complete. 2024-04-25T01:18:33.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.178+0000 7f0858366740 1 -- start start 2024-04-25T01:18:33.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.179+0000 7f0858366740 1 Processor -- start 2024-04-25T01:18:33.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.179+0000 7f0858366740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b6a2f8240 0x561b6a3083c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.179+0000 7f0858366740 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6a2f8730 0x561b6a2f3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.180+0000 7f0858366740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b6a315120 0x561b6a2f3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.180+0000 7f0858366740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561b6a3140f0 con 0x561b6a315120 2024-04-25T01:18:33.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.180+0000 7f0858366740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f08502004a0 con 0x561b6a2f8240 2024-04-25T01:18:33.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.180+0000 7f0858366740 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x561b6a2c8010 con 0x561b6a2f8730 2024-04-25T01:18:33.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.180+0000 7f0858366740 10 client.0 _ll_register_callbacks cb 0x7f08480092e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:18:33.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.180+0000 7f084f200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b6a315120 0x561b6a2f3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:33.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.180+0000 7f084f200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b6a315120 0x561b6a2f3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48016/0 (socket says 172.21.15.90:48016) 2024-04-25T01:18:33.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.180+0000 7f084f200640 1 -- 172.21.15.90:0/838455332 learned_addr learned my addr 172.21.15.90:0/838455332 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:33.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.180+0000 7f084f200640 1 -- 172.21.15.90:0/838455332 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6a2f8730 msgr2=0x561b6a2f3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:33.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.180+0000 7f084f200640 1 --2- 172.21.15.90:0/838455332 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6a2f8730 0x561b6a2f3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.180+0000 7f084f200640 1 -- 172.21.15.90:0/838455332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b6a2f8240 msgr2=0x561b6a3083c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:33.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.180+0000 7f084f200640 1 --2- 172.21.15.90:0/838455332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b6a2f8240 0x561b6a3083c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.180+0000 7f084f200640 1 -- 172.21.15.90:0/838455332 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b6a3246b0 con 0x561b6a315120 2024-04-25T01:18:33.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.180+0000 7f084f200640 1 --2- 172.21.15.90:0/838455332 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b6a315120 0x561b6a2f3e80 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f0848076540 tx=0x7f0848009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=981a62d97839d0e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:33.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.180+0000 7f0833400640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T01:18:33.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.181+0000 7f0833400640 1 -- 172.21.15.90:0/838455332 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0848092040 con 0x561b6a315120 2024-04-25T01:18:33.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.181+0000 7f0833400640 1 -- 172.21.15.90:0/838455332 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f0848078fa0 con 0x561b6a315120 2024-04-25T01:18:33.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.181+0000 7f0858366740 1 -- 172.21.15.90:0/838455332 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f083800cb60 con 0x561b6a315120 2024-04-25T01:18:33.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.181+0000 7f0833400640 1 -- 172.21.15.90:0/838455332 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0848079eb0 con 0x561b6a315120 2024-04-25T01:18:33.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.181+0000 7f0858366740 10 client.5895 fetch_fsmap learned FSMap version 28 2024-04-25T01:18:33.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.181+0000 7f0858366740 1 -- 172.21.15.90:0/838455332 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) -- 0x561b6a2cc300 con 0x561b6a315120 2024-04-25T01:18:33.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.181+0000 7f0833400640 1 -- 172.21.15.90:0/838455332 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f083800cb60 con 0x561b6a315120 2024-04-25T01:18:33.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.181+0000 7f0833400640 1 -- 172.21.15.90:0/838455332 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 4139+0+0 (secure 0 0 0) 0x7f08480791a0 con 0x561b6a315120 2024-04-25T01:18:33.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.181+0000 7f0858366740 10 client.5895 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:18:33.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.181+0000 7f0858366740 10 client.5895 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-25T01:18:33.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.181+0000 7f0858366740 20 client.5895 populate_metadata read hostname 'smithi090' 2024-04-25T01:18:33.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.181+0000 7f0858366740 1 --2- 172.21.15.90:0/838455332 >> [v2:172.21.15.103:6832/2850373997,v1:172.21.15.103:6834/2850373997] conn(0x561b6a2c6d40 0x561b6a321110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.181+0000 7f0858366740 4 client.5895 mds_command: new command op to 5601 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive", "repair"]}] 2024-04-25T01:18:33.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.181+0000 7f0858366740 1 -- 172.21.15.90:0/838455332 --> [v2:172.21.15.103:6832/2850373997,v1:172.21.15.103:6834/2850373997] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive", "repair"]}) -- 0x561b6a30c070 con 0x561b6a2c6d40 2024-04-25T01:18:33.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.182+0000 7f084e800640 1 --2- 172.21.15.90:0/838455332 >> [v2:172.21.15.103:6832/2850373997,v1:172.21.15.103:6834/2850373997] conn(0x561b6a2c6d40 0x561b6a321110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:33.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.183+0000 7f084e800640 1 --2- 172.21.15.90:0/838455332 >> [v2:172.21.15.103:6832/2850373997,v1:172.21.15.103:6834/2850373997] conn(0x561b6a2c6d40 0x561b6a321110 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b66c1cd00e04a04 server_cookie=52f6fae05dc6c218 in_seq=0 out_seq=0 2024-04-25T01:18:33.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.183+0000 7f0833400640 10 client.5895 ms_handle_connect on v2:172.21.15.103:6832/2850373997 2024-04-25T01:18:33.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.185+0000 7f0833400640 1 -- 172.21.15.90:0/838455332 <== mds.0 v2:172.21.15.103:6832/2850373997 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x561b6a30c070 con 0x561b6a2c6d40 2024-04-25T01:18:33.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.185+0000 7f0833400640 10 client.5895 handle_command_reply: tid=0 2024-04-25T01:18:33.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.185+0000 7f0858366740 1 client.5895 shutdown 2024-04-25T01:18:33.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.185+0000 7f0858366740 1 -- 172.21.15.90:0/838455332 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b6a315120 msgr2=0x561b6a2f3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:33.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.185+0000 7f0858366740 1 --2- 172.21.15.90:0/838455332 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b6a315120 0x561b6a2f3e80 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f0848076540 tx=0x7f0848009810 comp rx=0 tx=0).stop 2024-04-25T01:18:33.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.185+0000 7f0858366740 1 -- 172.21.15.90:0/838455332 shutdown_connections 2024-04-25T01:18:33.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.185+0000 7f0858366740 1 --2- 172.21.15.90:0/838455332 >> [v2:172.21.15.103:6832/2850373997,v1:172.21.15.103:6834/2850373997] conn(0x561b6a2c6d40 0x561b6a321110 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.185+0000 7f0858366740 1 --2- 172.21.15.90:0/838455332 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b6a315120 0x561b6a2f3e80 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.185+0000 7f0858366740 1 --2- 172.21.15.90:0/838455332 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x561b6a2f8730 0x561b6a2f3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.185+0000 7f0858366740 1 --2- 172.21.15.90:0/838455332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b6a2f8240 0x561b6a3083c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.185+0000 7f0833400640 0 client.5895 ms_handle_reset on v2:172.21.15.103:6832/2850373997 2024-04-25T01:18:33.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.185+0000 7f0858366740 1 -- 172.21.15.90:0/838455332 >> 172.21.15.90:0/838455332 conn(0x561b6a2f7b00 msgr2=0x561b6a3136a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:33.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.186+0000 7f0858366740 1 -- 172.21.15.90:0/838455332 shutdown_connections 2024-04-25T01:18:33.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.186+0000 7f0858366740 1 -- 172.21.15.90:0/838455332 wait complete. 2024-04-25T01:18:33.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.186+0000 7f0858366740 20 client.5895 trim_cache size 0 max 16384 2024-04-25T01:18:33.173 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T01:18:33.173 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-25T01:18:33.173 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "15bbd4d8-fcdd-4f82-9479-b4c3d2f99ce1", 2024-04-25T01:18:33.173 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-25T01:18:33.173 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T01:18:33.174 INFO:tasks.ceph.mds.d.smithi103.stderr:2024-04-25T01:18:33.186+0000 7f4f9e600640 -1 log_channel(cluster) log [ERR] : bad backtrace on directory inode 0x10000000000 2024-04-25T01:18:33.174 INFO:tasks.ceph.mds.d.smithi103.stderr:2024-04-25T01:18:33.189+0000 7f4f9e600640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000000 [...2,head] /subdir/ auth v1 pv4 ap=3 DIRTYPARENT f(v0 1=1+0) n() (inest lock dirty) (ifile lock->sync w=1 flushing) | dirtyscattered=2 lock=1 dirfrag=1 dirtyparent=1 waiter=1 authpin=1 scrubqueue=0 0x55d17e6b5080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(7)0x10000000000:[<0x1/subdir v1>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 1=1+0)","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 1=1+0)","memory_value.rstat":"n()","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-61} 2024-04-25T01:18:33.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.186+0000 7f0854a00640 1 -- 172.21.15.90:0/3200962980 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f082c06be30 msgr2=0x7f082c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:33.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.186+0000 7f0854a00640 1 --2- 172.21.15.90:0/3200962980 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f082c06be30 0x7f082c06e2f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0840000f60 tx=0x7f0840007a00 comp rx=0 tx=0).stop 2024-04-25T01:18:33.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.186+0000 7f0854a00640 1 -- 172.21.15.90:0/3200962980 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08500cc200 msgr2=0x7f08501482b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:33.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.186+0000 7f0854a00640 1 --2- 172.21.15.90:0/3200962980 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08500cc200 0x7f08501482b0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f0848034040 tx=0x7f0848004a40 comp rx=0 tx=0).stop 2024-04-25T01:18:33.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.187+0000 7f0854a00640 1 -- 172.21.15.90:0/3200962980 shutdown_connections 2024-04-25T01:18:33.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.187+0000 7f0854a00640 1 --2- 172.21.15.90:0/3200962980 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f082c06be30 0x7f082c06e2f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.187+0000 7f0854a00640 1 --2- 172.21.15.90:0/3200962980 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08500cc200 0x7f08501482b0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.187+0000 7f0854a00640 1 --2- 172.21.15.90:0/3200962980 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0850071f10 0x7f08501f4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.187+0000 7f0854a00640 1 --2- 172.21.15.90:0/3200962980 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f085006e230 0x7f08501e6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.187+0000 7f0854a00640 1 -- 172.21.15.90:0/3200962980 >> 172.21.15.90:0/3200962980 conn(0x7f08500b1460 msgr2=0x7f08500ad8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:33.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.187+0000 7f0854a00640 1 -- 172.21.15.90:0/3200962980 shutdown_connections 2024-04-25T01:18:33.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.187+0000 7f0854a00640 1 -- 172.21.15.90:0/3200962980 wait complete. 2024-04-25T01:18:33.176 INFO:tasks.ceph.mds.d.smithi103.stderr:2024-04-25T01:18:33.190+0000 7f4f9e600640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /subdir/sixmegs auth v1 ap=1 DIRTYPARENT s=6291456 n() | dirtyparent=1 authpin=1 scrubqueue=0 0x55d17e7ce580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(6)0x10000000001:[<0x10000000000/sixmegs v10>,<0x1/subdir v12>]//[]","memoryvalue":"(6)0x10000000001:[<0x10000000000/sixmegs v1>,<0x1/subdir v1>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-25T01:18:33.187 INFO:tasks.ceph.mds.d.smithi103.stderr:2024-04-25T01:18:33.203+0000 7f4f9e600640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x1 [...2,head] / auth v4 pv6 ap=5 snaprealm=0x55d17e6c6900 DIRTYPARENT f(v1 1=0+1) n(v0 1=0+1)/n() (inest mix->lock(2) w=1 flushing) (ifile mix->sync w=1 flushing) | dirtyscattered=2 lock=2 dirfrag=1 openingsnapparents=0 dirtyparent=1 dirty=1 waiter=1 authpin=1 scrubqueue=0 0x55d17e6b6680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(7)0x1:[]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 1=0+1)","ondisk_value.rstat":"n(v0 2=0+2)","memory_value.dirstat":"f(v1 1=0+1)","memory_value.rstat":"n(v0 1=0+1)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-61} 2024-04-25T01:18:33.189 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:18:33.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.403+0000 7f2c6a000640 1 Processor -- start 2024-04-25T01:18:33.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.403+0000 7f2c6a000640 1 -- start start 2024-04-25T01:18:33.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.405+0000 7f2c6a000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f2c640709d0 0x7f2c640cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.405+0000 7f2c6a000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c64070f90 con 0x7f2c640d0240 2024-04-25T01:18:33.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.405+0000 7f2c6a000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c64071270 con 0x7f2c641561c0 2024-04-25T01:18:33.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.405+0000 7f2c6a000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f2c640b4310 con 0x7f2c640709d0 2024-04-25T01:18:33.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.405+0000 7f2c5be00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f2c640d0240 0x7f2c640b2c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43776/0 (socket says 172.21.15.90:43776) 2024-04-25T01:18:33.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.405+0000 7f2c5be00640 1 -- 172.21.15.90:0/318755685 learned_addr learned my addr 172.21.15.90:0/318755685 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:33.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.406+0000 7f2c69600640 1 --2- 172.21.15.90:0/318755685 >> v2:172.21.15.103:3300/0 conn(0x7f2c640709d0 0x7f2c640cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:33.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.406+0000 7f2c63e00640 1 -- 172.21.15.90:0/318755685 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3259376544 0 0) 0x7f2c64070f90 con 0x7f2c640d0240 2024-04-25T01:18:33.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.406+0000 7f2c63e00640 1 -- 172.21.15.90:0/318755685 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c48003520 con 0x7f2c640d0240 2024-04-25T01:18:33.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.406+0000 7f2c63e00640 1 -- 172.21.15.90:0/318755685 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3381495936 0 0) 0x7f2c48003520 con 0x7f2c640d0240 2024-04-25T01:18:33.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.406+0000 7f2c63e00640 1 -- 172.21.15.90:0/318755685 >> v2:172.21.15.103:3300/0 conn(0x7f2c640709d0 msgr2=0x7f2c640cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:33.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.406+0000 7f2c63e00640 1 --2- 172.21.15.90:0/318755685 >> v2:172.21.15.103:3300/0 conn(0x7f2c640709d0 0x7f2c640cdbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.406+0000 7f2c63e00640 1 -- 172.21.15.90:0/318755685 >> v1:172.21.15.103:6789/0 conn(0x7f2c641561c0 legacy=0x7f2c640cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:33.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.406+0000 7f2c63e00640 1 -- 172.21.15.90:0/318755685 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c64071550 con 0x7f2c640d0240 2024-04-25T01:18:33.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.406+0000 7f2c63e00640 1 -- 172.21.15.90:0/318755685 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f2c4c001670 con 0x7f2c640d0240 2024-04-25T01:18:33.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.406+0000 7f2c63e00640 1 -- 172.21.15.90:0/318755685 >> v1:172.21.15.90:6789/0 conn(0x7f2c640d0240 legacy=0x7f2c640b2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:33.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.406+0000 7f2c63e00640 1 --2- 172.21.15.90:0/318755685 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c480039b0 0x7f2c48003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.406+0000 7f2c63e00640 1 --2- 172.21.15.90:0/318755685 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c48004800 0x7f2c48004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.406+0000 7f2c63e00640 1 --2- 172.21.15.90:0/318755685 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c480052e0 0x7f2c4800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.406+0000 7f2c63e00640 1 -- 172.21.15.90:0/318755685 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2c640b4310 con 0x7f2c48004800 2024-04-25T01:18:33.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.406+0000 7f2c63e00640 1 -- 172.21.15.90:0/318755685 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4800c1b0 con 0x7f2c480039b0 2024-04-25T01:18:33.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.406+0000 7f2c63e00640 1 -- 172.21.15.90:0/318755685 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2c4800c3b0 con 0x7f2c480052e0 2024-04-25T01:18:33.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.407+0000 7f2c5be00640 1 --2- 172.21.15.90:0/318755685 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c48004800 0x7f2c48004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:33.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.407+0000 7f2c68c00640 1 --2- 172.21.15.90:0/318755685 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c480039b0 0x7f2c48003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:33.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.407+0000 7f2c69600640 1 --2- 172.21.15.90:0/318755685 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c480052e0 0x7f2c4800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:33.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.407+0000 7f2c5be00640 1 -- 172.21.15.90:0/318755685 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c480052e0 msgr2=0x7f2c4800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:33.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.407+0000 7f2c5be00640 1 --2- 172.21.15.90:0/318755685 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c480052e0 0x7f2c4800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.407+0000 7f2c5be00640 1 -- 172.21.15.90:0/318755685 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c480039b0 msgr2=0x7f2c48003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:33.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.407+0000 7f2c5be00640 1 --2- 172.21.15.90:0/318755685 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c480039b0 0x7f2c48003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.407+0000 7f2c5be00640 1 -- 172.21.15.90:0/318755685 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c48003520 con 0x7f2c48004800 2024-04-25T01:18:33.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.407+0000 7f2c69600640 1 -- 172.21.15.90:0/318755685 reap_dead start 2024-04-25T01:18:33.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.408+0000 7f2c5be00640 1 --2- 172.21.15.90:0/318755685 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c48004800 0x7f2c48004c80 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c00c210 tx=0x7f2c48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=946b197eb503bf7f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:33.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.408+0000 7f2c63e00640 1 -- 172.21.15.90:0/318755685 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c007cf0 con 0x7f2c48004800 2024-04-25T01:18:33.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.408+0000 7f2c63e00640 1 -- 172.21.15.90:0/318755685 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c4c005af0 con 0x7f2c48004800 2024-04-25T01:18:33.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.408+0000 7f2c63e00640 1 -- 172.21.15.90:0/318755685 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c007070 con 0x7f2c48004800 2024-04-25T01:18:33.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.409+0000 7f2c6a000640 1 -- 172.21.15.90:0/318755685 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c48004800 msgr2=0x7f2c48004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:33.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.409+0000 7f2c6a000640 1 --2- 172.21.15.90:0/318755685 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c48004800 0x7f2c48004c80 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c00c210 tx=0x7f2c48005830 comp rx=0 tx=0).stop 2024-04-25T01:18:33.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.409+0000 7f2c6a000640 1 -- 172.21.15.90:0/318755685 shutdown_connections 2024-04-25T01:18:33.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.409+0000 7f2c6a000640 1 --2- 172.21.15.90:0/318755685 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c48004800 0x7f2c48004c80 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.409+0000 7f2c6a000640 1 -- 172.21.15.90:0/318755685 >> 172.21.15.90:0/318755685 conn(0x7f2c6406b320 msgr2=0x7f2c6406b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:33.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.409+0000 7f2c6a000640 1 -- 172.21.15.90:0/318755685 shutdown_connections 2024-04-25T01:18:33.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.409+0000 7f2c6a000640 1 -- 172.21.15.90:0/318755685 wait complete. 2024-04-25T01:18:33.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.411+0000 7f2c6a000640 1 Processor -- start 2024-04-25T01:18:33.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.411+0000 7f2c6a000640 1 -- start start 2024-04-25T01:18:33.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.412+0000 7f2c6a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c48004800 0x7f2c641f7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.412+0000 7f2c6a000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c64145470 0x7f2c641f38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.412+0000 7f2c6a000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c641f3e00 0x7f2c640ad8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.412+0000 7f2c6a000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2c640cdbf0 con 0x7f2c48004800 2024-04-25T01:18:33.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.412+0000 7f2c6a000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2c640cdd70 con 0x7f2c641f3e00 2024-04-25T01:18:33.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.412+0000 7f2c6a000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2c640cdef0 con 0x7f2c64145470 2024-04-25T01:18:33.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.412+0000 7f2c68c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c48004800 0x7f2c641f7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:33.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.412+0000 7f2c68c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c48004800 0x7f2c641f7970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48034/0 (socket says 172.21.15.90:48034) 2024-04-25T01:18:33.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.412+0000 7f2c68c00640 1 -- 172.21.15.90:0/666403265 learned_addr learned my addr 172.21.15.90:0/666403265 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:33.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.413+0000 7f2c68c00640 1 -- 172.21.15.90:0/666403265 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c64145470 msgr2=0x7f2c641f38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:33.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.413+0000 7f2c68c00640 1 --2- 172.21.15.90:0/666403265 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c64145470 0x7f2c641f38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.413+0000 7f2c68c00640 1 -- 172.21.15.90:0/666403265 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c641f3e00 msgr2=0x7f2c640ad8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:33.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.413+0000 7f2c69600640 1 --2- 172.21.15.90:0/666403265 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c641f3e00 0x7f2c640ad8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:33.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.413+0000 7f2c68c00640 1 --2- 172.21.15.90:0/666403265 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c641f3e00 0x7f2c640ad8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.413+0000 7f2c68c00640 1 -- 172.21.15.90:0/666403265 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c641f6eb0 con 0x7f2c48004800 2024-04-25T01:18:33.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.413+0000 7f2c68c00640 1 --2- 172.21.15.90:0/666403265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c48004800 0x7f2c641f7970 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f2c5400e4b0 tx=0x7f2c5400e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:33.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.413+0000 7f2c61600640 1 -- 172.21.15.90:0/666403265 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c540052e0 con 0x7f2c48004800 2024-04-25T01:18:33.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.413+0000 7f2c61600640 1 -- 172.21.15.90:0/666403265 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c54018070 con 0x7f2c48004800 2024-04-25T01:18:33.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.413+0000 7f2c61600640 1 -- 172.21.15.90:0/666403265 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c54003810 con 0x7f2c48004800 2024-04-25T01:18:33.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.413+0000 7f2c6a000640 1 -- 172.21.15.90:0/666403265 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c641f43f0 con 0x7f2c48004800 2024-04-25T01:18:33.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.413+0000 7f2c6a000640 1 -- 172.21.15.90:0/666403265 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c641f7170 con 0x7f2c48004800 2024-04-25T01:18:33.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.415+0000 7f2c6a000640 1 -- 172.21.15.90:0/666403265 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c641458b0 con 0x7f2c48004800 2024-04-25T01:18:33.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.416+0000 7f2c61600640 1 -- 172.21.15.90:0/666403265 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2c540039b0 con 0x7f2c48004800 2024-04-25T01:18:33.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.416+0000 7f2c61600640 1 --2- 172.21.15.90:0/666403265 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2c4006be60 0x7f2c4006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.417+0000 7f2c61600640 1 -- 172.21.15.90:0/666403265 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f2c5408ab80 con 0x7f2c48004800 2024-04-25T01:18:33.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.417+0000 7f2c5be00640 1 --2- 172.21.15.90:0/666403265 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2c4006be60 0x7f2c4006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:33.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.417+0000 7f2c5be00640 1 --2- 172.21.15.90:0/666403265 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2c4006be60 0x7f2c4006e320 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c032000 tx=0x7f2c4c007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:33.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.421+0000 7f2c61600640 1 -- 172.21.15.90:0/666403265 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2c641458b0 con 0x7f2c48004800 2024-04-25T01:18:33.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.655+0000 7f2c6a000640 1 -- 172.21.15.90:0/666403265 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c6404f500 con 0x7f2c48004800 2024-04-25T01:18:33.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.656+0000 7f2c61600640 1 -- 172.21.15.90:0/666403265 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6492 (secure 0 0 0) 0x7f2c54059150 con 0x7f2c48004800 2024-04-25T01:18:33.640 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:33.640 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1873497946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6833","nonce":1873497946},{"type":"v1","addr":"172.21.15.103:6835","nonce":1873497946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/3206865830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":3206865830},{"type":"v1","addr":"172.21.15.90:6835","nonce":3206865830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5598,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2208594231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2208594231},{"type":"v1","addr":"172.21.15.90:6837","nonce":2208594231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:18:02.701652+0000","modified":"2024-04-25T01:18:15.603132+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2},{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:18:19.371011+0000","modified":"2024-04-25T01:18:30.533899+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5601":{"gid":5601,"name":"d","rank":0,"incarnation":25,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6834/2850373997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":2850373997},{"type":"v1","addr":"172.21.15.103:6834","nonce":2850373997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":7,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5601,"qdb_cluster":[5601]},"id":3}]} 2024-04-25T01:18:33.640 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 28 2024-04-25T01:18:33.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.658+0000 7f2c6a000640 1 -- 172.21.15.90:0/666403265 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2c4006be60 msgr2=0x7f2c4006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:33.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.658+0000 7f2c6a000640 1 --2- 172.21.15.90:0/666403265 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2c4006be60 0x7f2c4006e320 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c032000 tx=0x7f2c4c007000 comp rx=0 tx=0).stop 2024-04-25T01:18:33.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.658+0000 7f2c6a000640 1 -- 172.21.15.90:0/666403265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c48004800 msgr2=0x7f2c641f7970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:33.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.658+0000 7f2c6a000640 1 --2- 172.21.15.90:0/666403265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c48004800 0x7f2c641f7970 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f2c5400e4b0 tx=0x7f2c5400e970 comp rx=0 tx=0).stop 2024-04-25T01:18:33.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.658+0000 7f2c6a000640 1 -- 172.21.15.90:0/666403265 shutdown_connections 2024-04-25T01:18:33.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.658+0000 7f2c6a000640 1 --2- 172.21.15.90:0/666403265 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2c4006be60 0x7f2c4006e320 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.658+0000 7f2c6a000640 1 --2- 172.21.15.90:0/666403265 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c641f3e00 0x7f2c640ad8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.658+0000 7f2c6a000640 1 --2- 172.21.15.90:0/666403265 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2c64145470 0x7f2c641f38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.658+0000 7f2c6a000640 1 --2- 172.21.15.90:0/666403265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c48004800 0x7f2c641f7970 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.658+0000 7f2c6a000640 1 -- 172.21.15.90:0/666403265 >> 172.21.15.90:0/666403265 conn(0x7f2c6406b320 msgr2=0x7f2c640ce900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:33.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.659+0000 7f2c6a000640 1 -- 172.21.15.90:0/666403265 shutdown_connections 2024-04-25T01:18:33.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.659+0000 7f2c6a000640 1 -- 172.21.15.90:0/666403265 wait complete. 2024-04-25T01:18:33.661 INFO:tasks.cephfs.test_recovery_pool:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 28, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T01:18:02.701652+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 19, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [ 0 ], "flags": 19, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": false, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": {}, "last_failure": 0, "last_failure_osd_epoch": 43, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 5, "modified": "2024-04-25T01:18:15.603132+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": {} } }, { "id": 3, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T01:18:19.371011+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 28, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "recovery_fs", "in": [ 0 ], "info": { "gid_5601": { "addr": "172.21.15.103:6834/2850373997", "addrs": { "addrvec": [ { "addr": "172.21.15.103:6832", "nonce": 2850373997, "type": "v2" }, { "addr": "172.21.15.103:6834", "nonce": 2850373997, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5601, "incarnation": 25, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 7, "modified": "2024-04-25T01:18:30.533899+0000", "qdb_cluster": [ 5601 ], "qdb_leader": 5601, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5601 } } } ], "standbys": [ { "addr": "172.21.15.103:6835/1873497946", "addrs": { "addrvec": [ { "addr": "172.21.15.103:6833", "nonce": 1873497946, "type": "v2" }, { "addr": "172.21.15.103:6835", "nonce": 1873497946, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 20, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4966, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.90:6835/3206865830", "addrs": { "addrvec": [ { "addr": "172.21.15.90:6834", "nonce": 3206865830, "type": "v2" }, { "addr": "172.21.15.90:6835", "nonce": 3206865830, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 20, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5595, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.90:6837/2208594231", "addrs": { "addrvec": [ { "addr": "172.21.15.90:6836", "nonce": 2208594231, "type": "v2" }, { "addr": "172.21.15.90:6837", "nonce": 2208594231, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 20, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5598, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T01:18:33.661 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T01:18:33.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.867+0000 7f9ef6a00640 1 Processor -- start 2024-04-25T01:18:33.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.867+0000 7f9ef6a00640 1 -- start start 2024-04-25T01:18:33.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.869+0000 7f9ef6a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ef015bb50 0x7f9ef00722b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.869+0000 7f9ef6a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ef015ee70 con 0x7f9ef006b850 2024-04-25T01:18:33.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.869+0000 7f9ef6a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ef015f150 con 0x7f9ef015e3e0 2024-04-25T01:18:33.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.869+0000 7f9ef6a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9ef00cbd60 con 0x7f9ef015bb50 2024-04-25T01:18:33.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.869+0000 7f9ef6000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f9ef006b850 0x7f9ef00ce830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43792/0 (socket says 172.21.15.90:43792) 2024-04-25T01:18:33.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.869+0000 7f9ef6000640 1 -- 172.21.15.90:0/2658829309 learned_addr learned my addr 172.21.15.90:0/2658829309 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:33.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.870+0000 7f9ef5600640 1 --2- 172.21.15.90:0/2658829309 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ef015bb50 0x7f9ef00722b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:33.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.870+0000 7f9eefe00640 1 -- 172.21.15.90:0/2658829309 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2394054429 0 0) 0x7f9ef015ee70 con 0x7f9ef006b850 2024-04-25T01:18:33.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.870+0000 7f9eefe00640 1 -- 172.21.15.90:0/2658829309 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ed8003520 con 0x7f9ef006b850 2024-04-25T01:18:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.870+0000 7f9eefe00640 1 -- 172.21.15.90:0/2658829309 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3121162167 0 0) 0x7f9ed8003520 con 0x7f9ef006b850 2024-04-25T01:18:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.870+0000 7f9eefe00640 1 -- 172.21.15.90:0/2658829309 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ef015bb50 msgr2=0x7f9ef00722b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.870+0000 7f9eefe00640 1 --2- 172.21.15.90:0/2658829309 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ef015bb50 0x7f9ef00722b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.870+0000 7f9eefe00640 1 -- 172.21.15.90:0/2658829309 >> v1:172.21.15.103:6789/0 conn(0x7f9ef015e3e0 legacy=0x7f9ef006b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.870+0000 7f9eefe00640 1 -- 172.21.15.90:0/2658829309 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ef0071ab0 con 0x7f9ef006b850 2024-04-25T01:18:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.870+0000 7f9eefe00640 1 -- 172.21.15.90:0/2658829309 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f9ee80019f0 con 0x7f9ef006b850 2024-04-25T01:18:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.870+0000 7f9eefe00640 1 -- 172.21.15.90:0/2658829309 >> v1:172.21.15.90:6789/0 conn(0x7f9ef006b850 legacy=0x7f9ef00ce830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.870+0000 7f9eefe00640 1 --2- 172.21.15.90:0/2658829309 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ed80039e0 0x7f9ed8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.870+0000 7f9eefe00640 1 --2- 172.21.15.90:0/2658829309 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ed80043d0 0x7f9ed80048a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.870+0000 7f9eefe00640 1 --2- 172.21.15.90:0/2658829309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ed80050a0 0x7f9ed800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.870+0000 7f9eefe00640 1 -- 172.21.15.90:0/2658829309 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed800bfb0 con 0x7f9ed80043d0 2024-04-25T01:18:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.870+0000 7f9eefe00640 1 -- 172.21.15.90:0/2658829309 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed800c130 con 0x7f9ed80050a0 2024-04-25T01:18:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.870+0000 7f9eefe00640 1 -- 172.21.15.90:0/2658829309 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9ed800c330 con 0x7f9ed80039e0 2024-04-25T01:18:33.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.870+0000 7f9ef4c00640 1 --2- 172.21.15.90:0/2658829309 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ed80043d0 0x7f9ed80048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:33.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.871+0000 7f9ef4c00640 1 -- 172.21.15.90:0/2658829309 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ed80039e0 msgr2=0x7f9ed8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:33.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.871+0000 7f9ef4c00640 1 --2- 172.21.15.90:0/2658829309 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ed80039e0 0x7f9ed8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.871+0000 7f9ef6000640 1 --2- 172.21.15.90:0/2658829309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ed80050a0 0x7f9ed800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:33.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.871+0000 7f9ef4c00640 1 -- 172.21.15.90:0/2658829309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ed80050a0 msgr2=0x7f9ed800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:33.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.871+0000 7f9ef4c00640 1 --2- 172.21.15.90:0/2658829309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ed80050a0 0x7f9ed800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.871+0000 7f9ef4c00640 1 -- 172.21.15.90:0/2658829309 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ed8003520 con 0x7f9ed80043d0 2024-04-25T01:18:33.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.871+0000 7f9ef6000640 1 --2- 172.21.15.90:0/2658829309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ed80050a0 0x7f9ed800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:33.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.872+0000 7f9ef4c00640 1 --2- 172.21.15.90:0/2658829309 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ed80043d0 0x7f9ed80048a0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f9edc001750 tx=0x7f9ed8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7fb308558737310 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:33.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.872+0000 7f9eefe00640 1 -- 172.21.15.90:0/2658829309 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9edc001370 con 0x7f9ed80043d0 2024-04-25T01:18:33.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.872+0000 7f9eefe00640 1 -- 172.21.15.90:0/2658829309 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9edc0052d0 con 0x7f9ed80043d0 2024-04-25T01:18:33.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.872+0000 7f9eefe00640 1 -- 172.21.15.90:0/2658829309 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9edc02abe0 con 0x7f9ed80043d0 2024-04-25T01:18:33.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.873+0000 7f9ef6a00640 1 -- 172.21.15.90:0/2658829309 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ed80043d0 msgr2=0x7f9ed80048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:33.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.873+0000 7f9ef6a00640 1 --2- 172.21.15.90:0/2658829309 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ed80043d0 0x7f9ed80048a0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f9edc001750 tx=0x7f9ed8005830 comp rx=0 tx=0).stop 2024-04-25T01:18:33.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.873+0000 7f9ef6000640 1 -- 172.21.15.90:0/2658829309 reap_dead start 2024-04-25T01:18:33.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.873+0000 7f9ef6a00640 1 -- 172.21.15.90:0/2658829309 shutdown_connections 2024-04-25T01:18:33.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.873+0000 7f9ef6a00640 1 -- 172.21.15.90:0/2658829309 >> 172.21.15.90:0/2658829309 conn(0x7f9ef00d0a50 msgr2=0x7f9ef00d0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:33.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.873+0000 7f9ef6a00640 1 -- 172.21.15.90:0/2658829309 shutdown_connections 2024-04-25T01:18:33.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.873+0000 7f9ef6a00640 1 -- 172.21.15.90:0/2658829309 wait complete. 2024-04-25T01:18:33.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.875+0000 7f9ef6a00640 1 Processor -- start 2024-04-25T01:18:33.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.875+0000 7f9ef6a00640 1 -- start start 2024-04-25T01:18:33.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.876+0000 7f9ef6a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ed80043d0 0x7f9ef01e4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.876+0000 7f9ef6a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ef0154cb0 0x7f9ef01e5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.876+0000 7f9ef6a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ef00c9870 0x7f9ef01540d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.876+0000 7f9ef6a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef006ffc0 con 0x7f9ef00c9870 2024-04-25T01:18:33.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.876+0000 7f9ef6a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef006fe40 con 0x7f9ef0154cb0 2024-04-25T01:18:33.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.876+0000 7f9ef6a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9ef015e840 con 0x7f9ed80043d0 2024-04-25T01:18:33.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.876+0000 7f9ef6000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ef00c9870 0x7f9ef01540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:33.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.876+0000 7f9ef6000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ef00c9870 0x7f9ef01540d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48054/0 (socket says 172.21.15.90:48054) 2024-04-25T01:18:33.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.876+0000 7f9ef6000640 1 -- 172.21.15.90:0/2528301040 learned_addr learned my addr 172.21.15.90:0/2528301040 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:33.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.876+0000 7f9ef6000640 1 -- 172.21.15.90:0/2528301040 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ed80043d0 msgr2=0x7f9ef01e4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:33.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.876+0000 7f9ef6000640 1 --2- 172.21.15.90:0/2528301040 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ed80043d0 0x7f9ef01e4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.876+0000 7f9ef4c00640 1 --2- 172.21.15.90:0/2528301040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ef0154cb0 0x7f9ef01e5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:33.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.876+0000 7f9ef6000640 1 -- 172.21.15.90:0/2528301040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ef0154cb0 msgr2=0x7f9ef01e5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:33.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.876+0000 7f9ef6000640 1 --2- 172.21.15.90:0/2528301040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ef0154cb0 0x7f9ef01e5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:33.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.876+0000 7f9ef6000640 1 -- 172.21.15.90:0/2528301040 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ef01e5fd0 con 0x7f9ef00c9870 2024-04-25T01:18:33.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.877+0000 7f9ef6000640 1 --2- 172.21.15.90:0/2528301040 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ef00c9870 0x7f9ef01540d0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0067bc0 tx=0x7f9ee800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:33.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.877+0000 7f9eed600640 1 -- 172.21.15.90:0/2528301040 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8004d60 con 0x7f9ef00c9870 2024-04-25T01:18:33.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.877+0000 7f9eed600640 1 -- 172.21.15.90:0/2528301040 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ee800bb80 con 0x7f9ef00c9870 2024-04-25T01:18:33.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.877+0000 7f9eed600640 1 -- 172.21.15.90:0/2528301040 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8007110 con 0x7f9ef00c9870 2024-04-25T01:18:33.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.877+0000 7f9ef6a00640 1 -- 172.21.15.90:0/2528301040 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ef00c46b0 con 0x7f9ef00c9870 2024-04-25T01:18:33.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.877+0000 7f9ef6a00640 1 -- 172.21.15.90:0/2528301040 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ef00c4c70 con 0x7f9ef00c9870 2024-04-25T01:18:33.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.878+0000 7f9ef6a00640 1 -- 172.21.15.90:0/2528301040 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ef004f500 con 0x7f9ef00c9870 2024-04-25T01:18:33.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.879+0000 7f9eed600640 1 -- 172.21.15.90:0/2528301040 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9ee801b020 con 0x7f9ef00c9870 2024-04-25T01:18:33.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.880+0000 7f9eed600640 1 --2- 172.21.15.90:0/2528301040 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9ed006be30 0x7f9ed006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:33.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.880+0000 7f9eed600640 1 -- 172.21.15.90:0/2528301040 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f9ee808b4b0 con 0x7f9ef00c9870 2024-04-25T01:18:33.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.880+0000 7f9ef5600640 1 --2- 172.21.15.90:0/2528301040 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9ed006be30 0x7f9ed006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:33.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.881+0000 7f9ef5600640 1 --2- 172.21.15.90:0/2528301040 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9ed006be30 0x7f9ed006e2f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9ee4006a90 tx=0x7f9ee4006a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:33.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:33.884+0000 7f9eed600640 1 -- 172.21.15.90:0/2528301040 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9ee8059a70 con 0x7f9ef00c9870 2024-04-25T01:18:34.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:34.125+0000 7f9ef6a00640 1 -- 172.21.15.90:0/2528301040 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f9ef006f3d0 con 0x7f9ef00c9870 2024-04-25T01:18:34.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:34.126+0000 7f9eed600640 1 -- 172.21.15.90:0/2528301040 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) ==== 53+0+156 (secure 0 0 0) 0x7f9ee8059370 con 0x7f9ef00c9870 2024-04-25T01:18:34.110 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T01:18:34.110 INFO:teuthology.orchestra.run.smithi090.stdout:name: recovery_fs, metadata pool: recovery_fs_metadata, data pools: [cephfs_data ] 2024-04-25T01:18:34.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:34.129+0000 7f9ef6a00640 1 -- 172.21.15.90:0/2528301040 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9ed006be30 msgr2=0x7f9ed006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:34.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:34.129+0000 7f9ef6a00640 1 --2- 172.21.15.90:0/2528301040 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9ed006be30 0x7f9ed006e2f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9ee4006a90 tx=0x7f9ee4006a20 comp rx=0 tx=0).stop 2024-04-25T01:18:34.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:34.129+0000 7f9ef6a00640 1 -- 172.21.15.90:0/2528301040 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ef00c9870 msgr2=0x7f9ef01540d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:34.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:34.129+0000 7f9ef6a00640 1 --2- 172.21.15.90:0/2528301040 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ef00c9870 0x7f9ef01540d0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0067bc0 tx=0x7f9ee800b4a0 comp rx=0 tx=0).stop 2024-04-25T01:18:34.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:34.129+0000 7f9ef6a00640 1 -- 172.21.15.90:0/2528301040 shutdown_connections 2024-04-25T01:18:34.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:34.129+0000 7f9ef6a00640 1 --2- 172.21.15.90:0/2528301040 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9ed006be30 0x7f9ed006e2f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:34.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:34.129+0000 7f9ef6a00640 1 --2- 172.21.15.90:0/2528301040 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ef00c9870 0x7f9ef01540d0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:34.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:34.129+0000 7f9ef6a00640 1 --2- 172.21.15.90:0/2528301040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ef0154cb0 0x7f9ef01e5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:34.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:34.129+0000 7f9ef6a00640 1 --2- 172.21.15.90:0/2528301040 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9ed80043d0 0x7f9ef01e4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:34.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:34.129+0000 7f9ef6a00640 1 -- 172.21.15.90:0/2528301040 >> 172.21.15.90:0/2528301040 conn(0x7f9ef00d0a50 msgr2=0x7f9ef00721b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:34.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:34.129+0000 7f9ef6a00640 1 -- 172.21.15.90:0/2528301040 shutdown_connections 2024-04-25T01:18:34.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:34.129+0000 7f9ef6a00640 1 -- 172.21.15.90:0/2528301040 wait complete. 2024-04-25T01:18:34.132 INFO:tasks.cephfs.mount:Mounting Ceph FS recovery_fs; just confirmed its presence on cluster 2024-04-25T01:18:34.132 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T01:18:34.132 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2024-04-25T01:18:34.132 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T01:18:34.132 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T01:18:34.132 INFO:tasks.cephfs.mount:self.cephfs_name = recovery_fs 2024-04-25T01:18:34.132 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-25T01:18:34.133 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T01:18:34.133 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T01:18:34.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:34.133 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2024-04-25T01:18:34.183 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T01:18:34.183 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T01:18:34.183 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T01:18:34.183 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:18:34.183 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-04-25T01:18:34.183 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:18:34.183 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:18:34.183 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:18:34.183 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f0 2024-04-25T01:18:34.183 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:18:34.183 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:18:34.183 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f1 2024-04-25T01:18:34.183 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:18:34.184 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:18:34.184 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T01:18:34.184 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:18:34.184 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-25T01:18:34.184 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 42525sec preferred_lft 42525sec 2024-04-25T01:18:34.184 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::ec4:7aff:fe8f:d767/64 scope link 2024-04-25T01:18:34.184 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:18:34.184 INFO:teuthology.orchestra.run.smithi090.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T01:18:34.184 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 1e:6a:0d:c3:bc:1e brd ff:ff:ff:ff:ff:ff 2024-04-25T01:18:34.184 INFO:teuthology.orchestra.run.smithi090.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T01:18:34.184 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:18:34.184 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::405b:4dff:feb7:7d54/64 scope link 2024-04-25T01:18:34.184 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:18:34.184 INFO:teuthology.orchestra.run.smithi090.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T01:18:34.185 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 1e:6a:0d:c3:bc:1e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T01:18:34.185 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::1c6a:dff:fec3:bc1e/64 scope link 2024-04-25T01:18:34.185 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:18:34.185 INFO:teuthology.orchestra.run.smithi090.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T01:18:34.185 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 9a:38:36:61:4f:7c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T01:18:34.185 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::9838:36ff:fe61:4f7c/64 scope link 2024-04-25T01:18:34.185 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:18:34.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:34.185 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T01:18:34.264 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-04-25T01:18:34.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:34.269 DEBUG:teuthology.orchestra.run.smithi090:> route 2024-04-25T01:18:34.319 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2024-04-25T01:18:34.319 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T01:18:34.319 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T01:18:34.319 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T01:18:34.319 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T01:18:34.320 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:18:34.320 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-04-25T01:18:34.320 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T01:18:34.320 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T01:18:34.320 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T01:18:34.320 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-04-25T01:18:34.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:34 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:18:34.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:18:34.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:34.524 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-04-25T01:18:34.570 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T01:18:34.571 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-25T01:18:34.572 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T01:18:34.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:34.572 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.0 up 2024-04-25T01:18:34.700 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:18:34.701 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T01:18:34.754 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T01:18:34.755 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:18:34.755 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T01:18:34.805 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-25T01:18:34.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:34.806 DEBUG:teuthology.orchestra.run.smithi090:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=recovery_fs 2024-04-25T01:18:34.931 INFO:teuthology.orchestra.run.smithi090.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=recovery_fs 2024-04-25T01:18:34.932 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=recovery_fs". 2024-04-25T01:18:34.932 INFO:teuthology.orchestra.run.smithi090.stdout:invalid new device string format 2024-04-25T01:18:34.953 INFO:teuthology.orchestra.run.smithi090.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=recovery_fs 2024-04-25T01:18:34.953 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=recovery_fs". 2024-04-25T01:18:34.953 INFO:teuthology.orchestra.run.smithi090.stdout:invalid new device string format 2024-04-25T01:18:34.953 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: resolved to: "172.21.15.90:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-25T01:18:34.953 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-25T01:18:34.983 INFO:teuthology.orchestra.run.smithi090.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=recovery_fs 2024-04-25T01:18:34.984 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=recovery_fs". 2024-04-25T01:18:34.984 INFO:teuthology.orchestra.run.smithi090.stdout:invalid new device string format 2024-04-25T01:18:34.984 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: resolved to: "172.21.15.90:3300,172.21.15.103:3300,172.21.15.103:3301" 2024-04-25T01:18:34.984 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-25T01:18:34.984 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: resolved to: "172.21.15.90:6789,172.21.15.103:6789,172.21.15.103:6790" 2024-04-25T01:18:34.984 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.90:6789,172.21.15.103:6789,172.21.15.103:6790:/ 2024-04-25T01:18:34.984 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=recovery_fs,key=0,fsid=e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21" will pass to kernel 2024-04-25T01:18:34.986 INFO:teuthology.orchestra.run.smithi090.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-25T01:18:34.986 INFO:teuthology.orchestra.run.smithi090.stdout: You just mounted a file system that supports labels which does not 2024-04-25T01:18:34.986 INFO:teuthology.orchestra.run.smithi090.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T01:18:34.986 INFO:teuthology.orchestra.run.smithi090.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T01:18:34.987 INFO:teuthology.orchestra.run.smithi090.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T01:18:34.990 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T01:18:34.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:34.990 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T01:18:35.064 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:18:35.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:35 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:18:35.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:18:35.208 INFO:teuthology.orchestra.run.smithi090.stdout:client4763 2024-04-25T01:18:35.209 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:18:35.209 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:18:35.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:35.264 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls subdir) 2024-04-25T01:18:35.349 INFO:teuthology.orchestra.run.smithi090.stdout:sixmegs 2024-04-25T01:18:35.351 DEBUG:teuthology.orchestra.run.smithi090:> stdin-killer -- python3 -c ' 2024-04-25T01:18:35.351 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-04-25T01:18:35.351 DEBUG:teuthology.orchestra.run.smithi090:> import stat 2024-04-25T01:18:35.351 DEBUG:teuthology.orchestra.run.smithi090:> import json 2024-04-25T01:18:35.351 DEBUG:teuthology.orchestra.run.smithi090:> import sys 2024-04-25T01:18:35.351 DEBUG:teuthology.orchestra.run.smithi090:> 2024-04-25T01:18:35.352 DEBUG:teuthology.orchestra.run.smithi090:> try: 2024-04-25T01:18:35.352 DEBUG:teuthology.orchestra.run.smithi090:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/subdir/sixmegs'"'"') 2024-04-25T01:18:35.352 DEBUG:teuthology.orchestra.run.smithi090:> except OSError as e: 2024-04-25T01:18:35.352 DEBUG:teuthology.orchestra.run.smithi090:> sys.exit(e.errno) 2024-04-25T01:18:35.352 DEBUG:teuthology.orchestra.run.smithi090:> 2024-04-25T01:18:35.352 DEBUG:teuthology.orchestra.run.smithi090:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-04-25T01:18:35.352 DEBUG:teuthology.orchestra.run.smithi090:> print(json.dumps( 2024-04-25T01:18:35.352 DEBUG:teuthology.orchestra.run.smithi090:> dict([(a, getattr(s, a)) for a in attrs]), 2024-04-25T01:18:35.352 DEBUG:teuthology.orchestra.run.smithi090:> indent=2)) 2024-04-25T01:18:35.352 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-04-25T01:18:35.520 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T01:18:35.521 INFO:teuthology.orchestra.run.smithi090.stdout: "st_mode": 33088, 2024-04-25T01:18:35.521 INFO:teuthology.orchestra.run.smithi090.stdout: "st_ino": 1099511627777, 2024-04-25T01:18:35.521 INFO:teuthology.orchestra.run.smithi090.stdout: "st_dev": 42, 2024-04-25T01:18:35.521 INFO:teuthology.orchestra.run.smithi090.stdout: "st_nlink": 1, 2024-04-25T01:18:35.521 INFO:teuthology.orchestra.run.smithi090.stdout: "st_uid": 0, 2024-04-25T01:18:35.521 INFO:teuthology.orchestra.run.smithi090.stdout: "st_gid": 0, 2024-04-25T01:18:35.521 INFO:teuthology.orchestra.run.smithi090.stdout: "st_size": 6291456, 2024-04-25T01:18:35.521 INFO:teuthology.orchestra.run.smithi090.stdout: "st_atime": 1714007895.0, 2024-04-25T01:18:35.521 INFO:teuthology.orchestra.run.smithi090.stdout: "st_mtime": 1714007895.0, 2024-04-25T01:18:35.521 INFO:teuthology.orchestra.run.smithi090.stdout: "st_ctime": 1714007895.0 2024-04-25T01:18:35.521 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T01:18:35.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:18:35.535 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-25T01:18:35.535 DEBUG:teuthology.orchestra.run.smithi090:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T01:18:35.685 DEBUG:teuthology.orchestra.run.smithi103:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T01:18:35.805 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-25T01:18:35.805 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T01:18:35.805 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:18:35.805 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:18:35.864 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:18:35.864 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:18:35.918 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:18:36.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.135+0000 7f60d0000640 1 Processor -- start 2024-04-25T01:18:36.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.135+0000 7f60d0000640 1 -- start start 2024-04-25T01:18:36.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.137+0000 7f60d0000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f60c8158590 0x7f60c8162450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:36.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.137+0000 7f60d0000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f60c80715f0 0x7f60c8071a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:36.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.137+0000 7f60d0000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60c8162990 con 0x7f60c80cfc90 2024-04-25T01:18:36.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.137+0000 7f60d0000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f60c8158970 con 0x7f60c80715f0 2024-04-25T01:18:36.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.137+0000 7f60d0000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f60c8070c10 con 0x7f60c8158590 2024-04-25T01:18:36.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.137+0000 7f60ccc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f60c80715f0 0x7f60c8071a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:36.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.137+0000 7f60ccc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f60c80715f0 0x7f60c8071a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51430/0 (socket says 172.21.15.90:51430) 2024-04-25T01:18:36.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.137+0000 7f60ccc00640 1 -- 172.21.15.90:0/1572296632 learned_addr learned my addr 172.21.15.90:0/1572296632 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:36.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.137+0000 7f60ccc00640 1 -- 172.21.15.90:0/1572296632 >> v2:172.21.15.103:3300/0 conn(0x7f60c8158590 msgr2=0x7f60c8162450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:36.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.137+0000 7f60cd600640 1 --2- 172.21.15.90:0/1572296632 >> v2:172.21.15.103:3300/0 conn(0x7f60c8158590 0x7f60c8162450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:36.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.138+0000 7f60ccc00640 1 --2- 172.21.15.90:0/1572296632 >> v2:172.21.15.103:3300/0 conn(0x7f60c8158590 0x7f60c8162450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:36.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.138+0000 7f60ccc00640 1 -- 172.21.15.90:0/1572296632 >> v1:172.21.15.90:6789/0 conn(0x7f60c80cfc90 legacy=0x7f60c815e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:36.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.138+0000 7f60ccc00640 1 -- 172.21.15.90:0/1572296632 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60c8070ef0 con 0x7f60c80715f0 2024-04-25T01:18:36.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.138+0000 7f60c7e00640 1 -- 172.21.15.90:0/1572296632 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1769179644 0 0) 0x7f60c0001490 con 0x7f60c80cfc90 2024-04-25T01:18:36.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.138+0000 7f60cd600640 1 --2- 172.21.15.90:0/1572296632 >> v2:172.21.15.103:3300/0 conn(0x7f60c8158590 0x7f60c8162450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:36.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.139+0000 7f60ccc00640 1 --2- 172.21.15.90:0/1572296632 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60c80715f0 0x7f60c8071a70 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f60c80aed80 tx=0x7f60b402a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dffa8926f29e1082 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:36.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.139+0000 7f60c7e00640 1 -- 172.21.15.90:0/1572296632 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b4037070 con 0x7f60c80715f0 2024-04-25T01:18:36.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.139+0000 7f60c7e00640 1 -- 172.21.15.90:0/1572296632 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60b402ae50 con 0x7f60c80715f0 2024-04-25T01:18:36.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.139+0000 7f60c7e00640 1 -- 172.21.15.90:0/1572296632 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b4032860 con 0x7f60c80715f0 2024-04-25T01:18:36.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.139+0000 7f60d0000640 1 -- 172.21.15.90:0/1572296632 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60c80715f0 msgr2=0x7f60c8071a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:36.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.139+0000 7f60d0000640 1 --2- 172.21.15.90:0/1572296632 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60c80715f0 0x7f60c8071a70 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f60c80aed80 tx=0x7f60b402a300 comp rx=0 tx=0).stop 2024-04-25T01:18:36.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.140+0000 7f60d0000640 1 -- 172.21.15.90:0/1572296632 shutdown_connections 2024-04-25T01:18:36.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.140+0000 7f60d0000640 1 --2- 172.21.15.90:0/1572296632 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60c80715f0 0x7f60c8071a70 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:36.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.140+0000 7f60d0000640 1 --2- 172.21.15.90:0/1572296632 >> v2:172.21.15.103:3300/0 conn(0x7f60c8158590 0x7f60c8162450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:36.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.140+0000 7f60d0000640 1 -- 172.21.15.90:0/1572296632 >> 172.21.15.90:0/1572296632 conn(0x7f60c806b320 msgr2=0x7f60c806b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:36.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.140+0000 7f60d0000640 1 -- 172.21.15.90:0/1572296632 shutdown_connections 2024-04-25T01:18:36.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.140+0000 7f60d0000640 1 -- 172.21.15.90:0/1572296632 wait complete. 2024-04-25T01:18:36.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.141+0000 7f60d0000640 1 Processor -- start 2024-04-25T01:18:36.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.142+0000 7f60d0000640 1 -- start start 2024-04-25T01:18:36.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.142+0000 7f60d0000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60c80715f0 0x7f60c8159620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:36.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.142+0000 7f60d0000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60c80cfc90 0x7f60c815ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:36.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.142+0000 7f60d0000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60c8158590 0x7f60c81493e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:36.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.142+0000 7f60d0000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f60c8161f30 con 0x7f60c80715f0 2024-04-25T01:18:36.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.142+0000 7f60d0000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f60c8161c50 con 0x7f60c80cfc90 2024-04-25T01:18:36.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.142+0000 7f60d0000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f60c80d0100 con 0x7f60c8158590 2024-04-25T01:18:36.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.143+0000 7f60cd600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60c80715f0 0x7f60c8159620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:36.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.143+0000 7f60cd600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60c80715f0 0x7f60c8159620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51442/0 (socket says 172.21.15.90:51442) 2024-04-25T01:18:36.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.143+0000 7f60cd600640 1 -- 172.21.15.90:0/1353024400 learned_addr learned my addr 172.21.15.90:0/1353024400 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:36.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.143+0000 7f60cd600640 1 -- 172.21.15.90:0/1353024400 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60c8158590 msgr2=0x7f60c81493e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:36.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.143+0000 7f60cd600640 1 --2- 172.21.15.90:0/1353024400 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60c8158590 0x7f60c81493e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:36.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.143+0000 7f60cd600640 1 -- 172.21.15.90:0/1353024400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60c80cfc90 msgr2=0x7f60c815ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:36.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.143+0000 7f60cd600640 1 --2- 172.21.15.90:0/1353024400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60c80cfc90 0x7f60c815ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:36.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.143+0000 7f60cd600640 1 -- 172.21.15.90:0/1353024400 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60c815f230 con 0x7f60c80715f0 2024-04-25T01:18:36.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.143+0000 7f60cd600640 1 --2- 172.21.15.90:0/1353024400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60c80715f0 0x7f60c8159620 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f60bc00c7f0 tx=0x7f60bc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:36.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.144+0000 7f60c5600640 1 -- 172.21.15.90:0/1353024400 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60bc015070 con 0x7f60c80715f0 2024-04-25T01:18:36.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.144+0000 7f60c5600640 1 -- 172.21.15.90:0/1353024400 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60bc007820 con 0x7f60c80715f0 2024-04-25T01:18:36.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.144+0000 7f60c5600640 1 -- 172.21.15.90:0/1353024400 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60bc007bd0 con 0x7f60c80715f0 2024-04-25T01:18:36.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.144+0000 7f60d0000640 1 -- 172.21.15.90:0/1353024400 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60c81f7640 con 0x7f60c80715f0 2024-04-25T01:18:36.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.144+0000 7f60d0000640 1 -- 172.21.15.90:0/1353024400 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60c81f7c00 con 0x7f60c80715f0 2024-04-25T01:18:36.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.145+0000 7f60d0000640 1 -- 172.21.15.90:0/1353024400 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60c8068190 con 0x7f60c80715f0 2024-04-25T01:18:36.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.146+0000 7f60c5600640 1 -- 172.21.15.90:0/1353024400 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f60bc018070 con 0x7f60c80715f0 2024-04-25T01:18:36.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.147+0000 7f60c5600640 1 --2- 172.21.15.90:0/1353024400 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f60a806be30 0x7f60a806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:36.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.147+0000 7f60c5600640 1 -- 172.21.15.90:0/1353024400 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f60bc08b3e0 con 0x7f60c80715f0 2024-04-25T01:18:36.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.147+0000 7f60ccc00640 1 --2- 172.21.15.90:0/1353024400 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f60a806be30 0x7f60a806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:36.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.148+0000 7f60ccc00640 1 --2- 172.21.15.90:0/1353024400 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f60a806be30 0x7f60a806e2f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f60b4034000 tx=0x7f60b402c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:36.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.151+0000 7f60c5600640 1 -- 172.21.15.90:0/1353024400 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f60bc059910 con 0x7f60c80715f0 2024-04-25T01:18:36.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.354+0000 7f60d0000640 1 -- 172.21.15.90:0/1353024400 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f60c8161a10 con 0x7f60c80715f0 2024-04-25T01:18:36.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.357+0000 7f60c5600640 1 -- 172.21.15.90:0/1353024400 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) ==== 74+0+18947 (secure 0 0 0) 0x7f60bc059210 con 0x7f60c80715f0 2024-04-25T01:18:36.342 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:36.342 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":50,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:18:20.389626+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:18:00.504992+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T01:18:01.714313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"recovery_fs_metadata","create_time":"2024-04-25T01:18:18.384671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.16","mappings":[{"from":4,"to":2}]},{"pgid":"6.20","mappings":[{"from":4,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:6837/431390375":"2024-04-26T01:18:15.602554+0000","172.21.15.90:6836/431390375":"2024-04-26T01:18:15.602554+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:18:36.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.360+0000 7f60d0000640 1 -- 172.21.15.90:0/1353024400 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f60a806be30 msgr2=0x7f60a806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:36.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.360+0000 7f60d0000640 1 --2- 172.21.15.90:0/1353024400 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f60a806be30 0x7f60a806e2f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f60b4034000 tx=0x7f60b402c000 comp rx=0 tx=0).stop 2024-04-25T01:18:36.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.360+0000 7f60d0000640 1 -- 172.21.15.90:0/1353024400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60c80715f0 msgr2=0x7f60c8159620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:36.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.360+0000 7f60d0000640 1 --2- 172.21.15.90:0/1353024400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60c80715f0 0x7f60c8159620 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f60bc00c7f0 tx=0x7f60bc00ccb0 comp rx=0 tx=0).stop 2024-04-25T01:18:36.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.361+0000 7f60d0000640 1 -- 172.21.15.90:0/1353024400 shutdown_connections 2024-04-25T01:18:36.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.361+0000 7f60d0000640 1 --2- 172.21.15.90:0/1353024400 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f60a806be30 0x7f60a806e2f0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:36.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.361+0000 7f60d0000640 1 --2- 172.21.15.90:0/1353024400 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60c8158590 0x7f60c81493e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:36.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.361+0000 7f60d0000640 1 --2- 172.21.15.90:0/1353024400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60c80cfc90 0x7f60c815ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:36.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.361+0000 7f60d0000640 1 --2- 172.21.15.90:0/1353024400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60c80715f0 0x7f60c8159620 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:36.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.361+0000 7f60d0000640 1 -- 172.21.15.90:0/1353024400 >> 172.21.15.90:0/1353024400 conn(0x7f60c806b320 msgr2=0x7f60c806b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:36.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.361+0000 7f60d0000640 1 -- 172.21.15.90:0/1353024400 shutdown_connections 2024-04-25T01:18:36.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.361+0000 7f60d0000640 1 -- 172.21.15.90:0/1353024400 wait complete. 2024-04-25T01:18:36.365 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:16:15.894126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:18:00.504992+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:18:01.714313+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'recovery_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'recovery_fs_metadata', 'create_time': '2024-04-25T01:18:18.384671+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'recovery_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:18:36.366 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:18:36.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.587+0000 7f0f74a00640 1 Processor -- start 2024-04-25T01:18:36.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.587+0000 7f0f74a00640 1 -- start start 2024-04-25T01:18:36.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.588+0000 7f0f74a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0f7015bcd0 0x7f0f700cc230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:36.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.588+0000 7f0f74a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f700b4df0 con 0x7f0f700d03d0 2024-04-25T01:18:36.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.588+0000 7f0f74a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f700b4b10 con 0x7f0f7015cd60 2024-04-25T01:18:36.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.588+0000 7f0f74a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f0f7015c8d0 con 0x7f0f7015bcd0 2024-04-25T01:18:36.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.589+0000 7f0f6f400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0f7015bcd0 0x7f0f700cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:36.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.589+0000 7f0f6f400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0f7015bcd0 0x7f0f700cc230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51448/0 (socket says 172.21.15.90:51448) 2024-04-25T01:18:36.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.589+0000 7f0f6f400640 1 -- 172.21.15.90:0/3177658394 learned_addr learned my addr 172.21.15.90:0/3177658394 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:36.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.589+0000 7f0f6fe00640 1 --1- 172.21.15.90:0/3177658394 >> v1:172.21.15.90:6789/0 conn(0x7f0f700d03d0 0x7f0f7015f6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41160/0 (socket says 172.21.15.90:41160) 2024-04-25T01:18:36.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.589+0000 7f0f6f400640 1 -- 172.21.15.90:0/3177658394 >> v1:172.21.15.103:6789/0 conn(0x7f0f7015cd60 legacy=0x7f0f700cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:36.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.589+0000 7f0f6f400640 1 -- 172.21.15.90:0/3177658394 >> v1:172.21.15.90:6789/0 conn(0x7f0f700d03d0 legacy=0x7f0f7015f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:36.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.589+0000 7f0f6f400640 1 -- 172.21.15.90:0/3177658394 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f7015c280 con 0x7f0f7015bcd0 2024-04-25T01:18:36.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.590+0000 7f0f6f400640 1 --2- 172.21.15.90:0/3177658394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f7015bcd0 0x7f0f700cc230 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f0f600098a0 tx=0x7f0f6002a2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6afa0e01e174c88e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:36.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.591+0000 7f0f6e000640 1 -- 172.21.15.90:0/3177658394 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f60037070 con 0x7f0f7015bcd0 2024-04-25T01:18:36.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.591+0000 7f0f6e000640 1 -- 172.21.15.90:0/3177658394 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f60032440 con 0x7f0f7015bcd0 2024-04-25T01:18:36.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.591+0000 7f0f6e000640 1 -- 172.21.15.90:0/3177658394 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f60032810 con 0x7f0f7015bcd0 2024-04-25T01:18:36.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.591+0000 7f0f74a00640 1 -- 172.21.15.90:0/3177658394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f7015bcd0 msgr2=0x7f0f700cc230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:36.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.591+0000 7f0f74a00640 1 --2- 172.21.15.90:0/3177658394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f7015bcd0 0x7f0f700cc230 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f0f600098a0 tx=0x7f0f6002a2e0 comp rx=0 tx=0).stop 2024-04-25T01:18:36.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.591+0000 7f0f74a00640 1 -- 172.21.15.90:0/3177658394 shutdown_connections 2024-04-25T01:18:36.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.591+0000 7f0f74a00640 1 --2- 172.21.15.90:0/3177658394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f7015bcd0 0x7f0f700cc230 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:36.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.591+0000 7f0f74a00640 1 -- 172.21.15.90:0/3177658394 >> 172.21.15.90:0/3177658394 conn(0x7f0f7006b560 msgr2=0x7f0f7006b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:36.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.591+0000 7f0f74a00640 1 -- 172.21.15.90:0/3177658394 shutdown_connections 2024-04-25T01:18:36.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.591+0000 7f0f74a00640 1 -- 172.21.15.90:0/3177658394 wait complete. 2024-04-25T01:18:36.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.593+0000 7f0f74a00640 1 Processor -- start 2024-04-25T01:18:36.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.593+0000 7f0f74a00640 1 -- start start 2024-04-25T01:18:36.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.594+0000 7f0f74a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f700d03d0 0x7f0f7013ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:36.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.594+0000 7f0f74a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f7015bcd0 0x7f0f700c9140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:36.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.594+0000 7f0f74a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f7015cd60 0x7f0f70071c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:36.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.594+0000 7f0f74a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7006e590 con 0x7f0f700d03d0 2024-04-25T01:18:36.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.594+0000 7f0f74a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0f700cc770 con 0x7f0f7015bcd0 2024-04-25T01:18:36.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.594+0000 7f0f74a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0f700b4790 con 0x7f0f7015cd60 2024-04-25T01:18:36.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.594+0000 7f0f6f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f700d03d0 0x7f0f7013ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:36.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.594+0000 7f0f6f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f700d03d0 0x7f0f7013ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51454/0 (socket says 172.21.15.90:51454) 2024-04-25T01:18:36.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.594+0000 7f0f6f400640 1 -- 172.21.15.90:0/2752214412 learned_addr learned my addr 172.21.15.90:0/2752214412 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:36.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.595+0000 7f0f6f400640 1 -- 172.21.15.90:0/2752214412 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f7015cd60 msgr2=0x7f0f70071c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:36.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.595+0000 7f0f6f400640 1 --2- 172.21.15.90:0/2752214412 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f7015cd60 0x7f0f70071c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:36.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.595+0000 7f0f6f400640 1 -- 172.21.15.90:0/2752214412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f7015bcd0 msgr2=0x7f0f700c9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:36.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.595+0000 7f0f6f400640 1 --2- 172.21.15.90:0/2752214412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f7015bcd0 0x7f0f700c9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:36.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.595+0000 7f0f6f400640 1 -- 172.21.15.90:0/2752214412 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f7013f530 con 0x7f0f700d03d0 2024-04-25T01:18:36.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.595+0000 7f0f6f400640 1 --2- 172.21.15.90:0/2752214412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f700d03d0 0x7f0f7013ee50 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f0f600098a0 tx=0x7f0f600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:36.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.595+0000 7f0f67400640 1 -- 172.21.15.90:0/2752214412 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f60037040 con 0x7f0f700d03d0 2024-04-25T01:18:36.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.595+0000 7f0f67400640 1 -- 172.21.15.90:0/2752214412 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f60004230 con 0x7f0f700d03d0 2024-04-25T01:18:36.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.595+0000 7f0f74a00640 1 -- 172.21.15.90:0/2752214412 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f7013f7d0 con 0x7f0f700d03d0 2024-04-25T01:18:36.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.595+0000 7f0f67400640 1 -- 172.21.15.90:0/2752214412 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f60003610 con 0x7f0f700d03d0 2024-04-25T01:18:36.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.596+0000 7f0f74a00640 1 -- 172.21.15.90:0/2752214412 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f7013fd90 con 0x7f0f700d03d0 2024-04-25T01:18:36.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.597+0000 7f0f74a00640 1 -- 172.21.15.90:0/2752214412 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f7004f500 con 0x7f0f700d03d0 2024-04-25T01:18:36.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.598+0000 7f0f67400640 1 -- 172.21.15.90:0/2752214412 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0f6003a070 con 0x7f0f700d03d0 2024-04-25T01:18:36.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.598+0000 7f0f67400640 1 --2- 172.21.15.90:0/2752214412 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0f5006be30 0x7f0f5006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:36.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.599+0000 7f0f67400640 1 -- 172.21.15.90:0/2752214412 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f0f600ab8d0 con 0x7f0f700d03d0 2024-04-25T01:18:36.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.599+0000 7f0f6ea00640 1 --2- 172.21.15.90:0/2752214412 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0f5006be30 0x7f0f5006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:36.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.600+0000 7f0f6ea00640 1 --2- 172.21.15.90:0/2752214412 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0f5006be30 0x7f0f5006e2f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c006fd0 tx=0x7f0f5c006bc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:36.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.603+0000 7f0f67400640 1 -- 172.21.15.90:0/2752214412 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0f60041020 con 0x7f0f700d03d0 2024-04-25T01:18:36.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.801+0000 7f0f74a00640 1 -- 172.21.15.90:0/2752214412 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0f700cf030 con 0x7f0f700d03d0 2024-04-25T01:18:36.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.802+0000 7f0f67400640 1 -- 172.21.15.90:0/2752214412 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) ==== 93+0+10 (secure 0 0 0) 0x7f0f60079f30 con 0x7f0f700d03d0 2024-04-25T01:18:36.786 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T01:18:36.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.805+0000 7f0f74a00640 1 -- 172.21.15.90:0/2752214412 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0f5006be30 msgr2=0x7f0f5006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:36.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.805+0000 7f0f74a00640 1 --2- 172.21.15.90:0/2752214412 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0f5006be30 0x7f0f5006e2f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c006fd0 tx=0x7f0f5c006bc0 comp rx=0 tx=0).stop 2024-04-25T01:18:36.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.805+0000 7f0f74a00640 1 -- 172.21.15.90:0/2752214412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f700d03d0 msgr2=0x7f0f7013ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:36.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.805+0000 7f0f74a00640 1 --2- 172.21.15.90:0/2752214412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f700d03d0 0x7f0f7013ee50 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f0f600098a0 tx=0x7f0f600046c0 comp rx=0 tx=0).stop 2024-04-25T01:18:36.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.805+0000 7f0f74a00640 1 -- 172.21.15.90:0/2752214412 shutdown_connections 2024-04-25T01:18:36.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.805+0000 7f0f74a00640 1 --2- 172.21.15.90:0/2752214412 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0f5006be30 0x7f0f5006e2f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:36.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.805+0000 7f0f74a00640 1 --2- 172.21.15.90:0/2752214412 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0f7015cd60 0x7f0f70071c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:36.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.805+0000 7f0f74a00640 1 --2- 172.21.15.90:0/2752214412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f7015bcd0 0x7f0f700c9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:36.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.805+0000 7f0f74a00640 1 --2- 172.21.15.90:0/2752214412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f700d03d0 0x7f0f7013ee50 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:36.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.805+0000 7f0f74a00640 1 -- 172.21.15.90:0/2752214412 >> 172.21.15.90:0/2752214412 conn(0x7f0f7006b560 msgr2=0x7f0f70158120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:36.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.805+0000 7f0f74a00640 1 -- 172.21.15.90:0/2752214412 shutdown_connections 2024-04-25T01:18:36.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:36.805+0000 7f0f74a00640 1 -- 172.21.15.90:0/2752214412 wait complete. 2024-04-25T01:18:36.809 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:18:37.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.025+0000 7fc10f200640 1 Processor -- start 2024-04-25T01:18:37.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.025+0000 7fc10f200640 1 -- start start 2024-04-25T01:18:37.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.026+0000 7fc10f200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fc11015bcd0 0x7fc1100cc230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:37.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.027+0000 7fc10f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc11015cd60 0x7fc1100cfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:37.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.027+0000 7fc10f200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc1100d0200 0x7fc11006c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:37.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.027+0000 7fc10f200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fc11015c8d0 con 0x7fc1100d0200 2024-04-25T01:18:37.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.027+0000 7fc10f200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fc1100b4df0 con 0x7fc11015bcd0 2024-04-25T01:18:37.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.027+0000 7fc10f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc1100b4b10 con 0x7fc11015cd60 2024-04-25T01:18:37.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.027+0000 7fc10e800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc1100d0200 0x7fc11006c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:37.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.027+0000 7fc10e800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc1100d0200 0x7fc11006c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51466/0 (socket says 172.21.15.90:51466) 2024-04-25T01:18:37.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.027+0000 7fc10e800640 1 -- 172.21.15.90:0/4104837298 learned_addr learned my addr 172.21.15.90:0/4104837298 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:37.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.027+0000 7fc10de00640 1 --2- 172.21.15.90:0/4104837298 >> v2:172.21.15.103:3300/0 conn(0x7fc11015bcd0 0x7fc1100cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:37.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.027+0000 7fc10e800640 1 -- 172.21.15.90:0/4104837298 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc11015cd60 msgr2=0x7fc1100cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:37.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.027+0000 7fc10d400640 1 --2- 172.21.15.90:0/4104837298 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc11015cd60 0x7fc1100cfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:37.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.028+0000 7fc10e800640 1 --2- 172.21.15.90:0/4104837298 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc11015cd60 0x7fc1100cfcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:37.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.028+0000 7fc10e800640 1 -- 172.21.15.90:0/4104837298 >> v2:172.21.15.103:3300/0 conn(0x7fc11015bcd0 msgr2=0x7fc1100cc230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:37.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.028+0000 7fc10e800640 1 --2- 172.21.15.90:0/4104837298 >> v2:172.21.15.103:3300/0 conn(0x7fc11015bcd0 0x7fc1100cc230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:37.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.028+0000 7fc10e800640 1 -- 172.21.15.90:0/4104837298 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc11015c280 con 0x7fc1100d0200 2024-04-25T01:18:37.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.029+0000 7fc10e800640 1 --2- 172.21.15.90:0/4104837298 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1100d0200 0x7fc11006c750 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fc11006da10 tx=0x7fc10802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5262b904d538658d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:37.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.029+0000 7fc10ca00640 1 -- 172.21.15.90:0/4104837298 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc108036070 con 0x7fc1100d0200 2024-04-25T01:18:37.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.029+0000 7fc10ca00640 1 -- 172.21.15.90:0/4104837298 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc108031600 con 0x7fc1100d0200 2024-04-25T01:18:37.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.029+0000 7fc10ca00640 1 -- 172.21.15.90:0/4104837298 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1080319d0 con 0x7fc1100d0200 2024-04-25T01:18:37.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.030+0000 7fc10f200640 1 -- 172.21.15.90:0/4104837298 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1100d0200 msgr2=0x7fc11006c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:37.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.030+0000 7fc10f200640 1 --2- 172.21.15.90:0/4104837298 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1100d0200 0x7fc11006c750 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fc11006da10 tx=0x7fc10802a6a0 comp rx=0 tx=0).stop 2024-04-25T01:18:37.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.030+0000 7fc10f200640 1 -- 172.21.15.90:0/4104837298 shutdown_connections 2024-04-25T01:18:37.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.030+0000 7fc10f200640 1 --2- 172.21.15.90:0/4104837298 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc11015cd60 0x7fc1100cfcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:37.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.030+0000 7fc10f200640 1 --2- 172.21.15.90:0/4104837298 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1100d0200 0x7fc11006c750 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:37.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.030+0000 7fc10f200640 1 --2- 172.21.15.90:0/4104837298 >> v2:172.21.15.103:3300/0 conn(0x7fc11015bcd0 0x7fc1100cc230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:37.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.030+0000 7fc10f200640 1 -- 172.21.15.90:0/4104837298 >> 172.21.15.90:0/4104837298 conn(0x7fc11006b560 msgr2=0x7fc11006b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:37.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.030+0000 7fc10f200640 1 -- 172.21.15.90:0/4104837298 shutdown_connections 2024-04-25T01:18:37.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.030+0000 7fc10f200640 1 -- 172.21.15.90:0/4104837298 wait complete. 2024-04-25T01:18:37.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.031+0000 7fc10f200640 1 Processor -- start 2024-04-25T01:18:37.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.032+0000 7fc10f200640 1 -- start start 2024-04-25T01:18:37.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.032+0000 7fc10f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1100d0200 0x7fc110144f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:37.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.032+0000 7fc10f200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc11015bcd0 0x7fc1101463c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:37.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.032+0000 7fc10f200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc11015cd60 0x7fc1101504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:37.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.032+0000 7fc10f200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc11006e590 con 0x7fc11015bcd0 2024-04-25T01:18:37.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.032+0000 7fc10f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc1100cc770 con 0x7fc1100d0200 2024-04-25T01:18:37.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.032+0000 7fc10f200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fc1100b4790 con 0x7fc11015cd60 2024-04-25T01:18:37.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.033+0000 7fc10d400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc11015bcd0 0x7fc1101463c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:37.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.033+0000 7fc10d400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc11015bcd0 0x7fc1101463c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51480/0 (socket says 172.21.15.90:51480) 2024-04-25T01:18:37.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.033+0000 7fc10d400640 1 -- 172.21.15.90:0/195546289 learned_addr learned my addr 172.21.15.90:0/195546289 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:37.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.033+0000 7fc10de00640 1 --2- 172.21.15.90:0/195546289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1100d0200 0x7fc110144f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:37.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.033+0000 7fc10e800640 1 --2- 172.21.15.90:0/195546289 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc11015cd60 0x7fc1101504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:37.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.033+0000 7fc10d400640 1 -- 172.21.15.90:0/195546289 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc11015cd60 msgr2=0x7fc1101504a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:37.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.033+0000 7fc10d400640 1 --2- 172.21.15.90:0/195546289 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc11015cd60 0x7fc1101504a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:37.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.033+0000 7fc10d400640 1 -- 172.21.15.90:0/195546289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1100d0200 msgr2=0x7fc110144f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:37.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.033+0000 7fc10d400640 1 --2- 172.21.15.90:0/195546289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1100d0200 0x7fc110144f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:37.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.033+0000 7fc10d400640 1 -- 172.21.15.90:0/195546289 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1101482e0 con 0x7fc11015bcd0 2024-04-25T01:18:37.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.033+0000 7fc10de00640 1 --2- 172.21.15.90:0/195546289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1100d0200 0x7fc110144f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:37.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.033+0000 7fc10d400640 1 --2- 172.21.15.90:0/195546289 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc11015bcd0 0x7fc1101463c0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc00c820 tx=0x7fc0fc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:37.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.034+0000 7fc106000640 1 -- 172.21.15.90:0/195546289 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc015070 con 0x7fc11015bcd0 2024-04-25T01:18:37.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.034+0000 7fc106000640 1 -- 172.21.15.90:0/195546289 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0fc004020 con 0x7fc11015bcd0 2024-04-25T01:18:37.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.034+0000 7fc106000640 1 -- 172.21.15.90:0/195546289 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc0043f0 con 0x7fc11015bcd0 2024-04-25T01:18:37.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.034+0000 7fc10f200640 1 -- 172.21.15.90:0/195546289 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1101485c0 con 0x7fc11015bcd0 2024-04-25T01:18:37.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.034+0000 7fc10f200640 1 -- 172.21.15.90:0/195546289 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc110148b80 con 0x7fc11015bcd0 2024-04-25T01:18:37.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.035+0000 7fc10f200640 1 -- 172.21.15.90:0/195546289 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc11004f500 con 0x7fc11015bcd0 2024-04-25T01:18:37.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.036+0000 7fc106000640 1 -- 172.21.15.90:0/195546289 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc0fc007830 con 0x7fc11015bcd0 2024-04-25T01:18:37.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.037+0000 7fc106000640 1 --2- 172.21.15.90:0/195546289 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc0f006be30 0x7fc0f006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:37.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.037+0000 7fc106000640 1 -- 172.21.15.90:0/195546289 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7fc0fc08b080 con 0x7fc11015bcd0 2024-04-25T01:18:37.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.037+0000 7fc10de00640 1 --2- 172.21.15.90:0/195546289 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc0f006be30 0x7fc0f006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:37.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.038+0000 7fc10de00640 1 --2- 172.21.15.90:0/195546289 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc0f006be30 0x7fc0f006e2f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc100008890 tx=0x7fc100008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:37.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.041+0000 7fc106000640 1 -- 172.21.15.90:0/195546289 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc0fc0595b0 con 0x7fc11015bcd0 2024-04-25T01:18:37.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.240+0000 7fc10f200640 1 -- 172.21.15.90:0/195546289 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc11006e140 con 0x7fc11015bcd0 2024-04-25T01:18:37.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.241+0000 7fc106000640 1 -- 172.21.15.90:0/195546289 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v50) ==== 104+0+11 (secure 0 0 0) 0x7fc0fc058eb0 con 0x7fc11015bcd0 2024-04-25T01:18:37.226 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:18:37.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.244+0000 7fc10f200640 1 -- 172.21.15.90:0/195546289 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc0f006be30 msgr2=0x7fc0f006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:37.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.244+0000 7fc10f200640 1 --2- 172.21.15.90:0/195546289 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc0f006be30 0x7fc0f006e2f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc100008890 tx=0x7fc100008040 comp rx=0 tx=0).stop 2024-04-25T01:18:37.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.244+0000 7fc10f200640 1 -- 172.21.15.90:0/195546289 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc11015bcd0 msgr2=0x7fc1101463c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:37.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.244+0000 7fc10f200640 1 --2- 172.21.15.90:0/195546289 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc11015bcd0 0x7fc1101463c0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc00c820 tx=0x7fc0fc00cce0 comp rx=0 tx=0).stop 2024-04-25T01:18:37.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.244+0000 7fc10f200640 1 -- 172.21.15.90:0/195546289 shutdown_connections 2024-04-25T01:18:37.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.244+0000 7fc10f200640 1 --2- 172.21.15.90:0/195546289 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fc0f006be30 0x7fc0f006e2f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:37.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.244+0000 7fc10f200640 1 --2- 172.21.15.90:0/195546289 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fc11015cd60 0x7fc1101504a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:37.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.244+0000 7fc10f200640 1 --2- 172.21.15.90:0/195546289 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc11015bcd0 0x7fc1101463c0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:37.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.244+0000 7fc10f200640 1 --2- 172.21.15.90:0/195546289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1100d0200 0x7fc110144f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:37.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.244+0000 7fc10f200640 1 -- 172.21.15.90:0/195546289 >> 172.21.15.90:0/195546289 conn(0x7fc11006b560 msgr2=0x7fc11014e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:37.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.244+0000 7fc10f200640 1 -- 172.21.15.90:0/195546289 shutdown_connections 2024-04-25T01:18:37.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.244+0000 7fc10f200640 1 -- 172.21.15.90:0/195546289 wait complete. 2024-04-25T01:18:37.247 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:18:37.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.470+0000 7f0ba1e00640 1 Processor -- start 2024-04-25T01:18:37.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.470+0000 7f0ba1e00640 1 -- start start 2024-04-25T01:18:37.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.472+0000 7f0ba1e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0b9c13ef30 0x7f0b9c141570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:37.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.472+0000 7f0ba1e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0b9c0ce770 0x7f0b9c071410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:37.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.472+0000 7f0ba1e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b9c0ced40 con 0x7f0b9c141ab0 2024-04-25T01:18:37.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.472+0000 7f0ba1e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f0b9c1423d0 con 0x7f0b9c13ef30 2024-04-25T01:18:37.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.472+0000 7f0ba1e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f0b9c070f80 con 0x7f0b9c0ce770 2024-04-25T01:18:37.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.472+0000 7f0b9aa00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0b9c13ef30 0x7f0b9c141570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:37.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.472+0000 7f0b9aa00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0b9c13ef30 0x7f0b9c141570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51482/0 (socket says 172.21.15.90:51482) 2024-04-25T01:18:37.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.472+0000 7f0b9aa00640 1 -- 172.21.15.90:0/2062404885 learned_addr learned my addr 172.21.15.90:0/2062404885 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:37.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.473+0000 7f0b9aa00640 1 -- 172.21.15.90:0/2062404885 >> v2:172.21.15.103:3300/0 conn(0x7f0b9c0ce770 msgr2=0x7f0b9c071410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:37.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.473+0000 7f0b9aa00640 1 --2- 172.21.15.90:0/2062404885 >> v2:172.21.15.103:3300/0 conn(0x7f0b9c0ce770 0x7f0b9c071410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:37.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.473+0000 7f0b9aa00640 1 -- 172.21.15.90:0/2062404885 >> v1:172.21.15.103:6789/0 conn(0x7f0b9c141ab0 legacy=0x7f0b9c0ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:37.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.473+0000 7f0b9aa00640 1 -- 172.21.15.90:0/2062404885 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b9c0cf020 con 0x7f0b9c13ef30 2024-04-25T01:18:37.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.474+0000 7f0b9aa00640 1 --2- 172.21.15.90:0/2062404885 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b9c13ef30 0x7f0b9c141570 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c160aa0 tx=0x7f0b8c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=636ad910e9f7de7d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:37.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.474+0000 7f0b9a000640 1 -- 172.21.15.90:0/2062404885 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c037070 con 0x7f0b9c13ef30 2024-04-25T01:18:37.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.474+0000 7f0b9a000640 1 -- 172.21.15.90:0/2062404885 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b8c032440 con 0x7f0b9c13ef30 2024-04-25T01:18:37.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.474+0000 7f0b9a000640 1 -- 172.21.15.90:0/2062404885 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c032810 con 0x7f0b9c13ef30 2024-04-25T01:18:37.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.475+0000 7f0ba1e00640 1 -- 172.21.15.90:0/2062404885 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b9c13ef30 msgr2=0x7f0b9c141570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:37.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.475+0000 7f0ba1e00640 1 --2- 172.21.15.90:0/2062404885 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b9c13ef30 0x7f0b9c141570 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c160aa0 tx=0x7f0b8c02a230 comp rx=0 tx=0).stop 2024-04-25T01:18:37.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.475+0000 7f0ba1e00640 1 -- 172.21.15.90:0/2062404885 shutdown_connections 2024-04-25T01:18:37.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.475+0000 7f0ba1e00640 1 --2- 172.21.15.90:0/2062404885 >> v2:172.21.15.103:3300/0 conn(0x7f0b9c0ce770 0x7f0b9c071410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:37.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.475+0000 7f0ba1e00640 1 --2- 172.21.15.90:0/2062404885 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b9c13ef30 0x7f0b9c141570 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:37.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.475+0000 7f0ba1e00640 1 -- 172.21.15.90:0/2062404885 >> 172.21.15.90:0/2062404885 conn(0x7f0b9c0b17b0 msgr2=0x7f0b9c0b1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:37.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.475+0000 7f0ba1e00640 1 -- 172.21.15.90:0/2062404885 shutdown_connections 2024-04-25T01:18:37.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.475+0000 7f0ba1e00640 1 -- 172.21.15.90:0/2062404885 wait complete. 2024-04-25T01:18:37.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.477+0000 7f0ba1e00640 1 Processor -- start 2024-04-25T01:18:37.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.477+0000 7f0ba1e00640 1 -- start start 2024-04-25T01:18:37.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.478+0000 7f0ba1e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b9c0ce770 0x7f0b9c13a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:37.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.478+0000 7f0ba1e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b9c13ef30 0x7f0b9c0aea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:37.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.478+0000 7f0ba1e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b9c141ab0 0x7f0b9c150e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:37.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.478+0000 7f0ba1e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0b9c071950 con 0x7f0b9c0ce770 2024-04-25T01:18:37.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.478+0000 7f0ba1e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0b9c070c10 con 0x7f0b9c141ab0 2024-04-25T01:18:37.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.478+0000 7f0ba1e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0b9c071ad0 con 0x7f0b9c13ef30 2024-04-25T01:18:37.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.478+0000 7f0b9aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b9c0ce770 0x7f0b9c13a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:37.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.478+0000 7f0b9aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b9c0ce770 0x7f0b9c13a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51498/0 (socket says 172.21.15.90:51498) 2024-04-25T01:18:37.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.478+0000 7f0b9aa00640 1 -- 172.21.15.90:0/2348066184 learned_addr learned my addr 172.21.15.90:0/2348066184 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:37.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.478+0000 7f0b9aa00640 1 -- 172.21.15.90:0/2348066184 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b9c13ef30 msgr2=0x7f0b9c0aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:37.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.478+0000 7f0b9aa00640 1 --2- 172.21.15.90:0/2348066184 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b9c13ef30 0x7f0b9c0aea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:37.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.478+0000 7f0b9aa00640 1 -- 172.21.15.90:0/2348066184 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b9c141ab0 msgr2=0x7f0b9c150e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:37.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.478+0000 7f0b9aa00640 1 --2- 172.21.15.90:0/2348066184 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b9c141ab0 0x7f0b9c150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:37.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.478+0000 7f0b9aa00640 1 -- 172.21.15.90:0/2348066184 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b9c13aa00 con 0x7f0b9c0ce770 2024-04-25T01:18:37.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.479+0000 7f0b9aa00640 1 --2- 172.21.15.90:0/2348066184 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b9c0ce770 0x7f0b9c13a4c0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c000c00 tx=0x7f0b8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:37.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.479+0000 7f0b93e00640 1 -- 172.21.15.90:0/2348066184 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c041070 con 0x7f0b9c0ce770 2024-04-25T01:18:37.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.479+0000 7f0b93e00640 1 -- 172.21.15.90:0/2348066184 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b8c03c890 con 0x7f0b9c0ce770 2024-04-25T01:18:37.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.479+0000 7f0ba1e00640 1 -- 172.21.15.90:0/2348066184 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b9c0ae280 con 0x7f0b9c0ce770 2024-04-25T01:18:37.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.479+0000 7f0b93e00640 1 -- 172.21.15.90:0/2348066184 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c037040 con 0x7f0b9c0ce770 2024-04-25T01:18:37.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.479+0000 7f0ba1e00640 1 -- 172.21.15.90:0/2348066184 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b9c1428e0 con 0x7f0b9c0ce770 2024-04-25T01:18:37.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.480+0000 7f0ba1e00640 1 -- 172.21.15.90:0/2348066184 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b9c04f5a0 con 0x7f0b9c0ce770 2024-04-25T01:18:37.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.481+0000 7f0b93e00640 1 -- 172.21.15.90:0/2348066184 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0b8c044070 con 0x7f0b9c0ce770 2024-04-25T01:18:37.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.482+0000 7f0b93e00640 1 --2- 172.21.15.90:0/2348066184 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0b7806be30 0x7f0b7806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:37.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.483+0000 7f0b93e00640 1 -- 172.21.15.90:0/2348066184 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f0b8c0abec0 con 0x7f0b9c0ce770 2024-04-25T01:18:37.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.483+0000 7f0b92000640 1 --2- 172.21.15.90:0/2348066184 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0b7806be30 0x7f0b7806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:37.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.483+0000 7f0b92000640 1 --2- 172.21.15.90:0/2348066184 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0b7806be30 0x7f0b7806e2f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c0af5f0 tx=0x7f0b84006ba0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:37.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.487+0000 7f0b93e00640 1 -- 172.21.15.90:0/2348066184 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0b9c13aa00 con 0x7f0b9c0ce770 2024-04-25T01:18:37.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.703+0000 7f0ba1e00640 1 -- 172.21.15.90:0/2348066184 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0b9c06ec40 con 0x7f0b9c0ce770 2024-04-25T01:18:37.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.704+0000 7f0b93e00640 1 -- 172.21.15.90:0/2348066184 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v50) ==== 100+0+11 (secure 0 0 0) 0x7f0b8c07a5b0 con 0x7f0b9c0ce770 2024-04-25T01:18:37.688 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-25T01:18:37.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.707+0000 7f0ba1e00640 1 -- 172.21.15.90:0/2348066184 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0b7806be30 msgr2=0x7f0b7806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:37.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.707+0000 7f0ba1e00640 1 --2- 172.21.15.90:0/2348066184 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0b7806be30 0x7f0b7806e2f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c0af5f0 tx=0x7f0b84006ba0 comp rx=0 tx=0).stop 2024-04-25T01:18:37.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.707+0000 7f0ba1e00640 1 -- 172.21.15.90:0/2348066184 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b9c0ce770 msgr2=0x7f0b9c13a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:37.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.707+0000 7f0ba1e00640 1 --2- 172.21.15.90:0/2348066184 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b9c0ce770 0x7f0b9c13a4c0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c000c00 tx=0x7f0b8c0046c0 comp rx=0 tx=0).stop 2024-04-25T01:18:37.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.707+0000 7f0ba1e00640 1 -- 172.21.15.90:0/2348066184 shutdown_connections 2024-04-25T01:18:37.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.707+0000 7f0ba1e00640 1 --2- 172.21.15.90:0/2348066184 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0b7806be30 0x7f0b7806e2f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:37.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.707+0000 7f0ba1e00640 1 --2- 172.21.15.90:0/2348066184 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b9c141ab0 0x7f0b9c150e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:37.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.707+0000 7f0ba1e00640 1 --2- 172.21.15.90:0/2348066184 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0b9c13ef30 0x7f0b9c0aea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:37.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.707+0000 7f0ba1e00640 1 --2- 172.21.15.90:0/2348066184 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b9c0ce770 0x7f0b9c13a4c0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:37.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.707+0000 7f0ba1e00640 1 -- 172.21.15.90:0/2348066184 >> 172.21.15.90:0/2348066184 conn(0x7f0b9c0b17b0 msgr2=0x7f0b9c06c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:37.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.707+0000 7f0ba1e00640 1 -- 172.21.15.90:0/2348066184 shutdown_connections 2024-04-25T01:18:37.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.707+0000 7f0ba1e00640 1 -- 172.21.15.90:0/2348066184 wait complete. 2024-04-25T01:18:37.710 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get recovery_fs_metadata pg_num 2024-04-25T01:18:37.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.927+0000 7f91c3400640 1 Processor -- start 2024-04-25T01:18:37.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.928+0000 7f91c3400640 1 -- start start 2024-04-25T01:18:37.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.929+0000 7f91c3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f91bc13ef30 0x7f91bc141570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:37.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.929+0000 7f91c3400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91bc141ab0 0x7f91bc0ce060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:37.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.929+0000 7f91c3400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91bc0ced40 con 0x7f91bc0ce5a0 2024-04-25T01:18:37.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.929+0000 7f91c3400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f91bc1423d0 con 0x7f91bc13ef30 2024-04-25T01:18:37.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.929+0000 7f91c3400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f91bc070f80 con 0x7f91bc141ab0 2024-04-25T01:18:37.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.930+0000 7f91c0a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f91bc13ef30 0x7f91bc141570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:37.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.930+0000 7f91c0a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f91bc13ef30 0x7f91bc141570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51504/0 (socket says 172.21.15.90:51504) 2024-04-25T01:18:37.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.930+0000 7f91c0a00640 1 -- 172.21.15.90:0/4026605443 learned_addr learned my addr 172.21.15.90:0/4026605443 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:37.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.930+0000 7f91c0a00640 1 -- 172.21.15.90:0/4026605443 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91bc141ab0 msgr2=0x7f91bc0ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:37.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.930+0000 7f91c0a00640 1 --2- 172.21.15.90:0/4026605443 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91bc141ab0 0x7f91bc0ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:37.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.930+0000 7f91c0a00640 1 -- 172.21.15.90:0/4026605443 >> v1:172.21.15.103:6789/0 conn(0x7f91bc0ce5a0 legacy=0x7f91bc15fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:37.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.930+0000 7f91c0a00640 1 -- 172.21.15.90:0/4026605443 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91bc0cf020 con 0x7f91bc13ef30 2024-04-25T01:18:37.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.931+0000 7f91c0a00640 1 --2- 172.21.15.90:0/4026605443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91bc13ef30 0x7f91bc141570 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f91b0009870 tx=0x7f91b002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9a7404960419115 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:37.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.932+0000 7f91bb400640 1 -- 172.21.15.90:0/4026605443 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b0037070 con 0x7f91bc13ef30 2024-04-25T01:18:37.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.932+0000 7f91bb400640 1 -- 172.21.15.90:0/4026605443 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91b0032440 con 0x7f91bc13ef30 2024-04-25T01:18:37.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.932+0000 7f91bb400640 1 -- 172.21.15.90:0/4026605443 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b0032810 con 0x7f91bc13ef30 2024-04-25T01:18:37.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.932+0000 7f91c3400640 1 -- 172.21.15.90:0/4026605443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91bc13ef30 msgr2=0x7f91bc141570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:37.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.932+0000 7f91c3400640 1 --2- 172.21.15.90:0/4026605443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91bc13ef30 0x7f91bc141570 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f91b0009870 tx=0x7f91b002a210 comp rx=0 tx=0).stop 2024-04-25T01:18:37.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.932+0000 7f91c3400640 1 -- 172.21.15.90:0/4026605443 shutdown_connections 2024-04-25T01:18:37.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.932+0000 7f91c3400640 1 --2- 172.21.15.90:0/4026605443 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91bc141ab0 0x7f91bc0ce060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:37.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.932+0000 7f91c3400640 1 --2- 172.21.15.90:0/4026605443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91bc13ef30 0x7f91bc141570 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:37.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.932+0000 7f91c3400640 1 -- 172.21.15.90:0/4026605443 >> 172.21.15.90:0/4026605443 conn(0x7f91bc0b17b0 msgr2=0x7f91bc0b1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:37.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.932+0000 7f91c3400640 1 -- 172.21.15.90:0/4026605443 shutdown_connections 2024-04-25T01:18:37.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.933+0000 7f91c3400640 1 -- 172.21.15.90:0/4026605443 wait complete. 2024-04-25T01:18:37.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.935+0000 7f91c3400640 1 Processor -- start 2024-04-25T01:18:37.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.935+0000 7f91c3400640 1 -- start start 2024-04-25T01:18:37.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.936+0000 7f91c3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91bc0ce5a0 0x7f91bc139dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:37.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.936+0000 7f91c3400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f91bc13ef30 0x7f91bc13d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:37.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.936+0000 7f91c3400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91bc141ab0 0x7f91bc13d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:37.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.936+0000 7f91c3400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f91bc0cea30 con 0x7f91bc0ce5a0 2024-04-25T01:18:37.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.936+0000 7f91c3400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f91bc070c10 con 0x7f91bc13ef30 2024-04-25T01:18:37.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.936+0000 7f91c3400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f91bc072420 con 0x7f91bc141ab0 2024-04-25T01:18:37.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.936+0000 7f91c0a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91bc0ce5a0 0x7f91bc139dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:37.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.936+0000 7f91c0a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91bc0ce5a0 0x7f91bc139dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51516/0 (socket says 172.21.15.90:51516) 2024-04-25T01:18:37.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.936+0000 7f91c0a00640 1 -- 172.21.15.90:0/1811345630 learned_addr learned my addr 172.21.15.90:0/1811345630 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:37.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.937+0000 7f91bbe00640 1 --2- 172.21.15.90:0/1811345630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f91bc13ef30 0x7f91bc13d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:37.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.937+0000 7f91c0a00640 1 -- 172.21.15.90:0/1811345630 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91bc141ab0 msgr2=0x7f91bc13d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:37.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.937+0000 7f91c0a00640 1 --2- 172.21.15.90:0/1811345630 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91bc141ab0 0x7f91bc13d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:37.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.937+0000 7f91c0a00640 1 -- 172.21.15.90:0/1811345630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f91bc13ef30 msgr2=0x7f91bc13d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:37.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.937+0000 7f91c0a00640 1 --2- 172.21.15.90:0/1811345630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f91bc13ef30 0x7f91bc13d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:37.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.937+0000 7f91c0a00640 1 -- 172.21.15.90:0/1811345630 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91bc144690 con 0x7f91bc0ce5a0 2024-04-25T01:18:37.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.937+0000 7f91c0a00640 1 --2- 172.21.15.90:0/1811345630 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91bc0ce5a0 0x7f91bc139dd0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f91b0009870 tx=0x7f91b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:37.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.937+0000 7f91b8c00640 1 -- 172.21.15.90:0/1811345630 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b0041070 con 0x7f91bc0ce5a0 2024-04-25T01:18:37.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.937+0000 7f91b8c00640 1 -- 172.21.15.90:0/1811345630 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91b003c870 con 0x7f91bc0ce5a0 2024-04-25T01:18:37.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.937+0000 7f91b8c00640 1 -- 172.21.15.90:0/1811345630 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b0037040 con 0x7f91bc0ce5a0 2024-04-25T01:18:37.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.937+0000 7f91c3400640 1 -- 172.21.15.90:0/1811345630 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91bc144970 con 0x7f91bc0ce5a0 2024-04-25T01:18:37.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.938+0000 7f91c3400640 1 -- 172.21.15.90:0/1811345630 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91bc144f30 con 0x7f91bc0ce5a0 2024-04-25T01:18:37.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.939+0000 7f91c3400640 1 -- 172.21.15.90:0/1811345630 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91bc04f5a0 con 0x7f91bc0ce5a0 2024-04-25T01:18:37.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.940+0000 7f91b8c00640 1 -- 172.21.15.90:0/1811345630 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f91b0044070 con 0x7f91bc0ce5a0 2024-04-25T01:18:37.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.940+0000 7f91b8c00640 1 --2- 172.21.15.90:0/1811345630 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f919c06be30 0x7f919c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:37.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.941+0000 7f91b8c00640 1 -- 172.21.15.90:0/1811345630 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f91b00abed0 con 0x7f91bc0ce5a0 2024-04-25T01:18:37.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.941+0000 7f91bbe00640 1 --2- 172.21.15.90:0/1811345630 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f919c06be30 0x7f919c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:37.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.941+0000 7f91bbe00640 1 --2- 172.21.15.90:0/1811345630 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f919c06be30 0x7f919c06e2f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f91bc0cdc00 tx=0x7f91a8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:37.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:37.946+0000 7f91b8c00640 1 -- 172.21.15.90:0/1811345630 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f91b007a400 con 0x7f91bc0ce5a0 2024-04-25T01:18:38.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.142+0000 7f91c3400640 1 -- 172.21.15.90:0/1811345630 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "recovery_fs_metadata", "var": "pg_num"} v 0) -- 0x7f91bc145950 con 0x7f91bc0ce5a0 2024-04-25T01:18:38.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.143+0000 7f91b8c00640 1 -- 172.21.15.90:0/1811345630 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "recovery_fs_metadata", "var": "pg_num"}]=0 v50) ==== 109+0+11 (secure 0 0 0) 0x7f91b0079d00 con 0x7f91bc0ce5a0 2024-04-25T01:18:38.127 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:18:38.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.145+0000 7f91c3400640 1 -- 172.21.15.90:0/1811345630 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f919c06be30 msgr2=0x7f919c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:38.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.145+0000 7f91c3400640 1 --2- 172.21.15.90:0/1811345630 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f919c06be30 0x7f919c06e2f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f91bc0cdc00 tx=0x7f91a8008040 comp rx=0 tx=0).stop 2024-04-25T01:18:38.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.145+0000 7f91c3400640 1 -- 172.21.15.90:0/1811345630 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91bc0ce5a0 msgr2=0x7f91bc139dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:38.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.145+0000 7f91c3400640 1 --2- 172.21.15.90:0/1811345630 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91bc0ce5a0 0x7f91bc139dd0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f91b0009870 tx=0x7f91b00046c0 comp rx=0 tx=0).stop 2024-04-25T01:18:38.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.145+0000 7f91c3400640 1 -- 172.21.15.90:0/1811345630 shutdown_connections 2024-04-25T01:18:38.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.145+0000 7f91c3400640 1 --2- 172.21.15.90:0/1811345630 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f919c06be30 0x7f919c06e2f0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:38.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.145+0000 7f91c3400640 1 --2- 172.21.15.90:0/1811345630 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f91bc141ab0 0x7f91bc13d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:38.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.145+0000 7f91c3400640 1 --2- 172.21.15.90:0/1811345630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f91bc13ef30 0x7f91bc13d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:38.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.145+0000 7f91c3400640 1 --2- 172.21.15.90:0/1811345630 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91bc0ce5a0 0x7f91bc139dd0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:38.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.145+0000 7f91c3400640 1 -- 172.21.15.90:0/1811345630 >> 172.21.15.90:0/1811345630 conn(0x7f91bc0b17b0 msgr2=0x7f91bc13fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:38.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.145+0000 7f91c3400640 1 -- 172.21.15.90:0/1811345630 shutdown_connections 2024-04-25T01:18:38.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.145+0000 7f91c3400640 1 -- 172.21.15.90:0/1811345630 wait complete. 2024-04-25T01:18:38.148 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T01:18:38.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.366+0000 7f6309800640 1 Processor -- start 2024-04-25T01:18:38.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.367+0000 7f6309800640 1 -- start start 2024-04-25T01:18:38.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.368+0000 7f6309800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f630415ae50 0x7f630406e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:38.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.368+0000 7f6309800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6304071570 0x7f63040b3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:38.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.368+0000 7f6309800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63040af0a0 con 0x7f63040709d0 2024-04-25T01:18:38.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.368+0000 7f6309800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f63040b2860 con 0x7f6304071570 2024-04-25T01:18:38.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.368+0000 7f6309800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f630406cbe0 con 0x7f630415ae50 2024-04-25T01:18:38.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.369+0000 7f6308e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6304071570 0x7f63040b3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:38.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.369+0000 7f6308e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6304071570 0x7f63040b3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51522/0 (socket says 172.21.15.90:51522) 2024-04-25T01:18:38.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.369+0000 7f6308e00640 1 -- 172.21.15.90:0/3625641268 learned_addr learned my addr 172.21.15.90:0/3625641268 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:38.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.369+0000 7f62fb400640 1 --1- 172.21.15.90:0/3625641268 >> v1:172.21.15.90:6789/0 conn(0x7f63040709d0 0x7f6304070e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41174/0 (socket says 172.21.15.90:41174) 2024-04-25T01:18:38.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.369+0000 7f6303e00640 1 --2- 172.21.15.90:0/3625641268 >> v2:172.21.15.103:3300/0 conn(0x7f630415ae50 0x7f630406e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:38.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.369+0000 7f6308e00640 1 -- 172.21.15.90:0/3625641268 >> v2:172.21.15.103:3300/0 conn(0x7f630415ae50 msgr2=0x7f630406e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:38.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.369+0000 7f6308e00640 1 --2- 172.21.15.90:0/3625641268 >> v2:172.21.15.103:3300/0 conn(0x7f630415ae50 0x7f630406e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:38.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.369+0000 7f6308e00640 1 -- 172.21.15.90:0/3625641268 >> v1:172.21.15.90:6789/0 conn(0x7f63040709d0 legacy=0x7f6304070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:38.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.369+0000 7f6308e00640 1 -- 172.21.15.90:0/3625641268 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f630406cec0 con 0x7f6304071570 2024-04-25T01:18:38.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.369+0000 7f6303400640 1 -- 172.21.15.90:0/3625641268 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1628091093 0 0) 0x7f62ec001110 con 0x7f63040709d0 2024-04-25T01:18:38.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.370+0000 7f6308e00640 1 --2- 172.21.15.90:0/3625641268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6304071570 0x7f63040b3020 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f630406ee50 tx=0x7f62fc02a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa6b30e19d0241ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:38.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.370+0000 7f6303400640 1 -- 172.21.15.90:0/3625641268 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62fc036070 con 0x7f6304071570 2024-04-25T01:18:38.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.370+0000 7f6303400640 1 -- 172.21.15.90:0/3625641268 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62fc031600 con 0x7f6304071570 2024-04-25T01:18:38.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.370+0000 7f6303400640 1 -- 172.21.15.90:0/3625641268 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62fc0319d0 con 0x7f6304071570 2024-04-25T01:18:38.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.371+0000 7f6309800640 1 -- 172.21.15.90:0/3625641268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6304071570 msgr2=0x7f63040b3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:38.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.371+0000 7f6309800640 1 --2- 172.21.15.90:0/3625641268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6304071570 0x7f63040b3020 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f630406ee50 tx=0x7f62fc02a730 comp rx=0 tx=0).stop 2024-04-25T01:18:38.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.371+0000 7f6309800640 1 -- 172.21.15.90:0/3625641268 shutdown_connections 2024-04-25T01:18:38.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.371+0000 7f6309800640 1 --2- 172.21.15.90:0/3625641268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6304071570 0x7f63040b3020 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:38.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.371+0000 7f6309800640 1 --2- 172.21.15.90:0/3625641268 >> v2:172.21.15.103:3300/0 conn(0x7f630415ae50 0x7f630406e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:38.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.371+0000 7f6309800640 1 -- 172.21.15.90:0/3625641268 >> 172.21.15.90:0/3625641268 conn(0x7f63040b1560 msgr2=0x7f63040b1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:38.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.371+0000 7f6309800640 1 -- 172.21.15.90:0/3625641268 shutdown_connections 2024-04-25T01:18:38.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.371+0000 7f6309800640 1 -- 172.21.15.90:0/3625641268 wait complete. 2024-04-25T01:18:38.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.373+0000 7f6309800640 1 Processor -- start 2024-04-25T01:18:38.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.373+0000 7f6309800640 1 -- start start 2024-04-25T01:18:38.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.374+0000 7f6309800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63040709d0 0x7f630415b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:38.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.374+0000 7f6309800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6304071570 0x7f63040cea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:38.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.374+0000 7f6309800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f630415ae50 0x7f63041f3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:38.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.374+0000 7f6309800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f63040b38f0 con 0x7f630415ae50 2024-04-25T01:18:38.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.374+0000 7f6309800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f63040b3770 con 0x7f6304071570 2024-04-25T01:18:38.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.374+0000 7f6309800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f63040b3a70 con 0x7f63040709d0 2024-04-25T01:18:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.375+0000 7f6308e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f630415ae50 0x7f63041f3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.375+0000 7f6308e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f630415ae50 0x7f63041f3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51534/0 (socket says 172.21.15.90:51534) 2024-04-25T01:18:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.375+0000 7f6308e00640 1 -- 172.21.15.90:0/1916144480 learned_addr learned my addr 172.21.15.90:0/1916144480 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.375+0000 7f6303e00640 1 --2- 172.21.15.90:0/1916144480 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63040709d0 0x7f630415b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.375+0000 7f62fb400640 1 --2- 172.21.15.90:0/1916144480 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6304071570 0x7f63040cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.375+0000 7f6308e00640 1 -- 172.21.15.90:0/1916144480 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63040709d0 msgr2=0x7f630415b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.375+0000 7f6308e00640 1 --2- 172.21.15.90:0/1916144480 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63040709d0 0x7f630415b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.375+0000 7f6308e00640 1 -- 172.21.15.90:0/1916144480 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6304071570 msgr2=0x7f63040cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.375+0000 7f6308e00640 1 --2- 172.21.15.90:0/1916144480 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6304071570 0x7f63040cea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.375+0000 7f6308e00640 1 -- 172.21.15.90:0/1916144480 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f630415bb60 con 0x7f630415ae50 2024-04-25T01:18:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.375+0000 7f6308e00640 1 --2- 172.21.15.90:0/1916144480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f630415ae50 0x7f63041f3750 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f62fc009f40 tx=0x7f62fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:38.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.376+0000 7f6300c00640 1 -- 172.21.15.90:0/1916144480 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62fc041020 con 0x7f630415ae50 2024-04-25T01:18:38.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.376+0000 7f6300c00640 1 -- 172.21.15.90:0/1916144480 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62fc0077a0 con 0x7f630415ae50 2024-04-25T01:18:38.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.376+0000 7f6300c00640 1 -- 172.21.15.90:0/1916144480 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62fc036040 con 0x7f630415ae50 2024-04-25T01:18:38.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.376+0000 7f6309800640 1 -- 172.21.15.90:0/1916144480 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f630415be40 con 0x7f630415ae50 2024-04-25T01:18:38.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.376+0000 7f6309800640 1 -- 172.21.15.90:0/1916144480 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f630415c400 con 0x7f630415ae50 2024-04-25T01:18:38.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.377+0000 7f6309800640 1 -- 172.21.15.90:0/1916144480 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f630404f500 con 0x7f630415ae50 2024-04-25T01:18:38.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.378+0000 7f6300c00640 1 -- 172.21.15.90:0/1916144480 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f62fc044020 con 0x7f630415ae50 2024-04-25T01:18:38.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.379+0000 7f6300c00640 1 --2- 172.21.15.90:0/1916144480 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f62e006be30 0x7f62e006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:38.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.379+0000 7f6300c00640 1 -- 172.21.15.90:0/1916144480 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f62fc0abf60 con 0x7f630415ae50 2024-04-25T01:18:38.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.379+0000 7f6303e00640 1 --2- 172.21.15.90:0/1916144480 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f62e006be30 0x7f62e006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:38.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.380+0000 7f6303e00640 1 --2- 172.21.15.90:0/1916144480 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f62e006be30 0x7f62e006e2f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f62f4006fd0 tx=0x7f62f4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:38.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.383+0000 7f6300c00640 1 -- 172.21.15.90:0/1916144480 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f62fc04b030 con 0x7f630415ae50 2024-04-25T01:18:38.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.597+0000 7f6309800640 1 -- 172.21.15.90:0/1916144480 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f63040b2270 con 0x7f630415ae50 2024-04-25T01:18:38.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.598+0000 7f6300c00640 1 -- 172.21.15.90:0/1916144480 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v50) ==== 80+0+120 (secure 0 0 0) 0x7f62fc07a550 con 0x7f630415ae50 2024-04-25T01:18:38.582 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6837/431390375 2024-04-26T01:18:15.602554+0000 2024-04-25T01:18:38.582 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6836/431390375 2024-04-26T01:18:15.602554+0000 2024-04-25T01:18:38.582 INFO:teuthology.orchestra.run.smithi090.stderr:listed 2 entries 2024-04-25T01:18:38.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.602+0000 7f6309800640 1 -- 172.21.15.90:0/1916144480 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f62e006be30 msgr2=0x7f62e006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:38.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.602+0000 7f6309800640 1 --2- 172.21.15.90:0/1916144480 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f62e006be30 0x7f62e006e2f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f62f4006fd0 tx=0x7f62f4008040 comp rx=0 tx=0).stop 2024-04-25T01:18:38.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.602+0000 7f6309800640 1 -- 172.21.15.90:0/1916144480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f630415ae50 msgr2=0x7f63041f3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:38.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.602+0000 7f6309800640 1 --2- 172.21.15.90:0/1916144480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f630415ae50 0x7f63041f3750 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f62fc009f40 tx=0x7f62fc004a40 comp rx=0 tx=0).stop 2024-04-25T01:18:38.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.602+0000 7f6309800640 1 -- 172.21.15.90:0/1916144480 shutdown_connections 2024-04-25T01:18:38.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.602+0000 7f6309800640 1 --2- 172.21.15.90:0/1916144480 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f62e006be30 0x7f62e006e2f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:38.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.602+0000 7f6309800640 1 --2- 172.21.15.90:0/1916144480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f630415ae50 0x7f63041f3750 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:38.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.602+0000 7f6309800640 1 --2- 172.21.15.90:0/1916144480 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6304071570 0x7f63040cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:38.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.602+0000 7f6309800640 1 --2- 172.21.15.90:0/1916144480 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f63040709d0 0x7f630415b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:38.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.602+0000 7f6309800640 1 -- 172.21.15.90:0/1916144480 >> 172.21.15.90:0/1916144480 conn(0x7f63040b1560 msgr2=0x7f63040cdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:38.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.602+0000 7f6309800640 1 -- 172.21.15.90:0/1916144480 shutdown_connections 2024-04-25T01:18:38.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:38.602+0000 7f6309800640 1 -- 172.21.15.90:0/1916144480 wait complete. 2024-04-25T01:18:38.604 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-25T01:18:38.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:38.604 DEBUG:teuthology.orchestra.run.smithi090:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-25T01:18:38.693 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-04-25T01:18:38.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:38.694 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T01:18:38.743 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T01:18:38.744 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:18:38.744 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:18:38.801 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052734,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T01:18:38.801 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T01:18:38.801 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:18:38.801 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:18:38.857 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052717,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T01:18:38.857 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T01:18:38.857 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:18:38.857 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:18:38.915 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052734,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T01:18:38.915 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:18:39.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.138+0000 7fe169200640 1 Processor -- start 2024-04-25T01:18:39.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.138+0000 7fe169200640 1 -- start start 2024-04-25T01:18:39.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.138+0000 7fe169200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe16415e250 0x7fe1640cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:39.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.138+0000 7fe169200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fe1640d4910 0x7fe1640d0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:39.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.138+0000 7fe169200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1640cfd30 con 0x7fe1640d0cd0 2024-04-25T01:18:39.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.138+0000 7fe169200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fe1640c80c0 con 0x7fe1640d4910 2024-04-25T01:18:39.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.138+0000 7fe169200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe1640c7a70 con 0x7fe16415e250 2024-04-25T01:18:39.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.140+0000 7fe163e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe1640d0cd0 0x7fe1640b4920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41184/0 (socket says 172.21.15.90:41184) 2024-04-25T01:18:39.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.140+0000 7fe163e00640 1 -- 172.21.15.90:0/116045384 learned_addr learned my addr 172.21.15.90:0/116045384 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:39.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.140+0000 7fe163400640 1 --2- 172.21.15.90:0/116045384 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe16415e250 0x7fe1640cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:39.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.140+0000 7fe162a00640 1 --2- 172.21.15.90:0/116045384 >> v2:172.21.15.103:3300/0 conn(0x7fe1640d4910 0x7fe1640d0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:39.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.140+0000 7fe162000640 1 -- 172.21.15.90:0/116045384 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2169672060 0 0) 0x7fe1640cfd30 con 0x7fe1640d0cd0 2024-04-25T01:18:39.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.140+0000 7fe162000640 1 -- 172.21.15.90:0/116045384 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe148003520 con 0x7fe1640d0cd0 2024-04-25T01:18:39.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.140+0000 7fe162000640 1 -- 172.21.15.90:0/116045384 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 522922058 0 0) 0x7fe148003520 con 0x7fe1640d0cd0 2024-04-25T01:18:39.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.140+0000 7fe162000640 1 -- 172.21.15.90:0/116045384 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe16415e250 msgr2=0x7fe1640cf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:39.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.140+0000 7fe162000640 1 --2- 172.21.15.90:0/116045384 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe16415e250 0x7fe1640cf3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:39.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.140+0000 7fe162000640 1 -- 172.21.15.90:0/116045384 >> v2:172.21.15.103:3300/0 conn(0x7fe1640d4910 msgr2=0x7fe1640d0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:39.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.140+0000 7fe162000640 1 --2- 172.21.15.90:0/116045384 >> v2:172.21.15.103:3300/0 conn(0x7fe1640d4910 0x7fe1640d0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:39.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.140+0000 7fe162000640 1 -- 172.21.15.90:0/116045384 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1640c7d50 con 0x7fe1640d0cd0 2024-04-25T01:18:39.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.140+0000 7fe162000640 1 -- 172.21.15.90:0/116045384 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7fe15c0019f0 con 0x7fe1640d0cd0 2024-04-25T01:18:39.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.140+0000 7fe162000640 1 -- 172.21.15.90:0/116045384 >> v1:172.21.15.90:6789/0 conn(0x7fe1640d0cd0 legacy=0x7fe1640b4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:39.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.140+0000 7fe162000640 1 --2- 172.21.15.90:0/116045384 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe16415e250 0x7fe1480039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:39.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.140+0000 7fe162000640 1 --2- 172.21.15.90:0/116045384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe148003f30 0x7fe148004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:39.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.140+0000 7fe162000640 1 --2- 172.21.15.90:0/116045384 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe148004d40 0x7fe1480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:39.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.140+0000 7fe162000640 1 -- 172.21.15.90:0/116045384 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe1640c80c0 con 0x7fe148004d40 2024-04-25T01:18:39.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.140+0000 7fe162000640 1 -- 172.21.15.90:0/116045384 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe1640c7a70 con 0x7fe148003f30 2024-04-25T01:18:39.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.140+0000 7fe162000640 1 -- 172.21.15.90:0/116045384 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe14800ba10 con 0x7fe16415e250 2024-04-25T01:18:39.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.140+0000 7fe163e00640 1 --2- 172.21.15.90:0/116045384 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe148004d40 0x7fe1480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:39.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.142+0000 7fe163e00640 1 -- 172.21.15.90:0/116045384 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe16415e250 msgr2=0x7fe1480039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:39.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.142+0000 7fe163400640 1 --2- 172.21.15.90:0/116045384 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe16415e250 0x7fe1480039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:39.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.142+0000 7fe162a00640 1 --2- 172.21.15.90:0/116045384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe148003f30 0x7fe148004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:39.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.142+0000 7fe163e00640 1 --2- 172.21.15.90:0/116045384 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe16415e250 0x7fe1480039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:39.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.142+0000 7fe163e00640 1 -- 172.21.15.90:0/116045384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe148003f30 msgr2=0x7fe148004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:39.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.142+0000 7fe163e00640 1 --2- 172.21.15.90:0/116045384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe148003f30 0x7fe148004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:39.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.142+0000 7fe163e00640 1 -- 172.21.15.90:0/116045384 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe148003520 con 0x7fe148004d40 2024-04-25T01:18:39.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.143+0000 7fe163e00640 1 --2- 172.21.15.90:0/116045384 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe148004d40 0x7fe1480051c0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fe15c00a8d0 tx=0x7fe148005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8805f58c5e61f589 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:39.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.143+0000 7fe162000640 1 -- 172.21.15.90:0/116045384 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe15c00a4c0 con 0x7fe148004d40 2024-04-25T01:18:39.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.143+0000 7fe162000640 1 -- 172.21.15.90:0/116045384 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe15c02f070 con 0x7fe148004d40 2024-04-25T01:18:39.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.143+0000 7fe162000640 1 -- 172.21.15.90:0/116045384 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe15c02bd40 con 0x7fe148004d40 2024-04-25T01:18:39.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.143+0000 7fe169200640 1 -- 172.21.15.90:0/116045384 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe148004d40 msgr2=0x7fe1480051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:39.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.143+0000 7fe169200640 1 --2- 172.21.15.90:0/116045384 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe148004d40 0x7fe1480051c0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fe15c00a8d0 tx=0x7fe148005830 comp rx=0 tx=0).stop 2024-04-25T01:18:39.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.143+0000 7fe163e00640 1 -- 172.21.15.90:0/116045384 reap_dead start 2024-04-25T01:18:39.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.144+0000 7fe169200640 1 -- 172.21.15.90:0/116045384 shutdown_connections 2024-04-25T01:18:39.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.144+0000 7fe169200640 1 -- 172.21.15.90:0/116045384 >> 172.21.15.90:0/116045384 conn(0x7fe1640d3d80 msgr2=0x7fe1640d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:39.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.144+0000 7fe169200640 1 -- 172.21.15.90:0/116045384 shutdown_connections 2024-04-25T01:18:39.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.144+0000 7fe169200640 1 -- 172.21.15.90:0/116045384 wait complete. 2024-04-25T01:18:39.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.144+0000 7fe169200640 1 Processor -- start 2024-04-25T01:18:39.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.145+0000 7fe169200640 1 -- start start 2024-04-25T01:18:39.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.145+0000 7fe169200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe148004d40 0x7fe1641f7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:39.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.145+0000 7fe169200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1641509e0 0x7fe1640b1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:39.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.145+0000 7fe169200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe164150e60 0x7fe1640b23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:39.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.145+0000 7fe169200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe1640b51b0 con 0x7fe164150e60 2024-04-25T01:18:39.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.145+0000 7fe169200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe1640b5030 con 0x7fe148004d40 2024-04-25T01:18:39.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.145+0000 7fe169200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fe1640b5330 con 0x7fe1641509e0 2024-04-25T01:18:39.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.146+0000 7fe163e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe164150e60 0x7fe1640b23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:39.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.146+0000 7fe163e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe164150e60 0x7fe1640b23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51540/0 (socket says 172.21.15.90:51540) 2024-04-25T01:18:39.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.146+0000 7fe163e00640 1 -- 172.21.15.90:0/2551407814 learned_addr learned my addr 172.21.15.90:0/2551407814 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:39.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.146+0000 7fe163400640 1 --2- 172.21.15.90:0/2551407814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe148004d40 0x7fe1641f7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:39.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.146+0000 7fe162a00640 1 --2- 172.21.15.90:0/2551407814 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1641509e0 0x7fe1640b1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:39.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.146+0000 7fe163e00640 1 -- 172.21.15.90:0/2551407814 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1641509e0 msgr2=0x7fe1640b1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:39.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.146+0000 7fe163e00640 1 --2- 172.21.15.90:0/2551407814 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1641509e0 0x7fe1640b1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:39.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.146+0000 7fe163e00640 1 -- 172.21.15.90:0/2551407814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe148004d40 msgr2=0x7fe1641f7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:39.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.146+0000 7fe163e00640 1 --2- 172.21.15.90:0/2551407814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe148004d40 0x7fe1641f7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:39.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.146+0000 7fe163e00640 1 -- 172.21.15.90:0/2551407814 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1641f8090 con 0x7fe164150e60 2024-04-25T01:18:39.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.146+0000 7fe163400640 1 --2- 172.21.15.90:0/2551407814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe148004d40 0x7fe1641f7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:39.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.148+0000 7fe163e00640 1 --2- 172.21.15.90:0/2551407814 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe164150e60 0x7fe1640b23f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fe15c00a8d0 tx=0x7fe15c001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:39.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.148+0000 7fe15b400640 1 -- 172.21.15.90:0/2551407814 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe15c044070 con 0x7fe164150e60 2024-04-25T01:18:39.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.148+0000 7fe15b400640 1 -- 172.21.15.90:0/2551407814 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe15c001a00 con 0x7fe164150e60 2024-04-25T01:18:39.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.148+0000 7fe15b400640 1 -- 172.21.15.90:0/2551407814 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe15c00a4c0 con 0x7fe164150e60 2024-04-25T01:18:39.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.148+0000 7fe169200640 1 -- 172.21.15.90:0/2551407814 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1640cd120 con 0x7fe164150e60 2024-04-25T01:18:39.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.148+0000 7fe169200640 1 -- 172.21.15.90:0/2551407814 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1640cd6e0 con 0x7fe164150e60 2024-04-25T01:18:39.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.149+0000 7fe169200640 1 -- 172.21.15.90:0/2551407814 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe16404f500 con 0x7fe164150e60 2024-04-25T01:18:39.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.150+0000 7fe15b400640 1 -- 172.21.15.90:0/2551407814 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe15c034670 con 0x7fe164150e60 2024-04-25T01:18:39.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.151+0000 7fe15b400640 1 --2- 172.21.15.90:0/2551407814 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe14406be30 0x7fe14406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:39.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.151+0000 7fe15b400640 1 -- 172.21.15.90:0/2551407814 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7fe15c0ab290 con 0x7fe164150e60 2024-04-25T01:18:39.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.151+0000 7fe163400640 1 --2- 172.21.15.90:0/2551407814 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe14406be30 0x7fe14406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:39.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.152+0000 7fe163400640 1 --2- 172.21.15.90:0/2551407814 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe14406be30 0x7fe14406e2f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe1540099e0 tx=0x7fe154008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:39.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.155+0000 7fe15b400640 1 -- 172.21.15.90:0/2551407814 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe1641f8090 con 0x7fe164150e60 2024-04-25T01:18:39.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.393+0000 7fe169200640 1 -- 172.21.15.90:0/2551407814 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe16415c930 con 0x7fe164150e60 2024-04-25T01:18:39.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.394+0000 7fe15b400640 1 -- 172.21.15.90:0/2551407814 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6492 (secure 0 0 0) 0x7fe15c079850 con 0x7fe164150e60 2024-04-25T01:18:39.377 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:39.377 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1873497946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6833","nonce":1873497946},{"type":"v1","addr":"172.21.15.103:6835","nonce":1873497946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/3206865830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":3206865830},{"type":"v1","addr":"172.21.15.90:6835","nonce":3206865830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5598,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2208594231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2208594231},{"type":"v1","addr":"172.21.15.90:6837","nonce":2208594231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:18:02.701652+0000","modified":"2024-04-25T01:18:15.603132+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2},{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:18:19.371011+0000","modified":"2024-04-25T01:18:30.533899+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5601":{"gid":5601,"name":"d","rank":0,"incarnation":25,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6834/2850373997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":2850373997},{"type":"v1","addr":"172.21.15.103:6834","nonce":2850373997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":7,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5601,"qdb_cluster":[5601]},"id":3}]} 2024-04-25T01:18:39.377 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 28 2024-04-25T01:18:39.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.396+0000 7fe169200640 1 -- 172.21.15.90:0/2551407814 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe14406be30 msgr2=0x7fe14406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:39.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.396+0000 7fe169200640 1 --2- 172.21.15.90:0/2551407814 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe14406be30 0x7fe14406e2f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe1540099e0 tx=0x7fe154008040 comp rx=0 tx=0).stop 2024-04-25T01:18:39.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.396+0000 7fe169200640 1 -- 172.21.15.90:0/2551407814 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe164150e60 msgr2=0x7fe1640b23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:39.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.396+0000 7fe169200640 1 --2- 172.21.15.90:0/2551407814 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe164150e60 0x7fe1640b23f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fe15c00a8d0 tx=0x7fe15c001dc0 comp rx=0 tx=0).stop 2024-04-25T01:18:39.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.397+0000 7fe169200640 1 -- 172.21.15.90:0/2551407814 shutdown_connections 2024-04-25T01:18:39.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.397+0000 7fe169200640 1 --2- 172.21.15.90:0/2551407814 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fe14406be30 0x7fe14406e2f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:39.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.397+0000 7fe169200640 1 --2- 172.21.15.90:0/2551407814 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe164150e60 0x7fe1640b23f0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:39.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.397+0000 7fe169200640 1 --2- 172.21.15.90:0/2551407814 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fe1641509e0 0x7fe1640b1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:39.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.397+0000 7fe169200640 1 --2- 172.21.15.90:0/2551407814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe148004d40 0x7fe1641f7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:39.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.397+0000 7fe169200640 1 -- 172.21.15.90:0/2551407814 >> 172.21.15.90:0/2551407814 conn(0x7fe1640d3d80 msgr2=0x7fe164160850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:39.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.397+0000 7fe169200640 1 -- 172.21.15.90:0/2551407814 shutdown_connections 2024-04-25T01:18:39.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.397+0000 7fe169200640 1 -- 172.21.15.90:0/2551407814 wait complete. 2024-04-25T01:18:39.399 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:18:39.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.608+0000 7f9e1b200640 1 Processor -- start 2024-04-25T01:18:39.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.609+0000 7f9e1b200640 1 -- start start 2024-04-25T01:18:39.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.610+0000 7f9e1b200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f9e1c1561c0 0x7f9e1c0cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:39.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.610+0000 7f9e1b200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9e1c0709d0 0x7f9e1c15cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:39.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.610+0000 7f9e1b200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e1c070f90 con 0x7f9e1c0cfb30 2024-04-25T01:18:39.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.610+0000 7f9e1b200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f9e1c0b4310 con 0x7f9e1c0709d0 2024-04-25T01:18:39.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.610+0000 7f9e1b200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f9e1c1565a0 con 0x7f9e1c1561c0 2024-04-25T01:18:39.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.611+0000 7f9e13e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9e1c0709d0 0x7f9e1c15cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:39.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.611+0000 7f9e1a800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f9e1c0cfb30 0x7f9e1c0cff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41186/0 (socket says 172.21.15.90:41186) 2024-04-25T01:18:39.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.611+0000 7f9e1a800640 1 -- 172.21.15.90:0/125455824 learned_addr learned my addr 172.21.15.90:0/125455824 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:39.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.611+0000 7f9e13e00640 1 --2- 172.21.15.90:0/125455824 >> v2:172.21.15.90:3300/0 conn(0x7f9e1c0709d0 0x7f9e1c15cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51546/0 (socket says 172.21.15.90:51546) 2024-04-25T01:18:39.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.611+0000 7f9e19e00640 1 --2- 172.21.15.90:0/125455824 >> v2:172.21.15.103:3300/0 conn(0x7f9e1c1561c0 0x7f9e1c0cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:39.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.611+0000 7f9e13e00640 1 -- 172.21.15.90:0/125455824 >> v2:172.21.15.103:3300/0 conn(0x7f9e1c1561c0 msgr2=0x7f9e1c0cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:39.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.611+0000 7f9e13e00640 1 --2- 172.21.15.90:0/125455824 >> v2:172.21.15.103:3300/0 conn(0x7f9e1c1561c0 0x7f9e1c0cdbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:39.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.611+0000 7f9e13e00640 1 -- 172.21.15.90:0/125455824 >> v1:172.21.15.90:6789/0 conn(0x7f9e1c0cfb30 legacy=0x7f9e1c0cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:39.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.611+0000 7f9e19400640 1 -- 172.21.15.90:0/125455824 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 523759250 0 0) 0x7f9e1c070f90 con 0x7f9e1c0cfb30 2024-04-25T01:18:39.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.611+0000 7f9e13e00640 1 -- 172.21.15.90:0/125455824 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e1c071270 con 0x7f9e1c0709d0 2024-04-25T01:18:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.612+0000 7f9e13e00640 1 --2- 172.21.15.90:0/125455824 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1c0709d0 0x7f9e1c15cb80 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c0706c0 tx=0x7f9e0402a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bde73bec3883e64 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.612+0000 7f9e19400640 1 -- 172.21.15.90:0/125455824 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e04037070 con 0x7f9e1c0709d0 2024-04-25T01:18:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.612+0000 7f9e19400640 1 -- 172.21.15.90:0/125455824 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e0402add0 con 0x7f9e1c0709d0 2024-04-25T01:18:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.612+0000 7f9e19400640 1 -- 172.21.15.90:0/125455824 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e04032660 con 0x7f9e1c0709d0 2024-04-25T01:18:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.613+0000 7f9e1b200640 1 -- 172.21.15.90:0/125455824 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1c0709d0 msgr2=0x7f9e1c15cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.613+0000 7f9e1b200640 1 --2- 172.21.15.90:0/125455824 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1c0709d0 0x7f9e1c15cb80 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c0706c0 tx=0x7f9e0402a080 comp rx=0 tx=0).stop 2024-04-25T01:18:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.613+0000 7f9e1b200640 1 -- 172.21.15.90:0/125455824 shutdown_connections 2024-04-25T01:18:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.613+0000 7f9e1b200640 1 --2- 172.21.15.90:0/125455824 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1c0709d0 0x7f9e1c15cb80 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.613+0000 7f9e1b200640 1 --2- 172.21.15.90:0/125455824 >> v2:172.21.15.103:3300/0 conn(0x7f9e1c1561c0 0x7f9e1c0cdbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.613+0000 7f9e1b200640 1 -- 172.21.15.90:0/125455824 >> 172.21.15.90:0/125455824 conn(0x7f9e1c06b320 msgr2=0x7f9e1c06b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.613+0000 7f9e1b200640 1 -- 172.21.15.90:0/125455824 shutdown_connections 2024-04-25T01:18:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.613+0000 7f9e1b200640 1 -- 172.21.15.90:0/125455824 wait complete. 2024-04-25T01:18:39.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.615+0000 7f9e1b200640 1 Processor -- start 2024-04-25T01:18:39.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.615+0000 7f9e1b200640 1 -- start start 2024-04-25T01:18:39.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.616+0000 7f9e1b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e1c0709d0 0x7f9e1c157d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:39.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.616+0000 7f9e1b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1c0cfb30 0x7f9e1c06e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:39.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.616+0000 7f9e1b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e1c1561c0 0x7f9e1c147260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:39.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.616+0000 7f9e1b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1c0b2c20 con 0x7f9e1c0cfb30 2024-04-25T01:18:39.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.616+0000 7f9e1b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1c071830 con 0x7f9e1c1561c0 2024-04-25T01:18:39.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.616+0000 7f9e1b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9e1c0b2da0 con 0x7f9e1c0709d0 2024-04-25T01:18:39.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.616+0000 7f9e13e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1c0cfb30 0x7f9e1c06e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:39.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.616+0000 7f9e13e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1c0cfb30 0x7f9e1c06e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51550/0 (socket says 172.21.15.90:51550) 2024-04-25T01:18:39.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.616+0000 7f9e13e00640 1 -- 172.21.15.90:0/3203818398 learned_addr learned my addr 172.21.15.90:0/3203818398 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:39.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.616+0000 7f9e13e00640 1 -- 172.21.15.90:0/3203818398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e1c0709d0 msgr2=0x7f9e1c157d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:39.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.616+0000 7f9e13e00640 1 --2- 172.21.15.90:0/3203818398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e1c0709d0 0x7f9e1c157d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:39.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.616+0000 7f9e13e00640 1 -- 172.21.15.90:0/3203818398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e1c1561c0 msgr2=0x7f9e1c147260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:39.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.616+0000 7f9e13e00640 1 --2- 172.21.15.90:0/3203818398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e1c1561c0 0x7f9e1c147260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:39.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.616+0000 7f9e13e00640 1 -- 172.21.15.90:0/3203818398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e1c06e9e0 con 0x7f9e1c0cfb30 2024-04-25T01:18:39.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.617+0000 7f9e13e00640 1 --2- 172.21.15.90:0/3203818398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1c0cfb30 0x7f9e1c06e4a0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f9e04000c00 tx=0x7f9e040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:39.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.617+0000 7f9e12000640 1 -- 172.21.15.90:0/3203818398 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e04037040 con 0x7f9e1c0cfb30 2024-04-25T01:18:39.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.617+0000 7f9e12000640 1 -- 172.21.15.90:0/3203818398 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e04042070 con 0x7f9e1c0cfb30 2024-04-25T01:18:39.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.617+0000 7f9e1b200640 1 -- 172.21.15.90:0/3203818398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e1c158460 con 0x7f9e1c0cfb30 2024-04-25T01:18:39.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.617+0000 7f9e12000640 1 -- 172.21.15.90:0/3203818398 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0403b890 con 0x7f9e1c0cfb30 2024-04-25T01:18:39.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.617+0000 7f9e1b200640 1 -- 172.21.15.90:0/3203818398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e1c158a20 con 0x7f9e1c0cfb30 2024-04-25T01:18:39.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.618+0000 7f9e1b200640 1 -- 172.21.15.90:0/3203818398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e1c04f500 con 0x7f9e1c0cfb30 2024-04-25T01:18:39.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.619+0000 7f9e12000640 1 -- 172.21.15.90:0/3203818398 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9e0402ac50 con 0x7f9e1c0cfb30 2024-04-25T01:18:39.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.620+0000 7f9e12000640 1 --2- 172.21.15.90:0/3203818398 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9df806be30 0x7f9df806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:39.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.620+0000 7f9e12000640 1 -- 172.21.15.90:0/3203818398 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f9e040ab8b0 con 0x7f9e1c0cfb30 2024-04-25T01:18:39.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.620+0000 7f9e19e00640 1 --2- 172.21.15.90:0/3203818398 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9df806be30 0x7f9df806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:39.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.621+0000 7f9e19e00640 1 --2- 172.21.15.90:0/3203818398 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9df806be30 0x7f9df806e2f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c006fd0 tx=0x7f9e0c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:39.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.624+0000 7f9e12000640 1 -- 172.21.15.90:0/3203818398 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9e04079e80 con 0x7f9e1c0cfb30 2024-04-25T01:18:39.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.822+0000 7f9e1b200640 1 -- 172.21.15.90:0/3203818398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9e1c06b7a0 con 0x7f9e1c0cfb30 2024-04-25T01:18:39.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.824+0000 7f9e12000640 1 -- 172.21.15.90:0/3203818398 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) ==== 74+0+18947 (secure 0 0 0) 0x7f9e040a9e20 con 0x7f9e1c0cfb30 2024-04-25T01:18:39.808 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:39.808 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":50,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:18:20.389626+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:18:00.504992+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T01:18:01.714313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"recovery_fs_metadata","create_time":"2024-04-25T01:18:18.384671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.16","mappings":[{"from":4,"to":2}]},{"pgid":"6.20","mappings":[{"from":4,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:6837/431390375":"2024-04-26T01:18:15.602554+0000","172.21.15.90:6836/431390375":"2024-04-26T01:18:15.602554+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:18:39.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.828+0000 7f9e1b200640 1 -- 172.21.15.90:0/3203818398 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9df806be30 msgr2=0x7f9df806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:39.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.828+0000 7f9e1b200640 1 --2- 172.21.15.90:0/3203818398 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9df806be30 0x7f9df806e2f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c006fd0 tx=0x7f9e0c008040 comp rx=0 tx=0).stop 2024-04-25T01:18:39.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.829+0000 7f9e1b200640 1 -- 172.21.15.90:0/3203818398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1c0cfb30 msgr2=0x7f9e1c06e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:39.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.829+0000 7f9e1b200640 1 --2- 172.21.15.90:0/3203818398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1c0cfb30 0x7f9e1c06e4a0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f9e04000c00 tx=0x7f9e040046c0 comp rx=0 tx=0).stop 2024-04-25T01:18:39.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.829+0000 7f9e1b200640 1 -- 172.21.15.90:0/3203818398 shutdown_connections 2024-04-25T01:18:39.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.829+0000 7f9e1b200640 1 --2- 172.21.15.90:0/3203818398 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9df806be30 0x7f9df806e2f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:39.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.829+0000 7f9e1b200640 1 --2- 172.21.15.90:0/3203818398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e1c1561c0 0x7f9e1c147260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:39.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.829+0000 7f9e1b200640 1 --2- 172.21.15.90:0/3203818398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1c0cfb30 0x7f9e1c06e4a0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:39.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.829+0000 7f9e1b200640 1 --2- 172.21.15.90:0/3203818398 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e1c0709d0 0x7f9e1c157d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:39.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.829+0000 7f9e1b200640 1 -- 172.21.15.90:0/3203818398 >> 172.21.15.90:0/3203818398 conn(0x7f9e1c06b320 msgr2=0x7f9e1c1554f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:39.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.829+0000 7f9e1b200640 1 -- 172.21.15.90:0/3203818398 shutdown_connections 2024-04-25T01:18:39.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:39.829+0000 7f9e1b200640 1 -- 172.21.15.90:0/3203818398 wait complete. 2024-04-25T01:18:39.831 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:16:15.894126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:18:00.504992+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:18:01.714313+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'recovery_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'recovery_fs_metadata', 'create_time': '2024-04-25T01:18:18.384671+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'recovery_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:18:39.832 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:18:40.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.056+0000 7fd173200640 1 Processor -- start 2024-04-25T01:18:40.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.056+0000 7fd173200640 1 -- start start 2024-04-25T01:18:40.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.058+0000 7fd173200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd16c15be40 0x7fd16c0b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:40.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.058+0000 7fd173200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd16c0b3250 con 0x7fd16c155460 2024-04-25T01:18:40.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.058+0000 7fd173200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd16c0b3530 con 0x7fd16c06ea40 2024-04-25T01:18:40.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.058+0000 7fd173200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fd16c15c2b0 con 0x7fd16c15be40 2024-04-25T01:18:40.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.058+0000 7fd171200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd16c15be40 0x7fd16c0b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:40.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.058+0000 7fd16be00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd16c155460 0x7fd16c06e330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41192/0 (socket says 172.21.15.90:41192) 2024-04-25T01:18:40.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.059+0000 7fd16be00640 1 -- 172.21.15.90:0/2161319122 learned_addr learned my addr 172.21.15.90:0/2161319122 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:40.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.059+0000 7fd171200640 1 -- 172.21.15.90:0/2161319122 >> v1:172.21.15.103:6789/0 conn(0x7fd16c06ea40 legacy=0x7fd16c0713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:40.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.059+0000 7fd171200640 1 -- 172.21.15.90:0/2161319122 >> v1:172.21.15.90:6789/0 conn(0x7fd16c155460 legacy=0x7fd16c06e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:40.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.059+0000 7fd171200640 1 -- 172.21.15.90:0/2161319122 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd16c0b3810 con 0x7fd16c15be40 2024-04-25T01:18:40.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.060+0000 7fd171200640 1 --2- 172.21.15.90:0/2161319122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd16c15be40 0x7fd16c0b2ab0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fd164009d90 tx=0x7fd16402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ee419fdc73d30d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:40.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.060+0000 7fd16b400640 1 -- 172.21.15.90:0/2161319122 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd164007740 con 0x7fd16c15be40 2024-04-25T01:18:40.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.060+0000 7fd16b400640 1 -- 172.21.15.90:0/2161319122 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1640078e0 con 0x7fd16c15be40 2024-04-25T01:18:40.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.060+0000 7fd16b400640 1 -- 172.21.15.90:0/2161319122 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd164007cb0 con 0x7fd16c15be40 2024-04-25T01:18:40.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.061+0000 7fd173200640 1 -- 172.21.15.90:0/2161319122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd16c15be40 msgr2=0x7fd16c0b2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:40.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.061+0000 7fd173200640 1 --2- 172.21.15.90:0/2161319122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd16c15be40 0x7fd16c0b2ab0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fd164009d90 tx=0x7fd16402a6a0 comp rx=0 tx=0).stop 2024-04-25T01:18:40.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.061+0000 7fd173200640 1 -- 172.21.15.90:0/2161319122 shutdown_connections 2024-04-25T01:18:40.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.061+0000 7fd173200640 1 --2- 172.21.15.90:0/2161319122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd16c15be40 0x7fd16c0b2ab0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:40.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.061+0000 7fd173200640 1 -- 172.21.15.90:0/2161319122 >> 172.21.15.90:0/2161319122 conn(0x7fd16c0b17b0 msgr2=0x7fd16c0b1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:40.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.061+0000 7fd173200640 1 -- 172.21.15.90:0/2161319122 shutdown_connections 2024-04-25T01:18:40.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.061+0000 7fd173200640 1 -- 172.21.15.90:0/2161319122 wait complete. 2024-04-25T01:18:40.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.063+0000 7fd173200640 1 Processor -- start 2024-04-25T01:18:40.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.063+0000 7fd173200640 1 -- start start 2024-04-25T01:18:40.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.064+0000 7fd173200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd16c06ea40 0x7fd16c1526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:40.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.064+0000 7fd173200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd16c155460 0x7fd16c06d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:40.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.064+0000 7fd173200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd16c15be40 0x7fd16c14bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:40.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.064+0000 7fd173200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c0b3af0 con 0x7fd16c15be40 2024-04-25T01:18:40.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.064+0000 7fd173200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c0b2ff0 con 0x7fd16c06ea40 2024-04-25T01:18:40.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.064+0000 7fd173200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd16c15c480 con 0x7fd16c155460 2024-04-25T01:18:40.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.064+0000 7fd171200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd16c15be40 0x7fd16c14bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:40.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.064+0000 7fd171200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd16c15be40 0x7fd16c14bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51572/0 (socket says 172.21.15.90:51572) 2024-04-25T01:18:40.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.064+0000 7fd171200640 1 -- 172.21.15.90:0/2745086174 learned_addr learned my addr 172.21.15.90:0/2745086174 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:40.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.065+0000 7fd16be00640 1 --2- 172.21.15.90:0/2745086174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd16c06ea40 0x7fd16c1526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:40.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.065+0000 7fd171200640 1 -- 172.21.15.90:0/2745086174 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd16c155460 msgr2=0x7fd16c06d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:40.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.065+0000 7fd171200640 1 --2- 172.21.15.90:0/2745086174 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd16c155460 0x7fd16c06d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:40.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.065+0000 7fd171200640 1 -- 172.21.15.90:0/2745086174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd16c06ea40 msgr2=0x7fd16c1526d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:40.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.065+0000 7fd171200640 1 --2- 172.21.15.90:0/2745086174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd16c06ea40 0x7fd16c1526d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:40.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.065+0000 7fd171200640 1 -- 172.21.15.90:0/2745086174 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd16c14d180 con 0x7fd16c15be40 2024-04-25T01:18:40.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.065+0000 7fd171200640 1 --2- 172.21.15.90:0/2745086174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd16c15be40 0x7fd16c14bdf0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fd164009d90 tx=0x7fd164004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:40.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.065+0000 7fd168c00640 1 -- 172.21.15.90:0/2745086174 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd164040070 con 0x7fd16c15be40 2024-04-25T01:18:40.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.065+0000 7fd168c00640 1 -- 172.21.15.90:0/2745086174 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd164031c90 con 0x7fd16c15be40 2024-04-25T01:18:40.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.065+0000 7fd168c00640 1 -- 172.21.15.90:0/2745086174 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16403c640 con 0x7fd16c15be40 2024-04-25T01:18:40.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.065+0000 7fd173200640 1 -- 172.21.15.90:0/2745086174 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd16c14d460 con 0x7fd16c15be40 2024-04-25T01:18:40.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.066+0000 7fd173200640 1 -- 172.21.15.90:0/2745086174 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd16c14da20 con 0x7fd16c15be40 2024-04-25T01:18:40.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.067+0000 7fd173200640 1 -- 172.21.15.90:0/2745086174 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd16c04f5a0 con 0x7fd16c15be40 2024-04-25T01:18:40.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.068+0000 7fd168c00640 1 -- 172.21.15.90:0/2745086174 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd164039040 con 0x7fd16c15be40 2024-04-25T01:18:40.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.068+0000 7fd168c00640 1 --2- 172.21.15.90:0/2745086174 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd14806be30 0x7fd14806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:40.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.069+0000 7fd168c00640 1 -- 172.21.15.90:0/2745086174 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7fd1640ac580 con 0x7fd16c15be40 2024-04-25T01:18:40.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.069+0000 7fd16be00640 1 --2- 172.21.15.90:0/2745086174 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd14806be30 0x7fd14806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:40.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.069+0000 7fd16be00640 1 --2- 172.21.15.90:0/2745086174 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd14806be30 0x7fd14806e2f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd15c000f60 tx=0x7fd15c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:40.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.073+0000 7fd168c00640 1 -- 172.21.15.90:0/2745086174 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd164039330 con 0x7fd16c15be40 2024-04-25T01:18:40.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.269+0000 7fd173200640 1 -- 172.21.15.90:0/2745086174 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd16c06ce50 con 0x7fd16c15be40 2024-04-25T01:18:40.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.270+0000 7fd168c00640 1 -- 172.21.15.90:0/2745086174 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) ==== 93+0+10 (secure 0 0 0) 0x7fd16407ab70 con 0x7fd16c15be40 2024-04-25T01:18:40.253 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T01:18:40.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.273+0000 7fd173200640 1 -- 172.21.15.90:0/2745086174 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd14806be30 msgr2=0x7fd14806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:40.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.273+0000 7fd173200640 1 --2- 172.21.15.90:0/2745086174 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd14806be30 0x7fd14806e2f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd15c000f60 tx=0x7fd15c009040 comp rx=0 tx=0).stop 2024-04-25T01:18:40.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.273+0000 7fd173200640 1 -- 172.21.15.90:0/2745086174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd16c15be40 msgr2=0x7fd16c14bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:40.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.273+0000 7fd173200640 1 --2- 172.21.15.90:0/2745086174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd16c15be40 0x7fd16c14bdf0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fd164009d90 tx=0x7fd164004a40 comp rx=0 tx=0).stop 2024-04-25T01:18:40.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.273+0000 7fd173200640 1 -- 172.21.15.90:0/2745086174 shutdown_connections 2024-04-25T01:18:40.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.273+0000 7fd173200640 1 --2- 172.21.15.90:0/2745086174 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd14806be30 0x7fd14806e2f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:40.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.273+0000 7fd173200640 1 --2- 172.21.15.90:0/2745086174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd16c15be40 0x7fd16c14bdf0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:40.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.273+0000 7fd173200640 1 --2- 172.21.15.90:0/2745086174 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd16c155460 0x7fd16c06d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:40.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.273+0000 7fd173200640 1 --2- 172.21.15.90:0/2745086174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd16c06ea40 0x7fd16c1526d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:40.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.273+0000 7fd173200640 1 -- 172.21.15.90:0/2745086174 >> 172.21.15.90:0/2745086174 conn(0x7fd16c0b17b0 msgr2=0x7fd16c1531c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:40.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.274+0000 7fd173200640 1 -- 172.21.15.90:0/2745086174 shutdown_connections 2024-04-25T01:18:40.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.274+0000 7fd173200640 1 -- 172.21.15.90:0/2745086174 wait complete. 2024-04-25T01:18:40.276 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:18:40.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.499+0000 7f847a800640 1 Processor -- start 2024-04-25T01:18:40.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.499+0000 7f847a800640 1 -- start start 2024-04-25T01:18:40.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.501+0000 7f847a800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f84740d03d0 0x7f8474166a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:40.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.501+0000 7f847a800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84740b4df0 con 0x7f847415bcd0 2024-04-25T01:18:40.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.501+0000 7f847a800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84740b4b10 con 0x7f847415cd60 2024-04-25T01:18:40.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.501+0000 7f847a800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f847415c8d0 con 0x7f84740d03d0 2024-04-25T01:18:40.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.501+0000 7f8479400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f847415bcd0 0x7f84740cc230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41204/0 (socket says 172.21.15.90:41204) 2024-04-25T01:18:40.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.501+0000 7f8479400640 1 -- 172.21.15.90:0/3120734884 learned_addr learned my addr 172.21.15.90:0/3120734884 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:40.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.501+0000 7f8479e00640 1 --2- 172.21.15.90:0/3120734884 >> v2:172.21.15.90:3300/0 conn(0x7f84740d03d0 0x7f8474166a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:40.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.501+0000 7f8473e00640 1 -- 172.21.15.90:0/3120734884 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3982518571 0 0) 0x7f84740b4df0 con 0x7f847415bcd0 2024-04-25T01:18:40.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.502+0000 7f8479e00640 1 -- 172.21.15.90:0/3120734884 >> v1:172.21.15.103:6789/0 conn(0x7f847415cd60 legacy=0x7f84740cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:40.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.502+0000 7f8479e00640 1 -- 172.21.15.90:0/3120734884 >> v1:172.21.15.90:6789/0 conn(0x7f847415bcd0 legacy=0x7f84740cc230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:40.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.502+0000 7f8479e00640 1 -- 172.21.15.90:0/3120734884 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f847415c280 con 0x7f84740d03d0 2024-04-25T01:18:40.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.503+0000 7f8479e00640 1 --2- 172.21.15.90:0/3120734884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84740d03d0 0x7f8474166a80 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f846c009dc0 tx=0x7f846c02a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb4b1b04928cf0b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:40.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.503+0000 7f8473e00640 1 -- 172.21.15.90:0/3120734884 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f846c007740 con 0x7f84740d03d0 2024-04-25T01:18:40.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.503+0000 7f8473e00640 1 -- 172.21.15.90:0/3120734884 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f846c0078e0 con 0x7f84740d03d0 2024-04-25T01:18:40.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.503+0000 7f8473e00640 1 -- 172.21.15.90:0/3120734884 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f846c007cb0 con 0x7f84740d03d0 2024-04-25T01:18:40.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.504+0000 7f847a800640 1 -- 172.21.15.90:0/3120734884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84740d03d0 msgr2=0x7f8474166a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:40.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.504+0000 7f847a800640 1 --2- 172.21.15.90:0/3120734884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84740d03d0 0x7f8474166a80 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f846c009dc0 tx=0x7f846c02a6d0 comp rx=0 tx=0).stop 2024-04-25T01:18:40.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.504+0000 7f847a800640 1 -- 172.21.15.90:0/3120734884 shutdown_connections 2024-04-25T01:18:40.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.504+0000 7f847a800640 1 --2- 172.21.15.90:0/3120734884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84740d03d0 0x7f8474166a80 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:40.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.504+0000 7f847a800640 1 -- 172.21.15.90:0/3120734884 >> 172.21.15.90:0/3120734884 conn(0x7f847406b560 msgr2=0x7f847406b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:40.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.504+0000 7f847a800640 1 -- 172.21.15.90:0/3120734884 shutdown_connections 2024-04-25T01:18:40.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.504+0000 7f847a800640 1 -- 172.21.15.90:0/3120734884 wait complete. 2024-04-25T01:18:40.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.505+0000 7f847a800640 1 Processor -- start 2024-04-25T01:18:40.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.506+0000 7f847a800640 1 -- start start 2024-04-25T01:18:40.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.506+0000 7f847a800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84740d03d0 0x7f84741547a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:40.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.506+0000 7f847a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f847415bcd0 0x7f84741412a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:40.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.506+0000 7f847a800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f847415cd60 0x7f8474164c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:40.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.506+0000 7f847a800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f84740b4790 con 0x7f847415bcd0 2024-04-25T01:18:40.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.506+0000 7f847a800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f847406e590 con 0x7f847415cd60 2024-04-25T01:18:40.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.506+0000 7f847a800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8474166fc0 con 0x7f84740d03d0 2024-04-25T01:18:40.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.507+0000 7f8478a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f847415bcd0 0x7f84741412a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:40.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.507+0000 7f8478a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f847415bcd0 0x7f84741412a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51592/0 (socket says 172.21.15.90:51592) 2024-04-25T01:18:40.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.507+0000 7f8478a00640 1 -- 172.21.15.90:0/3050316888 learned_addr learned my addr 172.21.15.90:0/3050316888 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:40.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.507+0000 7f8479400640 1 --2- 172.21.15.90:0/3050316888 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84740d03d0 0x7f84741547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:40.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.507+0000 7f8479e00640 1 --2- 172.21.15.90:0/3050316888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f847415cd60 0x7f8474164c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:40.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.507+0000 7f8478a00640 1 -- 172.21.15.90:0/3050316888 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84740d03d0 msgr2=0x7f84741547a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:40.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.507+0000 7f8478a00640 1 --2- 172.21.15.90:0/3050316888 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84740d03d0 0x7f84741547a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:40.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.507+0000 7f8478a00640 1 -- 172.21.15.90:0/3050316888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f847415cd60 msgr2=0x7f8474164c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:40.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.507+0000 7f8478a00640 1 --2- 172.21.15.90:0/3050316888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f847415cd60 0x7f8474164c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:40.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.507+0000 7f8478a00640 1 -- 172.21.15.90:0/3050316888 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8474165e60 con 0x7f847415bcd0 2024-04-25T01:18:40.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.507+0000 7f8478a00640 1 --2- 172.21.15.90:0/3050316888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f847415bcd0 0x7f84741412a0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f846000c430 tx=0x7f846000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:40.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.508+0000 7f8471600640 1 -- 172.21.15.90:0/3050316888 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8460011790 con 0x7f847415bcd0 2024-04-25T01:18:40.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.508+0000 7f8471600640 1 -- 172.21.15.90:0/3050316888 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8460017070 con 0x7f847415bcd0 2024-04-25T01:18:40.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.508+0000 7f847a800640 1 -- 172.21.15.90:0/3050316888 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84741417e0 con 0x7f847415bcd0 2024-04-25T01:18:40.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.508+0000 7f8471600640 1 -- 172.21.15.90:0/3050316888 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8460011bb0 con 0x7f847415bcd0 2024-04-25T01:18:40.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.508+0000 7f847a800640 1 -- 172.21.15.90:0/3050316888 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8474165b80 con 0x7f847415bcd0 2024-04-25T01:18:40.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.509+0000 7f847a800640 1 -- 172.21.15.90:0/3050316888 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f847404f500 con 0x7f847415bcd0 2024-04-25T01:18:40.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.510+0000 7f8471600640 1 -- 172.21.15.90:0/3050316888 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f846001c460 con 0x7f847415bcd0 2024-04-25T01:18:40.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.511+0000 7f8471600640 1 --2- 172.21.15.90:0/3050316888 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f845406be30 0x7f845406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:40.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.511+0000 7f8471600640 1 -- 172.21.15.90:0/3050316888 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f846008b1c0 con 0x7f847415bcd0 2024-04-25T01:18:40.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.511+0000 7f8479400640 1 --2- 172.21.15.90:0/3050316888 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f845406be30 0x7f845406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:40.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.512+0000 7f8479400640 1 --2- 172.21.15.90:0/3050316888 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f845406be30 0x7f845406e2f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8468000f60 tx=0x7f8468009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:40.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.515+0000 7f8471600640 1 -- 172.21.15.90:0/3050316888 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f84600596f0 con 0x7f847415bcd0 2024-04-25T01:18:40.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.709+0000 7f847a800640 1 -- 172.21.15.90:0/3050316888 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f84740cf030 con 0x7f847415bcd0 2024-04-25T01:18:40.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.710+0000 7f8471600640 1 -- 172.21.15.90:0/3050316888 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v50) ==== 104+0+11 (secure 0 0 0) 0x7f8460058ff0 con 0x7f847415bcd0 2024-04-25T01:18:40.693 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:18:40.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.712+0000 7f847a800640 1 -- 172.21.15.90:0/3050316888 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f845406be30 msgr2=0x7f845406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:40.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.712+0000 7f847a800640 1 --2- 172.21.15.90:0/3050316888 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f845406be30 0x7f845406e2f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8468000f60 tx=0x7f8468009040 comp rx=0 tx=0).stop 2024-04-25T01:18:40.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.712+0000 7f847a800640 1 -- 172.21.15.90:0/3050316888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f847415bcd0 msgr2=0x7f84741412a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:40.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.712+0000 7f847a800640 1 --2- 172.21.15.90:0/3050316888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f847415bcd0 0x7f84741412a0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f846000c430 tx=0x7f846000c8f0 comp rx=0 tx=0).stop 2024-04-25T01:18:40.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.712+0000 7f847a800640 1 -- 172.21.15.90:0/3050316888 shutdown_connections 2024-04-25T01:18:40.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.712+0000 7f847a800640 1 --2- 172.21.15.90:0/3050316888 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f845406be30 0x7f845406e2f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:40.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.712+0000 7f847a800640 1 --2- 172.21.15.90:0/3050316888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f847415cd60 0x7f8474164c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:40.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.712+0000 7f847a800640 1 --2- 172.21.15.90:0/3050316888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f847415bcd0 0x7f84741412a0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:40.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.712+0000 7f847a800640 1 --2- 172.21.15.90:0/3050316888 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84740d03d0 0x7f84741547a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:40.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.712+0000 7f847a800640 1 -- 172.21.15.90:0/3050316888 >> 172.21.15.90:0/3050316888 conn(0x7f847406b560 msgr2=0x7f8474153c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:40.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.712+0000 7f847a800640 1 -- 172.21.15.90:0/3050316888 shutdown_connections 2024-04-25T01:18:40.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.712+0000 7f847a800640 1 -- 172.21.15.90:0/3050316888 wait complete. 2024-04-25T01:18:40.714 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:18:40.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.919+0000 7f21af200640 1 Processor -- start 2024-04-25T01:18:40.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.919+0000 7f21af200640 1 -- start start 2024-04-25T01:18:40.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.920+0000 7f21af200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f21b015a170 0x7f21b00aec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:40.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.920+0000 7f21af200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f21b015a550 0x7f21b015a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:40.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.920+0000 7f21af200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21b015af00 0x7f21b015dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:40.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.921+0000 7f21af200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f21b00b3460 con 0x7f21b015a170 2024-04-25T01:18:40.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.921+0000 7f21af200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f21b00b3720 con 0x7f21b015a550 2024-04-25T01:18:40.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.921+0000 7f21af200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f21b006e100 con 0x7f21b015af00 2024-04-25T01:18:40.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.921+0000 7f21ade00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f21b015a170 0x7f21b00aec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:40.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.921+0000 7f21ade00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f21b015a170 0x7f21b00aec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51598/0 (socket says 172.21.15.90:51598) 2024-04-25T01:18:40.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.921+0000 7f21ade00640 1 -- 172.21.15.90:0/1896946367 learned_addr learned my addr 172.21.15.90:0/1896946367 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:40.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.921+0000 7f21a7e00640 1 --2- 172.21.15.90:0/1896946367 >> v2:172.21.15.103:3300/0 conn(0x7f21b015a550 0x7f21b015a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:40.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.921+0000 7f21ade00640 1 -- 172.21.15.90:0/1896946367 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21b015af00 msgr2=0x7f21b015dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:40.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.921+0000 7f21ae800640 1 --2- 172.21.15.90:0/1896946367 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21b015af00 0x7f21b015dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:40.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.921+0000 7f21ade00640 1 --2- 172.21.15.90:0/1896946367 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21b015af00 0x7f21b015dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:40.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.921+0000 7f21ade00640 1 -- 172.21.15.90:0/1896946367 >> v2:172.21.15.103:3300/0 conn(0x7f21b015a550 msgr2=0x7f21b015a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:40.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.921+0000 7f21ade00640 1 --2- 172.21.15.90:0/1896946367 >> v2:172.21.15.103:3300/0 conn(0x7f21b015a550 0x7f21b015a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:40.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.921+0000 7f21ade00640 1 -- 172.21.15.90:0/1896946367 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21b006c7f0 con 0x7f21b015a170 2024-04-25T01:18:40.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.921+0000 7f21a7e00640 1 --2- 172.21.15.90:0/1896946367 >> v2:172.21.15.103:3300/0 conn(0x7f21b015a550 0x7f21b015a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:40.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.922+0000 7f21ade00640 1 --2- 172.21.15.90:0/1896946367 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21b015a170 0x7f21b00aec40 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f21b0072620 tx=0x7f21a002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a716c50173676f84 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:40.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.923+0000 7f21ad400640 1 -- 172.21.15.90:0/1896946367 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a0037070 con 0x7f21b015a170 2024-04-25T01:18:40.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.923+0000 7f21ad400640 1 -- 172.21.15.90:0/1896946367 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21a0032440 con 0x7f21b015a170 2024-04-25T01:18:40.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.923+0000 7f21ad400640 1 -- 172.21.15.90:0/1896946367 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a0032810 con 0x7f21b015a170 2024-04-25T01:18:40.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.923+0000 7f21af200640 1 -- 172.21.15.90:0/1896946367 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21b015a170 msgr2=0x7f21b00aec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:40.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.923+0000 7f21af200640 1 --2- 172.21.15.90:0/1896946367 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21b015a170 0x7f21b00aec40 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f21b0072620 tx=0x7f21a002a270 comp rx=0 tx=0).stop 2024-04-25T01:18:40.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.923+0000 7f21af200640 1 -- 172.21.15.90:0/1896946367 shutdown_connections 2024-04-25T01:18:40.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.923+0000 7f21af200640 1 --2- 172.21.15.90:0/1896946367 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21b015af00 0x7f21b015dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:40.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.923+0000 7f21af200640 1 --2- 172.21.15.90:0/1896946367 >> v2:172.21.15.103:3300/0 conn(0x7f21b015a550 0x7f21b015a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:40.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.923+0000 7f21af200640 1 --2- 172.21.15.90:0/1896946367 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21b015a170 0x7f21b00aec40 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:40.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.923+0000 7f21af200640 1 -- 172.21.15.90:0/1896946367 >> 172.21.15.90:0/1896946367 conn(0x7f21b006b560 msgr2=0x7f21b006b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:40.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.923+0000 7f21af200640 1 -- 172.21.15.90:0/1896946367 shutdown_connections 2024-04-25T01:18:40.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.924+0000 7f21af200640 1 -- 172.21.15.90:0/1896946367 wait complete. 2024-04-25T01:18:40.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.925+0000 7f21af200640 1 Processor -- start 2024-04-25T01:18:40.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.925+0000 7f21af200640 1 -- start start 2024-04-25T01:18:40.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.926+0000 7f21af200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21b015a170 0x7f21b00ab210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:40.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.926+0000 7f21af200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21b015a550 0x7f21b00ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:40.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.926+0000 7f21af200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21b015af00 0x7f21b014d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:40.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.926+0000 7f21af200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f21b015b5b0 con 0x7f21b015af00 2024-04-25T01:18:40.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.926+0000 7f21af200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f21b00b3a00 con 0x7f21b015a170 2024-04-25T01:18:40.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.926+0000 7f21af200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f21b00abdb0 con 0x7f21b015a550 2024-04-25T01:18:40.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.926+0000 7f21ae800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21b015af00 0x7f21b014d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:40.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.926+0000 7f21ae800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21b015af00 0x7f21b014d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51610/0 (socket says 172.21.15.90:51610) 2024-04-25T01:18:40.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.926+0000 7f21ae800640 1 -- 172.21.15.90:0/622986764 learned_addr learned my addr 172.21.15.90:0/622986764 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:40.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.927+0000 7f21ae800640 1 -- 172.21.15.90:0/622986764 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21b015a550 msgr2=0x7f21b00ae180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:40.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.927+0000 7f21ae800640 1 --2- 172.21.15.90:0/622986764 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21b015a550 0x7f21b00ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:40.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.927+0000 7f21ae800640 1 -- 172.21.15.90:0/622986764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21b015a170 msgr2=0x7f21b00ab210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:40.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.927+0000 7f21ae800640 1 --2- 172.21.15.90:0/622986764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21b015a170 0x7f21b00ab210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:40.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.927+0000 7f21ae800640 1 -- 172.21.15.90:0/622986764 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21b00ab7f0 con 0x7f21b015af00 2024-04-25T01:18:40.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.927+0000 7f21ae800640 1 --2- 172.21.15.90:0/622986764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21b015af00 0x7f21b014d050 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f21a800cc30 tx=0x7f21a800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:40.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.927+0000 7f21a6000640 1 -- 172.21.15.90:0/622986764 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a800ce40 con 0x7f21b015af00 2024-04-25T01:18:40.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.927+0000 7f21a6000640 1 -- 172.21.15.90:0/622986764 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21a8003990 con 0x7f21b015af00 2024-04-25T01:18:40.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.927+0000 7f21a6000640 1 -- 172.21.15.90:0/622986764 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a8003d40 con 0x7f21b015af00 2024-04-25T01:18:40.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.927+0000 7f21af200640 1 -- 172.21.15.90:0/622986764 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21b00abad0 con 0x7f21b015af00 2024-04-25T01:18:40.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.927+0000 7f21af200640 1 -- 172.21.15.90:0/622986764 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21b00ae6c0 con 0x7f21b015af00 2024-04-25T01:18:40.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.929+0000 7f21af200640 1 -- 172.21.15.90:0/622986764 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21b004f500 con 0x7f21b015af00 2024-04-25T01:18:40.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.929+0000 7f21a6000640 1 -- 172.21.15.90:0/622986764 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f21a8003ee0 con 0x7f21b015af00 2024-04-25T01:18:40.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.930+0000 7f21a6000640 1 --2- 172.21.15.90:0/622986764 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f218c06be30 0x7f218c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:40.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.930+0000 7f21a6000640 1 -- 172.21.15.90:0/622986764 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f21a808b730 con 0x7f21b015af00 2024-04-25T01:18:40.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.931+0000 7f21ade00640 1 --2- 172.21.15.90:0/622986764 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f218c06be30 0x7f218c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:40.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.931+0000 7f21ade00640 1 --2- 172.21.15.90:0/622986764 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f218c06be30 0x7f218c06e2f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f21a0034000 tx=0x7f21a002c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:40.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:40.935+0000 7f21a6000640 1 -- 172.21.15.90:0/622986764 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f21a8014030 con 0x7f21b015af00 2024-04-25T01:18:41.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.122+0000 7f21af200640 1 -- 172.21.15.90:0/622986764 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f21b00cba40 con 0x7f21b015af00 2024-04-25T01:18:41.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.123+0000 7f21a6000640 1 -- 172.21.15.90:0/622986764 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v50) ==== 100+0+11 (secure 0 0 0) 0x7f21a8059cf0 con 0x7f21b015af00 2024-04-25T01:18:41.106 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-25T01:18:41.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.126+0000 7f21af200640 1 -- 172.21.15.90:0/622986764 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f218c06be30 msgr2=0x7f218c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:41.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.126+0000 7f21af200640 1 --2- 172.21.15.90:0/622986764 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f218c06be30 0x7f218c06e2f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f21a0034000 tx=0x7f21a002c000 comp rx=0 tx=0).stop 2024-04-25T01:18:41.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.126+0000 7f21af200640 1 -- 172.21.15.90:0/622986764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21b015af00 msgr2=0x7f21b014d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:41.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.126+0000 7f21af200640 1 --2- 172.21.15.90:0/622986764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21b015af00 0x7f21b014d050 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f21a800cc30 tx=0x7f21a800a5c0 comp rx=0 tx=0).stop 2024-04-25T01:18:41.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.126+0000 7f21af200640 1 -- 172.21.15.90:0/622986764 shutdown_connections 2024-04-25T01:18:41.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.126+0000 7f21af200640 1 --2- 172.21.15.90:0/622986764 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f218c06be30 0x7f218c06e2f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:41.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.126+0000 7f21af200640 1 --2- 172.21.15.90:0/622986764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21b015af00 0x7f21b014d050 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:41.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.126+0000 7f21af200640 1 --2- 172.21.15.90:0/622986764 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f21b015a550 0x7f21b00ae180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:41.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.126+0000 7f21af200640 1 --2- 172.21.15.90:0/622986764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21b015a170 0x7f21b00ab210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:41.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.126+0000 7f21af200640 1 -- 172.21.15.90:0/622986764 >> 172.21.15.90:0/622986764 conn(0x7f21b006b560 msgr2=0x7f21b015d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:41.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.126+0000 7f21af200640 1 -- 172.21.15.90:0/622986764 shutdown_connections 2024-04-25T01:18:41.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.126+0000 7f21af200640 1 -- 172.21.15.90:0/622986764 wait complete. 2024-04-25T01:18:41.127 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get recovery_fs_metadata pg_num 2024-04-25T01:18:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.334+0000 7f9f8b200640 1 Processor -- start 2024-04-25T01:18:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.334+0000 7f9f8b200640 1 -- start start 2024-04-25T01:18:41.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.336+0000 7f9f8b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f8c158300 0x7f9f8c0cfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:41.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.336+0000 7f9f8b200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9f8c162580 0x7f9f8c0d0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:41.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.336+0000 7f9f8b200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f8c15f070 con 0x7f9f8c15e310 2024-04-25T01:18:41.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.336+0000 7f9f8b200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f9f8c15ee50 con 0x7f9f8c162580 2024-04-25T01:18:41.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.336+0000 7f9f8b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9f8c0b2ae0 con 0x7f9f8c158300 2024-04-25T01:18:41.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.336+0000 7f9f89400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9f8c162580 0x7f9f8c0d0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:41.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.336+0000 7f9f89400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9f8c162580 0x7f9f8c0d0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51614/0 (socket says 172.21.15.90:51614) 2024-04-25T01:18:41.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.336+0000 7f9f89400640 1 -- 172.21.15.90:0/2877782131 learned_addr learned my addr 172.21.15.90:0/2877782131 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:41.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.336+0000 7f9f89400640 1 -- 172.21.15.90:0/2877782131 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f8c158300 msgr2=0x7f9f8c0cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:41.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.337+0000 7f9f89e00640 1 --2- 172.21.15.90:0/2877782131 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f8c158300 0x7f9f8c0cfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:41.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.337+0000 7f9f89400640 1 --2- 172.21.15.90:0/2877782131 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f8c158300 0x7f9f8c0cfcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:41.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.337+0000 7f9f89400640 1 -- 172.21.15.90:0/2877782131 >> v1:172.21.15.90:6789/0 conn(0x7f9f8c15e310 legacy=0x7f9f8c15e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:41.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.337+0000 7f9f89400640 1 -- 172.21.15.90:0/2877782131 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f8c161aa0 con 0x7f9f8c162580 2024-04-25T01:18:41.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.337+0000 7f9f89e00640 1 --2- 172.21.15.90:0/2877782131 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f8c158300 0x7f9f8c0cfcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:41.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.337+0000 7f9f88a00640 1 -- 172.21.15.90:0/2877782131 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2128661212 0 0) 0x7f9f8c15f070 con 0x7f9f8c15e310 2024-04-25T01:18:41.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.338+0000 7f9f89400640 1 --2- 172.21.15.90:0/2877782131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f8c162580 0x7f9f8c0d0200 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f9f78009870 tx=0x7f9f7802a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ab8119b2a3868fb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:41.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.338+0000 7f9f88a00640 1 -- 172.21.15.90:0/2877782131 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f78037070 con 0x7f9f8c162580 2024-04-25T01:18:41.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.338+0000 7f9f88a00640 1 -- 172.21.15.90:0/2877782131 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f7802ae50 con 0x7f9f8c162580 2024-04-25T01:18:41.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.338+0000 7f9f88a00640 1 -- 172.21.15.90:0/2877782131 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f78032860 con 0x7f9f8c162580 2024-04-25T01:18:41.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.339+0000 7f9f8b200640 1 -- 172.21.15.90:0/2877782131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f8c162580 msgr2=0x7f9f8c0d0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:41.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.339+0000 7f9f8b200640 1 --2- 172.21.15.90:0/2877782131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f8c162580 0x7f9f8c0d0200 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f9f78009870 tx=0x7f9f7802a300 comp rx=0 tx=0).stop 2024-04-25T01:18:41.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.339+0000 7f9f8b200640 1 -- 172.21.15.90:0/2877782131 shutdown_connections 2024-04-25T01:18:41.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.339+0000 7f9f8b200640 1 --2- 172.21.15.90:0/2877782131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f8c162580 0x7f9f8c0d0200 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:41.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.339+0000 7f9f8b200640 1 --2- 172.21.15.90:0/2877782131 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f8c158300 0x7f9f8c0cfcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:41.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.339+0000 7f9f8b200640 1 -- 172.21.15.90:0/2877782131 >> 172.21.15.90:0/2877782131 conn(0x7f9f8c06b560 msgr2=0x7f9f8c06b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:41.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.339+0000 7f9f8b200640 1 -- 172.21.15.90:0/2877782131 shutdown_connections 2024-04-25T01:18:41.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.339+0000 7f9f8b200640 1 -- 172.21.15.90:0/2877782131 wait complete. 2024-04-25T01:18:41.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.341+0000 7f9f8b200640 1 Processor -- start 2024-04-25T01:18:41.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.341+0000 7f9f8b200640 1 -- start start 2024-04-25T01:18:41.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.342+0000 7f9f8b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f8c158300 0x7f9f8c1516f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:41.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.342+0000 7f9f8b200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f8c15e310 0x7f9f8c15faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:41.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.342+0000 7f9f8b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f8c162580 0x7f9f8c15d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:41.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.342+0000 7f9f8b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8c1586e0 con 0x7f9f8c162580 2024-04-25T01:18:41.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.342+0000 7f9f8b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8c162060 con 0x7f9f8c158300 2024-04-25T01:18:41.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.342+0000 7f9f8b200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9f8c06cab0 con 0x7f9f8c15e310 2024-04-25T01:18:41.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.342+0000 7f9f8a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f8c162580 0x7f9f8c15d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:41.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.342+0000 7f9f8a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f8c162580 0x7f9f8c15d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51622/0 (socket says 172.21.15.90:51622) 2024-04-25T01:18:41.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.342+0000 7f9f8a800640 1 -- 172.21.15.90:0/3748027050 learned_addr learned my addr 172.21.15.90:0/3748027050 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:41.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.342+0000 7f9f89e00640 1 --2- 172.21.15.90:0/3748027050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f8c158300 0x7f9f8c1516f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:41.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.342+0000 7f9f8a800640 1 -- 172.21.15.90:0/3748027050 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f8c15e310 msgr2=0x7f9f8c15faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:41.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.342+0000 7f9f8a800640 1 --2- 172.21.15.90:0/3748027050 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f8c15e310 0x7f9f8c15faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:41.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.342+0000 7f9f8a800640 1 -- 172.21.15.90:0/3748027050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f8c158300 msgr2=0x7f9f8c1516f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:41.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.342+0000 7f9f8a800640 1 --2- 172.21.15.90:0/3748027050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f8c158300 0x7f9f8c1516f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:41.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.342+0000 7f9f8a800640 1 -- 172.21.15.90:0/3748027050 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f8c15bf10 con 0x7f9f8c162580 2024-04-25T01:18:41.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.343+0000 7f9f8a800640 1 --2- 172.21.15.90:0/3748027050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f8c162580 0x7f9f8c15d290 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f9f8400c760 tx=0x7f9f8400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:41.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.343+0000 7f9f82000640 1 -- 172.21.15.90:0/3748027050 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f84003da0 con 0x7f9f8c162580 2024-04-25T01:18:41.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.343+0000 7f9f82000640 1 -- 172.21.15.90:0/3748027050 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f84005070 con 0x7f9f8c162580 2024-04-25T01:18:41.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.343+0000 7f9f8b200640 1 -- 172.21.15.90:0/3748027050 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f8c15c1f0 con 0x7f9f8c162580 2024-04-25T01:18:41.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.343+0000 7f9f82000640 1 -- 172.21.15.90:0/3748027050 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f84005460 con 0x7f9f8c162580 2024-04-25T01:18:41.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.343+0000 7f9f8b200640 1 -- 172.21.15.90:0/3748027050 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f8c15c7b0 con 0x7f9f8c162580 2024-04-25T01:18:41.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.344+0000 7f9f8b200640 1 -- 172.21.15.90:0/3748027050 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f8c04f500 con 0x7f9f8c162580 2024-04-25T01:18:41.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.345+0000 7f9f82000640 1 -- 172.21.15.90:0/3748027050 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9f84005600 con 0x7f9f8c162580 2024-04-25T01:18:41.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.346+0000 7f9f82000640 1 --2- 172.21.15.90:0/3748027050 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9f6c06be30 0x7f9f6c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:41.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.346+0000 7f9f82000640 1 -- 172.21.15.90:0/3748027050 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f9f8408b100 con 0x7f9f8c162580 2024-04-25T01:18:41.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.346+0000 7f9f89e00640 1 --2- 172.21.15.90:0/3748027050 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9f6c06be30 0x7f9f6c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:41.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.347+0000 7f9f89e00640 1 --2- 172.21.15.90:0/3748027050 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9f6c06be30 0x7f9f6c06e2f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c008850 tx=0x7f9f7c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:41.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.351+0000 7f9f82000640 1 -- 172.21.15.90:0/3748027050 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9f84059760 con 0x7f9f8c162580 2024-04-25T01:18:41.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.536+0000 7f9f8b200640 1 -- 172.21.15.90:0/3748027050 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "recovery_fs_metadata", "var": "pg_num"} v 0) -- 0x7f9f8c0b4dd0 con 0x7f9f8c162580 2024-04-25T01:18:41.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.537+0000 7f9f82000640 1 -- 172.21.15.90:0/3748027050 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "recovery_fs_metadata", "var": "pg_num"}]=0 v50) ==== 109+0+11 (secure 0 0 0) 0x7f9f84059060 con 0x7f9f8c162580 2024-04-25T01:18:41.520 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:18:41.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.539+0000 7f9f8b200640 1 -- 172.21.15.90:0/3748027050 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9f6c06be30 msgr2=0x7f9f6c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:41.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.539+0000 7f9f8b200640 1 --2- 172.21.15.90:0/3748027050 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9f6c06be30 0x7f9f6c06e2f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c008850 tx=0x7f9f7c008040 comp rx=0 tx=0).stop 2024-04-25T01:18:41.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.539+0000 7f9f8b200640 1 -- 172.21.15.90:0/3748027050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f8c162580 msgr2=0x7f9f8c15d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:41.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.539+0000 7f9f8b200640 1 --2- 172.21.15.90:0/3748027050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f8c162580 0x7f9f8c15d290 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f9f8400c760 tx=0x7f9f8400cc20 comp rx=0 tx=0).stop 2024-04-25T01:18:41.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.539+0000 7f9f8b200640 1 -- 172.21.15.90:0/3748027050 shutdown_connections 2024-04-25T01:18:41.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.539+0000 7f9f8b200640 1 --2- 172.21.15.90:0/3748027050 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9f6c06be30 0x7f9f6c06e2f0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:41.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.539+0000 7f9f8b200640 1 --2- 172.21.15.90:0/3748027050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f8c162580 0x7f9f8c15d290 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:41.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.539+0000 7f9f8b200640 1 --2- 172.21.15.90:0/3748027050 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9f8c15e310 0x7f9f8c15faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:41.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.539+0000 7f9f8b200640 1 --2- 172.21.15.90:0/3748027050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f8c158300 0x7f9f8c1516f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:41.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.539+0000 7f9f8b200640 1 -- 172.21.15.90:0/3748027050 >> 172.21.15.90:0/3748027050 conn(0x7f9f8c06b560 msgr2=0x7f9f8c0b2150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:41.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.540+0000 7f9f8b200640 1 -- 172.21.15.90:0/3748027050 shutdown_connections 2024-04-25T01:18:41.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.540+0000 7f9f8b200640 1 -- 172.21.15.90:0/3748027050 wait complete. 2024-04-25T01:18:41.540 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:18:41.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.748+0000 7f1e38c00640 1 Processor -- start 2024-04-25T01:18:41.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.748+0000 7f1e38c00640 1 -- start start 2024-04-25T01:18:41.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.749+0000 7f1e38c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1e341445e0 0x7f1e340cfc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:41.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.749+0000 7f1e38c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1e34071320 0x7f1e340d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:41.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.749+0000 7f1e38c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e341449c0 con 0x7f1e340728c0 2024-04-25T01:18:41.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.749+0000 7f1e38c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f1e34072430 con 0x7f1e34071320 2024-04-25T01:18:41.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.749+0000 7f1e38c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1e340b2d70 con 0x7f1e341445e0 2024-04-25T01:18:41.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.750+0000 7f1e29400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f1e340728c0 0x7f1e34070c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41216/0 (socket says 172.21.15.90:41216) 2024-04-25T01:18:41.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.750+0000 7f1e29400640 1 -- 172.21.15.90:0/3305153410 learned_addr learned my addr 172.21.15.90:0/3305153410 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:41.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.750+0000 7f1e32800640 1 --2- 172.21.15.90:0/3305153410 >> v2:172.21.15.90:3300/0 conn(0x7f1e34071320 0x7f1e340d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:41.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.750+0000 7f1e31e00640 1 --2- 172.21.15.90:0/3305153410 >> v2:172.21.15.103:3300/0 conn(0x7f1e341445e0 0x7f1e340cfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:41.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.750+0000 7f1e31400640 1 -- 172.21.15.90:0/3305153410 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4217459691 0 0) 0x7f1e341449c0 con 0x7f1e340728c0 2024-04-25T01:18:41.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.750+0000 7f1e31400640 1 -- 172.21.15.90:0/3305153410 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e18003520 con 0x7f1e340728c0 2024-04-25T01:18:41.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.750+0000 7f1e32800640 1 -- 172.21.15.90:0/3305153410 >> v2:172.21.15.103:3300/0 conn(0x7f1e341445e0 msgr2=0x7f1e340cfc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:41.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.750+0000 7f1e32800640 1 --2- 172.21.15.90:0/3305153410 >> v2:172.21.15.103:3300/0 conn(0x7f1e341445e0 0x7f1e340cfc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:41.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.750+0000 7f1e32800640 1 -- 172.21.15.90:0/3305153410 >> v1:172.21.15.90:6789/0 conn(0x7f1e340728c0 legacy=0x7f1e34070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:41.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.750+0000 7f1e32800640 1 -- 172.21.15.90:0/3305153410 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e34071820 con 0x7f1e34071320 2024-04-25T01:18:41.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.751+0000 7f1e32800640 1 --2- 172.21.15.90:0/3305153410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e34071320 0x7f1e340d01d0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f1e340d08a0 tx=0x7f1e2c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0041c2fdb9f258e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:41.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.751+0000 7f1e31400640 1 -- 172.21.15.90:0/3305153410 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e2c037070 con 0x7f1e34071320 2024-04-25T01:18:41.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.752+0000 7f1e31400640 1 -- 172.21.15.90:0/3305153410 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e2c032440 con 0x7f1e34071320 2024-04-25T01:18:41.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.752+0000 7f1e31400640 1 -- 172.21.15.90:0/3305153410 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e2c032810 con 0x7f1e34071320 2024-04-25T01:18:41.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.752+0000 7f1e38c00640 1 -- 172.21.15.90:0/3305153410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e34071320 msgr2=0x7f1e340d01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:41.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.752+0000 7f1e38c00640 1 --2- 172.21.15.90:0/3305153410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e34071320 0x7f1e340d01d0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f1e340d08a0 tx=0x7f1e2c02a1a0 comp rx=0 tx=0).stop 2024-04-25T01:18:41.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.752+0000 7f1e38c00640 1 -- 172.21.15.90:0/3305153410 shutdown_connections 2024-04-25T01:18:41.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.752+0000 7f1e38c00640 1 --2- 172.21.15.90:0/3305153410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e34071320 0x7f1e340d01d0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:41.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.752+0000 7f1e38c00640 1 --2- 172.21.15.90:0/3305153410 >> v2:172.21.15.103:3300/0 conn(0x7f1e341445e0 0x7f1e340cfc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:41.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.752+0000 7f1e38c00640 1 -- 172.21.15.90:0/3305153410 >> 172.21.15.90:0/3305153410 conn(0x7f1e3406b320 msgr2=0x7f1e3406b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:41.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.752+0000 7f1e38c00640 1 -- 172.21.15.90:0/3305153410 shutdown_connections 2024-04-25T01:18:41.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.752+0000 7f1e38c00640 1 -- 172.21.15.90:0/3305153410 wait complete. 2024-04-25T01:18:41.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.754+0000 7f1e38c00640 1 Processor -- start 2024-04-25T01:18:41.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.754+0000 7f1e38c00640 1 -- start start 2024-04-25T01:18:41.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.755+0000 7f1e38c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e34071320 0x7f1e34165c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:41.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.755+0000 7f1e38c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e340728c0 0x7f1e3415aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:41.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.755+0000 7f1e38c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e341445e0 0x7f1e340ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:41.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.755+0000 7f1e38c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1e340720c0 con 0x7f1e340728c0 2024-04-25T01:18:41.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.755+0000 7f1e38c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1e34071de0 con 0x7f1e341445e0 2024-04-25T01:18:41.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.755+0000 7f1e38c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1e3406cbc0 con 0x7f1e34071320 2024-04-25T01:18:41.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.755+0000 7f1e29400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e340728c0 0x7f1e3415aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:41.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.755+0000 7f1e29400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e340728c0 0x7f1e3415aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51650/0 (socket says 172.21.15.90:51650) 2024-04-25T01:18:41.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.755+0000 7f1e29400640 1 -- 172.21.15.90:0/2546828283 learned_addr learned my addr 172.21.15.90:0/2546828283 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:41.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.756+0000 7f1e29400640 1 -- 172.21.15.90:0/2546828283 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e34071320 msgr2=0x7f1e34165c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:41.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.756+0000 7f1e29400640 1 --2- 172.21.15.90:0/2546828283 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e34071320 0x7f1e34165c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:41.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.756+0000 7f1e29400640 1 -- 172.21.15.90:0/2546828283 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e341445e0 msgr2=0x7f1e340ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:41.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.756+0000 7f1e32800640 1 --2- 172.21.15.90:0/2546828283 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e341445e0 0x7f1e340ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:41.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.756+0000 7f1e29400640 1 --2- 172.21.15.90:0/2546828283 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e341445e0 0x7f1e340ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:41.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.756+0000 7f1e29400640 1 -- 172.21.15.90:0/2546828283 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e34166220 con 0x7f1e340728c0 2024-04-25T01:18:41.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.756+0000 7f1e29400640 1 --2- 172.21.15.90:0/2546828283 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e340728c0 0x7f1e3415aa20 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c00cbe0 tx=0x7f1e1c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:41.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.756+0000 7f1e2aa00640 1 -- 172.21.15.90:0/2546828283 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c017070 con 0x7f1e340728c0 2024-04-25T01:18:41.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.756+0000 7f1e2aa00640 1 -- 172.21.15.90:0/2546828283 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e1c00ad60 con 0x7f1e340728c0 2024-04-25T01:18:41.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.756+0000 7f1e2aa00640 1 -- 172.21.15.90:0/2546828283 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c012640 con 0x7f1e340728c0 2024-04-25T01:18:41.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.756+0000 7f1e38c00640 1 -- 172.21.15.90:0/2546828283 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e34166500 con 0x7f1e340728c0 2024-04-25T01:18:41.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.756+0000 7f1e38c00640 1 -- 172.21.15.90:0/2546828283 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e34166ac0 con 0x7f1e340728c0 2024-04-25T01:18:41.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.758+0000 7f1e38c00640 1 -- 172.21.15.90:0/2546828283 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e34068190 con 0x7f1e340728c0 2024-04-25T01:18:41.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.758+0000 7f1e2aa00640 1 -- 172.21.15.90:0/2546828283 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1e1c01a070 con 0x7f1e340728c0 2024-04-25T01:18:41.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.759+0000 7f1e2aa00640 1 --2- 172.21.15.90:0/2546828283 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1e1006be30 0x7f1e1006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:41.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.759+0000 7f1e2aa00640 1 -- 172.21.15.90:0/2546828283 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f1e1c08b120 con 0x7f1e340728c0 2024-04-25T01:18:41.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.760+0000 7f1e31e00640 1 --2- 172.21.15.90:0/2546828283 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1e1006be30 0x7f1e1006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:41.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.760+0000 7f1e31e00640 1 --2- 172.21.15.90:0/2546828283 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1e1006be30 0x7f1e1006e2f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1e24006fd0 tx=0x7f1e24008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:41.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.764+0000 7f1e2aa00640 1 -- 172.21.15.90:0/2546828283 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1e1c059660 con 0x7f1e340728c0 2024-04-25T01:18:41.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.992+0000 7f1e38c00640 1 -- 172.21.15.90:0/2546828283 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e340d04f0 con 0x7f1e340728c0 2024-04-25T01:18:41.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.993+0000 7f1e2aa00640 1 -- 172.21.15.90:0/2546828283 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6492 (secure 0 0 0) 0x7f1e1c058f60 con 0x7f1e340728c0 2024-04-25T01:18:41.977 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:41.977 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1873497946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6833","nonce":1873497946},{"type":"v1","addr":"172.21.15.103:6835","nonce":1873497946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/3206865830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":3206865830},{"type":"v1","addr":"172.21.15.90:6835","nonce":3206865830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5598,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2208594231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2208594231},{"type":"v1","addr":"172.21.15.90:6837","nonce":2208594231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:18:02.701652+0000","modified":"2024-04-25T01:18:15.603132+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2},{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:18:19.371011+0000","modified":"2024-04-25T01:18:30.533899+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5601":{"gid":5601,"name":"d","rank":0,"incarnation":25,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6834/2850373997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":2850373997},{"type":"v1","addr":"172.21.15.103:6834","nonce":2850373997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":7,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5601,"qdb_cluster":[5601]},"id":3}]} 2024-04-25T01:18:41.977 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 28 2024-04-25T01:18:41.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.996+0000 7f1e38c00640 1 -- 172.21.15.90:0/2546828283 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1e1006be30 msgr2=0x7f1e1006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:41.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.996+0000 7f1e38c00640 1 --2- 172.21.15.90:0/2546828283 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1e1006be30 0x7f1e1006e2f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1e24006fd0 tx=0x7f1e24008040 comp rx=0 tx=0).stop 2024-04-25T01:18:41.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.996+0000 7f1e38c00640 1 -- 172.21.15.90:0/2546828283 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e340728c0 msgr2=0x7f1e3415aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:41.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.996+0000 7f1e38c00640 1 --2- 172.21.15.90:0/2546828283 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e340728c0 0x7f1e3415aa20 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c00cbe0 tx=0x7f1e1c00a430 comp rx=0 tx=0).stop 2024-04-25T01:18:41.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.996+0000 7f1e38c00640 1 -- 172.21.15.90:0/2546828283 shutdown_connections 2024-04-25T01:18:41.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.996+0000 7f1e38c00640 1 --2- 172.21.15.90:0/2546828283 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1e1006be30 0x7f1e1006e2f0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:41.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.996+0000 7f1e38c00640 1 --2- 172.21.15.90:0/2546828283 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e341445e0 0x7f1e340ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:41.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.996+0000 7f1e38c00640 1 --2- 172.21.15.90:0/2546828283 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e340728c0 0x7f1e3415aa20 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:41.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.996+0000 7f1e38c00640 1 --2- 172.21.15.90:0/2546828283 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e34071320 0x7f1e34165c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:41.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.996+0000 7f1e38c00640 1 -- 172.21.15.90:0/2546828283 >> 172.21.15.90:0/2546828283 conn(0x7f1e3406b320 msgr2=0x7f1e3406b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:41.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.996+0000 7f1e38c00640 1 -- 172.21.15.90:0/2546828283 shutdown_connections 2024-04-25T01:18:41.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:41.996+0000 7f1e38c00640 1 -- 172.21.15.90:0/2546828283 wait complete. 2024-04-25T01:18:41.997 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:18:42.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.205+0000 7f6c42200640 1 Processor -- start 2024-04-25T01:18:42.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.205+0000 7f6c42200640 1 -- start start 2024-04-25T01:18:42.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.207+0000 7f6c42200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6c3c15a980 0x7f6c3c06d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:42.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.207+0000 7f6c42200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c3c0d0240 0x7f6c3c0ccfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:42.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.207+0000 7f6c42200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c3c0ccb10 con 0x7f6c3c071dd0 2024-04-25T01:18:42.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.207+0000 7f6c42200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f6c3c072230 con 0x7f6c3c15a980 2024-04-25T01:18:42.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.207+0000 7f6c42200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6c3c0aede0 con 0x7f6c3c0d0240 2024-04-25T01:18:42.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.207+0000 7f6c40e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6c3c15a980 0x7f6c3c06d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:42.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.207+0000 7f6c40e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6c3c15a980 0x7f6c3c06d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51662/0 (socket says 172.21.15.90:51662) 2024-04-25T01:18:42.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.207+0000 7f6c40e00640 1 -- 172.21.15.90:0/3409401233 learned_addr learned my addr 172.21.15.90:0/3409401233 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:42.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.208+0000 7f6c40e00640 1 -- 172.21.15.90:0/3409401233 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c3c0d0240 msgr2=0x7f6c3c0ccfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:42.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.208+0000 7f6c40e00640 1 --2- 172.21.15.90:0/3409401233 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c3c0d0240 0x7f6c3c0ccfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:42.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.208+0000 7f6c40e00640 1 -- 172.21.15.90:0/3409401233 >> v1:172.21.15.103:6789/0 conn(0x7f6c3c071dd0 legacy=0x7f6c3c0cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:42.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.208+0000 7f6c40e00640 1 -- 172.21.15.90:0/3409401233 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c3c06ddc0 con 0x7f6c3c15a980 2024-04-25T01:18:42.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.209+0000 7f6c40e00640 1 --2- 172.21.15.90:0/3409401233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c3c15a980 0x7f6c3c06d770 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c009870 tx=0x7f6c2c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d221ec5819c2c178 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:42.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.209+0000 7f6c3b400640 1 -- 172.21.15.90:0/3409401233 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c037070 con 0x7f6c3c15a980 2024-04-25T01:18:42.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.209+0000 7f6c3b400640 1 -- 172.21.15.90:0/3409401233 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c2c032440 con 0x7f6c3c15a980 2024-04-25T01:18:42.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.209+0000 7f6c3b400640 1 -- 172.21.15.90:0/3409401233 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c032810 con 0x7f6c3c15a980 2024-04-25T01:18:42.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.210+0000 7f6c42200640 1 -- 172.21.15.90:0/3409401233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c3c15a980 msgr2=0x7f6c3c06d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:42.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.210+0000 7f6c42200640 1 --2- 172.21.15.90:0/3409401233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c3c15a980 0x7f6c3c06d770 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c009870 tx=0x7f6c2c02a180 comp rx=0 tx=0).stop 2024-04-25T01:18:42.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.210+0000 7f6c42200640 1 -- 172.21.15.90:0/3409401233 shutdown_connections 2024-04-25T01:18:42.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.210+0000 7f6c42200640 1 --2- 172.21.15.90:0/3409401233 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c3c0d0240 0x7f6c3c0ccfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:42.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.210+0000 7f6c42200640 1 --2- 172.21.15.90:0/3409401233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c3c15a980 0x7f6c3c06d770 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:42.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.210+0000 7f6c42200640 1 -- 172.21.15.90:0/3409401233 >> 172.21.15.90:0/3409401233 conn(0x7f6c3c06b320 msgr2=0x7f6c3c06b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:42.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.210+0000 7f6c42200640 1 -- 172.21.15.90:0/3409401233 shutdown_connections 2024-04-25T01:18:42.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.210+0000 7f6c42200640 1 -- 172.21.15.90:0/3409401233 wait complete. 2024-04-25T01:18:42.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.211+0000 7f6c42200640 1 Processor -- start 2024-04-25T01:18:42.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.212+0000 7f6c42200640 1 -- start start 2024-04-25T01:18:42.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.212+0000 7f6c42200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c3c071dd0 0x7f6c3c14e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:42.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.212+0000 7f6c42200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c3c0d0240 0x7f6c3c14f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:42.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.212+0000 7f6c42200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c3c15a980 0x7f6c3c0ad8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:42.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.213+0000 7f6c42200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3c06e660 con 0x7f6c3c15a980 2024-04-25T01:18:42.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.213+0000 7f6c42200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3c06e380 con 0x7f6c3c071dd0 2024-04-25T01:18:42.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.213+0000 7f6c42200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6c3c15ad60 con 0x7f6c3c0d0240 2024-04-25T01:18:42.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.213+0000 7f6c41800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c3c15a980 0x7f6c3c0ad8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:42.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.213+0000 7f6c41800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c3c15a980 0x7f6c3c0ad8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51664/0 (socket says 172.21.15.90:51664) 2024-04-25T01:18:42.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.213+0000 7f6c41800640 1 -- 172.21.15.90:0/4002633139 learned_addr learned my addr 172.21.15.90:0/4002633139 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:42.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.213+0000 7f6c41800640 1 -- 172.21.15.90:0/4002633139 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c3c0d0240 msgr2=0x7f6c3c14f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:42.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.213+0000 7f6c3be00640 1 --2- 172.21.15.90:0/4002633139 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c3c0d0240 0x7f6c3c14f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:42.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.213+0000 7f6c40e00640 1 --2- 172.21.15.90:0/4002633139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c3c071dd0 0x7f6c3c14e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:42.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.213+0000 7f6c41800640 1 --2- 172.21.15.90:0/4002633139 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c3c0d0240 0x7f6c3c14f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:42.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.213+0000 7f6c41800640 1 -- 172.21.15.90:0/4002633139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c3c071dd0 msgr2=0x7f6c3c14e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:42.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.213+0000 7f6c41800640 1 --2- 172.21.15.90:0/4002633139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c3c071dd0 0x7f6c3c14e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:42.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.213+0000 7f6c41800640 1 -- 172.21.15.90:0/4002633139 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c3c14e920 con 0x7f6c3c15a980 2024-04-25T01:18:42.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.214+0000 7f6c41800640 1 --2- 172.21.15.90:0/4002633139 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c3c15a980 0x7f6c3c0ad8c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f6c3400b420 tx=0x7f6c3400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:42.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.214+0000 7f6c38c00640 1 -- 172.21.15.90:0/4002633139 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c34011830 con 0x7f6c3c15a980 2024-04-25T01:18:42.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.214+0000 7f6c38c00640 1 -- 172.21.15.90:0/4002633139 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c34016070 con 0x7f6c3c15a980 2024-04-25T01:18:42.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.214+0000 7f6c42200640 1 -- 172.21.15.90:0/4002633139 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c3c0ae380 con 0x7f6c3c15a980 2024-04-25T01:18:42.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.214+0000 7f6c38c00640 1 -- 172.21.15.90:0/4002633139 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c34011c10 con 0x7f6c3c15a980 2024-04-25T01:18:42.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.214+0000 7f6c42200640 1 -- 172.21.15.90:0/4002633139 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c3c14d900 con 0x7f6c3c15a980 2024-04-25T01:18:42.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.215+0000 7f6c42200640 1 -- 172.21.15.90:0/4002633139 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c3c04f500 con 0x7f6c3c15a980 2024-04-25T01:18:42.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.216+0000 7f6c38c00640 1 -- 172.21.15.90:0/4002633139 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6c3401b460 con 0x7f6c3c15a980 2024-04-25T01:18:42.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.217+0000 7f6c38c00640 1 --2- 172.21.15.90:0/4002633139 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f6c1c06be60 0x7f6c1c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:42.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.217+0000 7f6c38c00640 1 -- 172.21.15.90:0/4002633139 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f6c3408af20 con 0x7f6c3c15a980 2024-04-25T01:18:42.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.217+0000 7f6c40e00640 1 --2- 172.21.15.90:0/4002633139 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f6c1c06be60 0x7f6c1c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:42.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.218+0000 7f6c40e00640 1 --2- 172.21.15.90:0/4002633139 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f6c1c06be60 0x7f6c1c06e320 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c034000 tx=0x7f6c2c02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:42.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.221+0000 7f6c38c00640 1 -- 172.21.15.90:0/4002633139 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6c34059450 con 0x7f6c3c15a980 2024-04-25T01:18:42.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.409+0000 7f6c42200640 1 -- 172.21.15.90:0/4002633139 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6c3c0b11a0 con 0x7f6c3c15a980 2024-04-25T01:18:42.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.412+0000 7f6c38c00640 1 -- 172.21.15.90:0/4002633139 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) ==== 74+0+18947 (secure 0 0 0) 0x7f6c34058d50 con 0x7f6c3c15a980 2024-04-25T01:18:42.395 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:42.395 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":50,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:18:20.389626+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:18:00.504992+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T01:18:01.714313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"recovery_fs_metadata","create_time":"2024-04-25T01:18:18.384671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.16","mappings":[{"from":4,"to":2}]},{"pgid":"6.20","mappings":[{"from":4,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:6837/431390375":"2024-04-26T01:18:15.602554+0000","172.21.15.90:6836/431390375":"2024-04-26T01:18:15.602554+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:18:42.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.414+0000 7f6c42200640 1 -- 172.21.15.90:0/4002633139 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f6c1c06be60 msgr2=0x7f6c1c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:42.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.414+0000 7f6c42200640 1 --2- 172.21.15.90:0/4002633139 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f6c1c06be60 0x7f6c1c06e320 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c034000 tx=0x7f6c2c02e000 comp rx=0 tx=0).stop 2024-04-25T01:18:42.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.414+0000 7f6c42200640 1 -- 172.21.15.90:0/4002633139 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c3c15a980 msgr2=0x7f6c3c0ad8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:42.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.414+0000 7f6c42200640 1 --2- 172.21.15.90:0/4002633139 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c3c15a980 0x7f6c3c0ad8c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f6c3400b420 tx=0x7f6c3400b8e0 comp rx=0 tx=0).stop 2024-04-25T01:18:42.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.415+0000 7f6c42200640 1 -- 172.21.15.90:0/4002633139 shutdown_connections 2024-04-25T01:18:42.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.415+0000 7f6c42200640 1 --2- 172.21.15.90:0/4002633139 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f6c1c06be60 0x7f6c1c06e320 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:42.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.415+0000 7f6c42200640 1 --2- 172.21.15.90:0/4002633139 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c3c15a980 0x7f6c3c0ad8c0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:42.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.415+0000 7f6c42200640 1 --2- 172.21.15.90:0/4002633139 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f6c3c0d0240 0x7f6c3c14f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:42.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.415+0000 7f6c42200640 1 --2- 172.21.15.90:0/4002633139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c3c071dd0 0x7f6c3c14e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:42.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.415+0000 7f6c42200640 1 -- 172.21.15.90:0/4002633139 >> 172.21.15.90:0/4002633139 conn(0x7f6c3c06b320 msgr2=0x7f6c3c0b3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:42.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.415+0000 7f6c42200640 1 -- 172.21.15.90:0/4002633139 shutdown_connections 2024-04-25T01:18:42.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.415+0000 7f6c42200640 1 -- 172.21.15.90:0/4002633139 wait complete. 2024-04-25T01:18:42.417 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T01:18:42.417 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:18:42.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.633+0000 7fcc77200640 1 Processor -- start 2024-04-25T01:18:42.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.633+0000 7fcc77200640 1 -- start start 2024-04-25T01:18:42.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.635+0000 7fcc77200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcc781561c0 0x7fcc780cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:42.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.635+0000 7fcc77200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcc780709d0 0x7fcc7815cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:42.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.635+0000 7fcc77200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc78070f90 con 0x7fcc780cfb30 2024-04-25T01:18:42.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.635+0000 7fcc77200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fcc780b4310 con 0x7fcc780709d0 2024-04-25T01:18:42.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.635+0000 7fcc77200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcc781565a0 con 0x7fcc781561c0 2024-04-25T01:18:42.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.635+0000 7fcc6d400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcc780709d0 0x7fcc7815cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:42.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.635+0000 7fcc6d400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcc780709d0 0x7fcc7815cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51680/0 (socket says 172.21.15.90:51680) 2024-04-25T01:18:42.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.635+0000 7fcc6d400640 1 -- 172.21.15.90:0/551034464 learned_addr learned my addr 172.21.15.90:0/551034464 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:42.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.635+0000 7fcc75e00640 1 --2- 172.21.15.90:0/551034464 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcc781561c0 0x7fcc780cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:42.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.636+0000 7fcc6d400640 1 -- 172.21.15.90:0/551034464 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcc781561c0 msgr2=0x7fcc780cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:42.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.636+0000 7fcc6d400640 1 --2- 172.21.15.90:0/551034464 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcc781561c0 0x7fcc780cdbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:42.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.636+0000 7fcc6d400640 1 -- 172.21.15.90:0/551034464 >> v1:172.21.15.103:6789/0 conn(0x7fcc780cfb30 legacy=0x7fcc780b2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:42.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.636+0000 7fcc6d400640 1 -- 172.21.15.90:0/551034464 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc78071270 con 0x7fcc780709d0 2024-04-25T01:18:42.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.637+0000 7fcc6d400640 1 --2- 172.21.15.90:0/551034464 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc780709d0 0x7fcc7815cb80 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fcc60009870 tx=0x7fcc6002a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62dd104b333ec455 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:42.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.637+0000 7fcc75400640 1 -- 172.21.15.90:0/551034464 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc60037070 con 0x7fcc780709d0 2024-04-25T01:18:42.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.637+0000 7fcc75400640 1 -- 172.21.15.90:0/551034464 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc6002ae10 con 0x7fcc780709d0 2024-04-25T01:18:42.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.637+0000 7fcc75400640 1 -- 172.21.15.90:0/551034464 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc60032660 con 0x7fcc780709d0 2024-04-25T01:18:42.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.638+0000 7fcc77200640 1 -- 172.21.15.90:0/551034464 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc780709d0 msgr2=0x7fcc7815cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:42.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.638+0000 7fcc77200640 1 --2- 172.21.15.90:0/551034464 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc780709d0 0x7fcc7815cb80 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fcc60009870 tx=0x7fcc6002a100 comp rx=0 tx=0).stop 2024-04-25T01:18:42.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.638+0000 7fcc77200640 1 -- 172.21.15.90:0/551034464 shutdown_connections 2024-04-25T01:18:42.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.638+0000 7fcc77200640 1 --2- 172.21.15.90:0/551034464 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc780709d0 0x7fcc7815cb80 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:42.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.638+0000 7fcc77200640 1 --2- 172.21.15.90:0/551034464 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcc781561c0 0x7fcc780cdbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:42.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.638+0000 7fcc77200640 1 -- 172.21.15.90:0/551034464 >> 172.21.15.90:0/551034464 conn(0x7fcc7806b320 msgr2=0x7fcc7806b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:42.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.638+0000 7fcc77200640 1 -- 172.21.15.90:0/551034464 shutdown_connections 2024-04-25T01:18:42.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.638+0000 7fcc77200640 1 -- 172.21.15.90:0/551034464 wait complete. 2024-04-25T01:18:42.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.640+0000 7fcc77200640 1 Processor -- start 2024-04-25T01:18:42.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.640+0000 7fcc77200640 1 -- start start 2024-04-25T01:18:42.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.641+0000 7fcc77200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc780709d0 0x7fcc781534c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:42.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.641+0000 7fcc77200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcc780cfb30 0x7fcc78160040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:42.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.641+0000 7fcc77200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc781561c0 0x7fcc780b2b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:42.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.641+0000 7fcc77200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcc780cffa0 con 0x7fcc780709d0 2024-04-25T01:18:42.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.641+0000 7fcc77200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcc78071830 con 0x7fcc781561c0 2024-04-25T01:18:42.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.641+0000 7fcc77200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcc780d0120 con 0x7fcc780cfb30 2024-04-25T01:18:42.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.641+0000 7fcc75e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc780709d0 0x7fcc781534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:42.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.642+0000 7fcc75e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc780709d0 0x7fcc781534c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51682/0 (socket says 172.21.15.90:51682) 2024-04-25T01:18:42.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.642+0000 7fcc75e00640 1 -- 172.21.15.90:0/1907863671 learned_addr learned my addr 172.21.15.90:0/1907863671 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:42.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.642+0000 7fcc75e00640 1 -- 172.21.15.90:0/1907863671 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcc780cfb30 msgr2=0x7fcc78160040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:42.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.642+0000 7fcc75e00640 1 --2- 172.21.15.90:0/1907863671 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcc780cfb30 0x7fcc78160040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:42.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.642+0000 7fcc76800640 1 --2- 172.21.15.90:0/1907863671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc781561c0 0x7fcc780b2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:42.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.642+0000 7fcc75e00640 1 -- 172.21.15.90:0/1907863671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc781561c0 msgr2=0x7fcc780b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:42.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.642+0000 7fcc75e00640 1 --2- 172.21.15.90:0/1907863671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc781561c0 0x7fcc780b2b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:42.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.642+0000 7fcc75e00640 1 -- 172.21.15.90:0/1907863671 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc780adbf0 con 0x7fcc780709d0 2024-04-25T01:18:42.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.642+0000 7fcc75e00640 1 --2- 172.21.15.90:0/1907863671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc780709d0 0x7fcc781534c0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fcc6800d620 tx=0x7fcc6800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:42.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.642+0000 7fcc6ea00640 1 -- 172.21.15.90:0/1907863671 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc6800b620 con 0x7fcc780709d0 2024-04-25T01:18:42.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.642+0000 7fcc6ea00640 1 -- 172.21.15.90:0/1907863671 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc68019070 con 0x7fcc780709d0 2024-04-25T01:18:42.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.643+0000 7fcc77200640 1 -- 172.21.15.90:0/1907863671 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc78153a00 con 0x7fcc780709d0 2024-04-25T01:18:42.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.643+0000 7fcc6ea00640 1 -- 172.21.15.90:0/1907863671 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc6800ba00 con 0x7fcc780709d0 2024-04-25T01:18:42.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.643+0000 7fcc77200640 1 -- 172.21.15.90:0/1907863671 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc78158d40 con 0x7fcc780709d0 2024-04-25T01:18:42.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.644+0000 7fcc77200640 1 -- 172.21.15.90:0/1907863671 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc7804f500 con 0x7fcc780709d0 2024-04-25T01:18:42.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.645+0000 7fcc6ea00640 1 -- 172.21.15.90:0/1907863671 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fcc6800bba0 con 0x7fcc780709d0 2024-04-25T01:18:42.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.645+0000 7fcc6ea00640 1 --2- 172.21.15.90:0/1907863671 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fcc5406be60 0x7fcc5406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:42.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.646+0000 7fcc6ea00640 1 -- 172.21.15.90:0/1907863671 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7fcc6808b580 con 0x7fcc780709d0 2024-04-25T01:18:42.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.646+0000 7fcc6d400640 1 --2- 172.21.15.90:0/1907863671 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fcc5406be60 0x7fcc5406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:42.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.647+0000 7fcc6d400640 1 --2- 172.21.15.90:0/1907863671 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fcc5406be60 0x7fcc5406e320 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fcc60034000 tx=0x7fcc6002c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:42.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.650+0000 7fcc6ea00640 1 -- 172.21.15.90:0/1907863671 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fcc78153a00 con 0x7fcc780709d0 2024-04-25T01:18:42.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.890+0000 7fcc77200640 1 -- 172.21.15.90:0/1907863671 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc7806b910 con 0x7fcc780709d0 2024-04-25T01:18:42.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.892+0000 7fcc6ea00640 1 -- 172.21.15.90:0/1907863671 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6492 (secure 0 0 0) 0x7fcc780adbf0 con 0x7fcc780709d0 2024-04-25T01:18:42.875 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:42.875 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1873497946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6833","nonce":1873497946},{"type":"v1","addr":"172.21.15.103:6835","nonce":1873497946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/3206865830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":3206865830},{"type":"v1","addr":"172.21.15.90:6835","nonce":3206865830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5598,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2208594231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2208594231},{"type":"v1","addr":"172.21.15.90:6837","nonce":2208594231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:18:02.701652+0000","modified":"2024-04-25T01:18:15.603132+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2},{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:18:19.371011+0000","modified":"2024-04-25T01:18:30.533899+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5601":{"gid":5601,"name":"d","rank":0,"incarnation":25,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6834/2850373997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":2850373997},{"type":"v1","addr":"172.21.15.103:6834","nonce":2850373997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":7,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5601,"qdb_cluster":[5601]},"id":3}]} 2024-04-25T01:18:42.875 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 28 2024-04-25T01:18:42.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.894+0000 7fcc77200640 1 -- 172.21.15.90:0/1907863671 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fcc5406be60 msgr2=0x7fcc5406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:42.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.894+0000 7fcc77200640 1 --2- 172.21.15.90:0/1907863671 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fcc5406be60 0x7fcc5406e320 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fcc60034000 tx=0x7fcc6002c000 comp rx=0 tx=0).stop 2024-04-25T01:18:42.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.894+0000 7fcc77200640 1 -- 172.21.15.90:0/1907863671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc780709d0 msgr2=0x7fcc781534c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:42.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.894+0000 7fcc77200640 1 --2- 172.21.15.90:0/1907863671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc780709d0 0x7fcc781534c0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fcc6800d620 tx=0x7fcc6800dae0 comp rx=0 tx=0).stop 2024-04-25T01:18:42.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.895+0000 7fcc77200640 1 -- 172.21.15.90:0/1907863671 shutdown_connections 2024-04-25T01:18:42.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.895+0000 7fcc77200640 1 --2- 172.21.15.90:0/1907863671 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fcc5406be60 0x7fcc5406e320 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:42.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.895+0000 7fcc77200640 1 --2- 172.21.15.90:0/1907863671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc781561c0 0x7fcc780b2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:42.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.895+0000 7fcc77200640 1 --2- 172.21.15.90:0/1907863671 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcc780cfb30 0x7fcc78160040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:42.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.895+0000 7fcc77200640 1 --2- 172.21.15.90:0/1907863671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc780709d0 0x7fcc781534c0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:42.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.895+0000 7fcc77200640 1 -- 172.21.15.90:0/1907863671 >> 172.21.15.90:0/1907863671 conn(0x7fcc7806b320 msgr2=0x7fcc78153fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:42.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.895+0000 7fcc77200640 1 -- 172.21.15.90:0/1907863671 shutdown_connections 2024-04-25T01:18:42.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:42.895+0000 7fcc77200640 1 -- 172.21.15.90:0/1907863671 wait complete. 2024-04-25T01:18:42.896 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:18:43.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.112+0000 7fb420e00640 1 Processor -- start 2024-04-25T01:18:43.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.112+0000 7fb420e00640 1 -- start start 2024-04-25T01:18:43.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.114+0000 7fb420e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb41c072920 0x7fb41c0cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:43.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.114+0000 7fb420e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb41c0d0070 0x7fb41c0b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:43.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.114+0000 7fb420e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb41c15e1b0 con 0x7fb41c15ddd0 2024-04-25T01:18:43.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.114+0000 7fb420e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fb41c0d04d0 con 0x7fb41c0d0070 2024-04-25T01:18:43.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.114+0000 7fb420e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb41c0713b0 con 0x7fb41c072920 2024-04-25T01:18:43.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.114+0000 7fb41be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb41c0d0070 0x7fb41c0b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:43.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.114+0000 7fb41be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb41c0d0070 0x7fb41c0b2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51690/0 (socket says 172.21.15.90:51690) 2024-04-25T01:18:43.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.114+0000 7fb41be00640 1 -- 172.21.15.90:0/4238447627 learned_addr learned my addr 172.21.15.90:0/4238447627 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:43.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.114+0000 7fb41be00640 1 -- 172.21.15.90:0/4238447627 >> v2:172.21.15.103:3300/0 conn(0x7fb41c072920 msgr2=0x7fb41c0cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:43.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.114+0000 7fb41be00640 1 --2- 172.21.15.90:0/4238447627 >> v2:172.21.15.103:3300/0 conn(0x7fb41c072920 0x7fb41c0cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:43.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.114+0000 7fb41be00640 1 -- 172.21.15.90:0/4238447627 >> v1:172.21.15.103:6789/0 conn(0x7fb41c15ddd0 legacy=0x7fb41c070b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:43.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.114+0000 7fb41be00640 1 -- 172.21.15.90:0/4238447627 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb41c071690 con 0x7fb41c0d0070 2024-04-25T01:18:43.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.115+0000 7fb41be00640 1 --2- 172.21.15.90:0/4238447627 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb41c0d0070 0x7fb41c0b2c00 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fb41c074fb0 tx=0x7fb41402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66d7bdc8a07c2731 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:43.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.116+0000 7fb41aa00640 1 -- 172.21.15.90:0/4238447627 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb414037070 con 0x7fb41c0d0070 2024-04-25T01:18:43.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.116+0000 7fb41aa00640 1 -- 172.21.15.90:0/4238447627 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb414032440 con 0x7fb41c0d0070 2024-04-25T01:18:43.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.116+0000 7fb41aa00640 1 -- 172.21.15.90:0/4238447627 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb414032810 con 0x7fb41c0d0070 2024-04-25T01:18:43.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.116+0000 7fb420e00640 1 -- 172.21.15.90:0/4238447627 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb41c0d0070 msgr2=0x7fb41c0b2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:43.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.116+0000 7fb420e00640 1 --2- 172.21.15.90:0/4238447627 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb41c0d0070 0x7fb41c0b2c00 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fb41c074fb0 tx=0x7fb41402a230 comp rx=0 tx=0).stop 2024-04-25T01:18:43.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.117+0000 7fb420e00640 1 -- 172.21.15.90:0/4238447627 shutdown_connections 2024-04-25T01:18:43.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.117+0000 7fb420e00640 1 --2- 172.21.15.90:0/4238447627 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb41c0d0070 0x7fb41c0b2c00 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:43.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.117+0000 7fb420e00640 1 --2- 172.21.15.90:0/4238447627 >> v2:172.21.15.103:3300/0 conn(0x7fb41c072920 0x7fb41c0cfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:43.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.117+0000 7fb420e00640 1 -- 172.21.15.90:0/4238447627 >> 172.21.15.90:0/4238447627 conn(0x7fb41c06b320 msgr2=0x7fb41c06b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:43.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.117+0000 7fb420e00640 1 -- 172.21.15.90:0/4238447627 shutdown_connections 2024-04-25T01:18:43.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.117+0000 7fb420e00640 1 -- 172.21.15.90:0/4238447627 wait complete. 2024-04-25T01:18:43.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.118+0000 7fb420e00640 1 Processor -- start 2024-04-25T01:18:43.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.118+0000 7fb420e00640 1 -- start start 2024-04-25T01:18:43.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.119+0000 7fb420e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb41c072920 0x7fb41c145370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:43.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.119+0000 7fb420e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb41c0d0070 0x7fb41c15a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:43.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.119+0000 7fb420e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb41c15ddd0 0x7fb41c15a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:43.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.119+0000 7fb420e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb41c0b3140 con 0x7fb41c15ddd0 2024-04-25T01:18:43.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.119+0000 7fb420e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb41c071c50 con 0x7fb41c0d0070 2024-04-25T01:18:43.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.119+0000 7fb420e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb41c0b32c0 con 0x7fb41c072920 2024-04-25T01:18:43.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.120+0000 7fb41be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb41c15ddd0 0x7fb41c15a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:43.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.120+0000 7fb41be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb41c15ddd0 0x7fb41c15a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51698/0 (socket says 172.21.15.90:51698) 2024-04-25T01:18:43.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.120+0000 7fb41be00640 1 -- 172.21.15.90:0/1061824301 learned_addr learned my addr 172.21.15.90:0/1061824301 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:43.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.120+0000 7fb41b400640 1 --2- 172.21.15.90:0/1061824301 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb41c072920 0x7fb41c145370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:43.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.120+0000 7fb41be00640 1 -- 172.21.15.90:0/1061824301 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb41c072920 msgr2=0x7fb41c145370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:43.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.120+0000 7fb41be00640 1 --2- 172.21.15.90:0/1061824301 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb41c072920 0x7fb41c145370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:43.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.120+0000 7fb41be00640 1 -- 172.21.15.90:0/1061824301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb41c0d0070 msgr2=0x7fb41c15a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:43.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.120+0000 7fb41be00640 1 --2- 172.21.15.90:0/1061824301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb41c0d0070 0x7fb41c15a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:43.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.120+0000 7fb41be00640 1 -- 172.21.15.90:0/1061824301 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb41c1458b0 con 0x7fb41c15ddd0 2024-04-25T01:18:43.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.120+0000 7fb41be00640 1 --2- 172.21.15.90:0/1061824301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb41c15ddd0 0x7fb41c15a770 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fb41402a270 tx=0x7fb4140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:43.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.121+0000 7fb413e00640 1 -- 172.21.15.90:0/1061824301 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb414041070 con 0x7fb41c15ddd0 2024-04-25T01:18:43.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.121+0000 7fb413e00640 1 -- 172.21.15.90:0/1061824301 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb41403c950 con 0x7fb41c15ddd0 2024-04-25T01:18:43.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.121+0000 7fb413e00640 1 -- 172.21.15.90:0/1061824301 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb414037040 con 0x7fb41c15ddd0 2024-04-25T01:18:43.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.121+0000 7fb420e00640 1 -- 172.21.15.90:0/1061824301 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb41c0cf000 con 0x7fb41c15ddd0 2024-04-25T01:18:43.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.121+0000 7fb420e00640 1 -- 172.21.15.90:0/1061824301 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb41c1f7080 con 0x7fb41c15ddd0 2024-04-25T01:18:43.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.122+0000 7fb420e00640 1 -- 172.21.15.90:0/1061824301 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb41c04f500 con 0x7fb41c15ddd0 2024-04-25T01:18:43.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.123+0000 7fb413e00640 1 -- 172.21.15.90:0/1061824301 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb414044070 con 0x7fb41c15ddd0 2024-04-25T01:18:43.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.124+0000 7fb413e00640 1 --2- 172.21.15.90:0/1061824301 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb3f806be30 0x7fb3f806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:43.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.124+0000 7fb413e00640 1 -- 172.21.15.90:0/1061824301 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7fb4140abc50 con 0x7fb41c15ddd0 2024-04-25T01:18:43.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.124+0000 7fb41b400640 1 --2- 172.21.15.90:0/1061824301 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb3f806be30 0x7fb3f806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:43.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.125+0000 7fb41b400640 1 --2- 172.21.15.90:0/1061824301 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb3f806be30 0x7fb3f806e2f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb41c1581c0 tx=0x7fb40c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:43.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.128+0000 7fb413e00640 1 -- 172.21.15.90:0/1061824301 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb4140b1020 con 0x7fb41c15ddd0 2024-04-25T01:18:43.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.363+0000 7fb420e00640 1 -- 172.21.15.90:0/1061824301 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb41c0b2d80 con 0x7fb41c15ddd0 2024-04-25T01:18:43.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.364+0000 7fb413e00640 1 -- 172.21.15.90:0/1061824301 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6492 (secure 0 0 0) 0x7fb41c1458b0 con 0x7fb41c15ddd0 2024-04-25T01:18:43.347 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:43.348 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1873497946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6833","nonce":1873497946},{"type":"v1","addr":"172.21.15.103:6835","nonce":1873497946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/3206865830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":3206865830},{"type":"v1","addr":"172.21.15.90:6835","nonce":3206865830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5598,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2208594231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2208594231},{"type":"v1","addr":"172.21.15.90:6837","nonce":2208594231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:18:02.701652+0000","modified":"2024-04-25T01:18:15.603132+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2},{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:18:19.371011+0000","modified":"2024-04-25T01:18:30.533899+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5601":{"gid":5601,"name":"d","rank":0,"incarnation":25,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6834/2850373997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":2850373997},{"type":"v1","addr":"172.21.15.103:6834","nonce":2850373997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":7,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5601,"qdb_cluster":[5601]},"id":3}]} 2024-04-25T01:18:43.348 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 28 2024-04-25T01:18:43.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.367+0000 7fb420e00640 1 -- 172.21.15.90:0/1061824301 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb3f806be30 msgr2=0x7fb3f806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:43.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.367+0000 7fb420e00640 1 --2- 172.21.15.90:0/1061824301 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb3f806be30 0x7fb3f806e2f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb41c1581c0 tx=0x7fb40c008040 comp rx=0 tx=0).stop 2024-04-25T01:18:43.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.367+0000 7fb420e00640 1 -- 172.21.15.90:0/1061824301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb41c15ddd0 msgr2=0x7fb41c15a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:43.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.367+0000 7fb420e00640 1 --2- 172.21.15.90:0/1061824301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb41c15ddd0 0x7fb41c15a770 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fb41402a270 tx=0x7fb4140046c0 comp rx=0 tx=0).stop 2024-04-25T01:18:43.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.367+0000 7fb420e00640 1 -- 172.21.15.90:0/1061824301 shutdown_connections 2024-04-25T01:18:43.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.367+0000 7fb420e00640 1 --2- 172.21.15.90:0/1061824301 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb3f806be30 0x7fb3f806e2f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:43.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.367+0000 7fb420e00640 1 --2- 172.21.15.90:0/1061824301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb41c15ddd0 0x7fb41c15a770 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:43.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.367+0000 7fb420e00640 1 --2- 172.21.15.90:0/1061824301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb41c0d0070 0x7fb41c15a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:43.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.367+0000 7fb420e00640 1 --2- 172.21.15.90:0/1061824301 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb41c072920 0x7fb41c145370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:43.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.367+0000 7fb420e00640 1 -- 172.21.15.90:0/1061824301 >> 172.21.15.90:0/1061824301 conn(0x7fb41c06b320 msgr2=0x7fb41c0ae260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:43.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.367+0000 7fb420e00640 1 -- 172.21.15.90:0/1061824301 shutdown_connections 2024-04-25T01:18:43.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.367+0000 7fb420e00640 1 -- 172.21.15.90:0/1061824301 wait complete. 2024-04-25T01:18:43.369 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:18:43.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.580+0000 7fbb17200640 1 Processor -- start 2024-04-25T01:18:43.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.580+0000 7fbb17200640 1 -- start start 2024-04-25T01:18:43.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.582+0000 7fbb17200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbb180709d0 0x7fbb180cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:43.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.582+0000 7fbb17200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb18070f90 con 0x7fbb180d0240 2024-04-25T01:18:43.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.582+0000 7fbb17200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb18071270 con 0x7fbb181561c0 2024-04-25T01:18:43.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.582+0000 7fbb17200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fbb180b4310 con 0x7fbb180709d0 2024-04-25T01:18:43.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.582+0000 7fbb15400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fbb180d0240 0x7fbb180b2c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41224/0 (socket says 172.21.15.90:41224) 2024-04-25T01:18:43.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.582+0000 7fbb15400640 1 -- 172.21.15.90:0/3791853634 learned_addr learned my addr 172.21.15.90:0/3791853634 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:43.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.582+0000 7fbb16800640 1 --2- 172.21.15.90:0/3791853634 >> v2:172.21.15.90:3300/0 conn(0x7fbb180709d0 0x7fbb180cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:43.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.582+0000 7fbb16800640 1 -- 172.21.15.90:0/3791853634 >> v1:172.21.15.103:6789/0 conn(0x7fbb181561c0 legacy=0x7fbb180cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:43.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.583+0000 7fbb16800640 1 -- 172.21.15.90:0/3791853634 >> v1:172.21.15.90:6789/0 conn(0x7fbb180d0240 legacy=0x7fbb180b2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:43.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.583+0000 7fbb16800640 1 -- 172.21.15.90:0/3791853634 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb18071550 con 0x7fbb180709d0 2024-04-25T01:18:43.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.583+0000 7fbb14a00640 1 -- 172.21.15.90:0/3791853634 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4220129901 0 0) 0x7fbb18070f90 con 0x7fbb180d0240 2024-04-25T01:18:43.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.584+0000 7fbb16800640 1 --2- 172.21.15.90:0/3791853634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb180709d0 0x7fbb180cdbf0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fbb10009d90 tx=0x7fbb1002a580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93311dc62e714fc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:43.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.584+0000 7fbb14a00640 1 -- 172.21.15.90:0/3791853634 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb10007740 con 0x7fbb180709d0 2024-04-25T01:18:43.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.584+0000 7fbb14a00640 1 -- 172.21.15.90:0/3791853634 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb100078e0 con 0x7fbb180709d0 2024-04-25T01:18:43.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.584+0000 7fbb14a00640 1 -- 172.21.15.90:0/3791853634 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb10007cb0 con 0x7fbb180709d0 2024-04-25T01:18:43.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.584+0000 7fbb17200640 1 -- 172.21.15.90:0/3791853634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb180709d0 msgr2=0x7fbb180cdbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:43.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.584+0000 7fbb17200640 1 --2- 172.21.15.90:0/3791853634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb180709d0 0x7fbb180cdbf0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fbb10009d90 tx=0x7fbb1002a580 comp rx=0 tx=0).stop 2024-04-25T01:18:43.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.584+0000 7fbb17200640 1 -- 172.21.15.90:0/3791853634 shutdown_connections 2024-04-25T01:18:43.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.584+0000 7fbb17200640 1 --2- 172.21.15.90:0/3791853634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb180709d0 0x7fbb180cdbf0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:43.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.584+0000 7fbb17200640 1 -- 172.21.15.90:0/3791853634 >> 172.21.15.90:0/3791853634 conn(0x7fbb1806b320 msgr2=0x7fbb1806b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:43.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.585+0000 7fbb17200640 1 -- 172.21.15.90:0/3791853634 shutdown_connections 2024-04-25T01:18:43.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.585+0000 7fbb17200640 1 -- 172.21.15.90:0/3791853634 wait complete. 2024-04-25T01:18:43.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.586+0000 7fbb17200640 1 Processor -- start 2024-04-25T01:18:43.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.586+0000 7fbb17200640 1 -- start start 2024-04-25T01:18:43.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.587+0000 7fbb17200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb180709d0 0x7fbb18153370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:43.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.587+0000 7fbb17200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb180d0240 0x7fbb1813ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:43.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.587+0000 7fbb17200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb181561c0 0x7fbb180ad8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:43.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.587+0000 7fbb17200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1815cb80 con 0x7fbb180709d0 2024-04-25T01:18:43.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.587+0000 7fbb17200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1815cd00 con 0x7fbb180d0240 2024-04-25T01:18:43.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.587+0000 7fbb17200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbb1815ce80 con 0x7fbb181561c0 2024-04-25T01:18:43.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.587+0000 7fbb15e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb180709d0 0x7fbb18153370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:43.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.588+0000 7fbb15e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb180709d0 0x7fbb18153370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51714/0 (socket says 172.21.15.90:51714) 2024-04-25T01:18:43.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.588+0000 7fbb15e00640 1 -- 172.21.15.90:0/1696802052 learned_addr learned my addr 172.21.15.90:0/1696802052 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:43.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.588+0000 7fbb15e00640 1 -- 172.21.15.90:0/1696802052 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb181561c0 msgr2=0x7fbb180ad8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:43.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.588+0000 7fbb15e00640 1 --2- 172.21.15.90:0/1696802052 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb181561c0 0x7fbb180ad8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:43.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.588+0000 7fbb15e00640 1 -- 172.21.15.90:0/1696802052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb180d0240 msgr2=0x7fbb1813ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:43.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.588+0000 7fbb15e00640 1 --2- 172.21.15.90:0/1696802052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb180d0240 0x7fbb1813ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:43.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.588+0000 7fbb15e00640 1 -- 172.21.15.90:0/1696802052 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb181419c0 con 0x7fbb180709d0 2024-04-25T01:18:43.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.588+0000 7fbb15e00640 1 --2- 172.21.15.90:0/1696802052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb180709d0 0x7fbb18153370 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fbb0800b3f0 tx=0x7fbb0800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:43.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.588+0000 7fbb0e000640 1 -- 172.21.15.90:0/1696802052 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb08015020 con 0x7fbb180709d0 2024-04-25T01:18:43.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.588+0000 7fbb0e000640 1 -- 172.21.15.90:0/1696802052 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb08011830 con 0x7fbb180709d0 2024-04-25T01:18:43.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.588+0000 7fbb0e000640 1 -- 172.21.15.90:0/1696802052 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb08011c20 con 0x7fbb180709d0 2024-04-25T01:18:43.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.588+0000 7fbb17200640 1 -- 172.21.15.90:0/1696802052 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb18141ca0 con 0x7fbb180709d0 2024-04-25T01:18:43.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.589+0000 7fbb17200640 1 -- 172.21.15.90:0/1696802052 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb18142260 con 0x7fbb180709d0 2024-04-25T01:18:43.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.590+0000 7fbb17200640 1 -- 172.21.15.90:0/1696802052 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb18142640 con 0x7fbb180709d0 2024-04-25T01:18:43.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.591+0000 7fbb0e000640 1 -- 172.21.15.90:0/1696802052 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbb08018020 con 0x7fbb180709d0 2024-04-25T01:18:43.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.592+0000 7fbb0e000640 1 --2- 172.21.15.90:0/1696802052 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbaf806be30 0x7fbaf806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:43.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.592+0000 7fbb0e000640 1 -- 172.21.15.90:0/1696802052 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7fbb0808ae30 con 0x7fbb180709d0 2024-04-25T01:18:43.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.592+0000 7fbb15400640 1 --2- 172.21.15.90:0/1696802052 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbaf806be30 0x7fbaf806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:43.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.593+0000 7fbb15400640 1 --2- 172.21.15.90:0/1696802052 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbaf806be30 0x7fbaf806e2f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fbb04007fd0 tx=0x7fbb04007920 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:43.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.597+0000 7fbb0e000640 1 -- 172.21.15.90:0/1696802052 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fbb18142640 con 0x7fbb180709d0 2024-04-25T01:18:43.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.796+0000 7fbb17200640 1 -- 172.21.15.90:0/1696802052 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbb1804f500 con 0x7fbb180709d0 2024-04-25T01:18:43.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.799+0000 7fbb0e000640 1 -- 172.21.15.90:0/1696802052 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) ==== 74+0+18947 (secure 0 0 0) 0x7fbb0801f030 con 0x7fbb180709d0 2024-04-25T01:18:43.783 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:43.783 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":50,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:18:20.389626+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:18:00.504992+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T01:18:01.714313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"recovery_fs_metadata","create_time":"2024-04-25T01:18:18.384671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.16","mappings":[{"from":4,"to":2}]},{"pgid":"6.20","mappings":[{"from":4,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:6837/431390375":"2024-04-26T01:18:15.602554+0000","172.21.15.90:6836/431390375":"2024-04-26T01:18:15.602554+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:18:43.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.803+0000 7fbb17200640 1 -- 172.21.15.90:0/1696802052 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbaf806be30 msgr2=0x7fbaf806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:43.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.803+0000 7fbb17200640 1 --2- 172.21.15.90:0/1696802052 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbaf806be30 0x7fbaf806e2f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fbb04007fd0 tx=0x7fbb04007920 comp rx=0 tx=0).stop 2024-04-25T01:18:43.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.803+0000 7fbb17200640 1 -- 172.21.15.90:0/1696802052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb180709d0 msgr2=0x7fbb18153370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:43.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.803+0000 7fbb17200640 1 --2- 172.21.15.90:0/1696802052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb180709d0 0x7fbb18153370 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fbb0800b3f0 tx=0x7fbb0800b8b0 comp rx=0 tx=0).stop 2024-04-25T01:18:43.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.803+0000 7fbb17200640 1 -- 172.21.15.90:0/1696802052 shutdown_connections 2024-04-25T01:18:43.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.803+0000 7fbb17200640 1 --2- 172.21.15.90:0/1696802052 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbaf806be30 0x7fbaf806e2f0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:43.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.803+0000 7fbb17200640 1 --2- 172.21.15.90:0/1696802052 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbb181561c0 0x7fbb180ad8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:43.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.803+0000 7fbb17200640 1 --2- 172.21.15.90:0/1696802052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb180d0240 0x7fbb1813ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:43.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.803+0000 7fbb17200640 1 --2- 172.21.15.90:0/1696802052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb180709d0 0x7fbb18153370 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:43.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.803+0000 7fbb17200640 1 -- 172.21.15.90:0/1696802052 >> 172.21.15.90:0/1696802052 conn(0x7fbb1806b320 msgr2=0x7fbb180cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:43.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.803+0000 7fbb17200640 1 -- 172.21.15.90:0/1696802052 shutdown_connections 2024-04-25T01:18:43.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:43.803+0000 7fbb17200640 1 -- 172.21.15.90:0/1696802052 wait complete. 2024-04-25T01:18:43.805 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T01:18:44.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.025+0000 7f287e200640 1 Processor -- start 2024-04-25T01:18:44.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.025+0000 7f287e200640 1 -- start start 2024-04-25T01:18:44.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.027+0000 7f287e200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f28780adfd0 0x7f28780b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:44.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.027+0000 7f287e200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2878070a80 con 0x7f2878072380 2024-04-25T01:18:44.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.027+0000 7f287e200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2878070d20 con 0x7f287815e010 2024-04-25T01:18:44.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.027+0000 7f287e200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f28780aeb60 con 0x7f28780adfd0 2024-04-25T01:18:44.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.027+0000 7f287d800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f2878072380 0x7f287806c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41238/0 (socket says 172.21.15.90:41238) 2024-04-25T01:18:44.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.027+0000 7f287d800640 1 -- 172.21.15.90:0/2759259887 learned_addr learned my addr 172.21.15.90:0/2759259887 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:44.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.027+0000 7f2877e00640 1 --2- 172.21.15.90:0/2759259887 >> v2:172.21.15.103:3300/0 conn(0x7f28780adfd0 0x7f28780b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:44.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.028+0000 7f2877400640 1 -- 172.21.15.90:0/2759259887 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3883153931 0 0) 0x7f2878070a80 con 0x7f2878072380 2024-04-25T01:18:44.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.028+0000 7f2877400640 1 -- 172.21.15.90:0/2759259887 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2860003520 con 0x7f2878072380 2024-04-25T01:18:44.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.028+0000 7f2877400640 1 -- 172.21.15.90:0/2759259887 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1897506798 0 0) 0x7f2860003520 con 0x7f2878072380 2024-04-25T01:18:44.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.028+0000 7f2877400640 1 -- 172.21.15.90:0/2759259887 >> v2:172.21.15.103:3300/0 conn(0x7f28780adfd0 msgr2=0x7f28780b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:44.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.028+0000 7f2877400640 1 --2- 172.21.15.90:0/2759259887 >> v2:172.21.15.103:3300/0 conn(0x7f28780adfd0 0x7f28780b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:44.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.028+0000 7f2877400640 1 -- 172.21.15.90:0/2759259887 >> v1:172.21.15.103:6789/0 conn(0x7f287815e010 legacy=0x7f28780ad8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:44.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.028+0000 7f2877400640 1 -- 172.21.15.90:0/2759259887 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2878071000 con 0x7f2878072380 2024-04-25T01:18:44.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.028+0000 7f2877400640 1 -- 172.21.15.90:0/2759259887 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f28700019f0 con 0x7f2878072380 2024-04-25T01:18:44.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.028+0000 7f2877400640 1 -- 172.21.15.90:0/2759259887 >> v1:172.21.15.90:6789/0 conn(0x7f2878072380 legacy=0x7f287806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:44.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.028+0000 7f2877400640 1 --2- 172.21.15.90:0/2759259887 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28600039b0 0x7f2860003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:44.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.028+0000 7f2877400640 1 --2- 172.21.15.90:0/2759259887 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2860004800 0x7f2860004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:44.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.028+0000 7f2877400640 1 --2- 172.21.15.90:0/2759259887 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28600051c0 0x7f286000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:44.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.028+0000 7f2877400640 1 -- 172.21.15.90:0/2759259887 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f28780aeb60 con 0x7f28600051c0 2024-04-25T01:18:44.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.028+0000 7f2877400640 1 -- 172.21.15.90:0/2759259887 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f286000c070 con 0x7f2860004800 2024-04-25T01:18:44.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.028+0000 7f2877400640 1 -- 172.21.15.90:0/2759259887 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f286000c270 con 0x7f28600039b0 2024-04-25T01:18:44.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.028+0000 7f287d800640 1 --2- 172.21.15.90:0/2759259887 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28600051c0 0x7f286000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:44.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.029+0000 7f287ce00640 1 --2- 172.21.15.90:0/2759259887 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28600039b0 0x7f2860003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:44.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.029+0000 7f2877e00640 1 --2- 172.21.15.90:0/2759259887 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2860004800 0x7f2860004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:44.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.029+0000 7f287d800640 1 -- 172.21.15.90:0/2759259887 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28600039b0 msgr2=0x7f2860003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:44.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.029+0000 7f287d800640 1 --2- 172.21.15.90:0/2759259887 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28600039b0 0x7f2860003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:44.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.029+0000 7f287d800640 1 -- 172.21.15.90:0/2759259887 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2860004800 msgr2=0x7f2860004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:44.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.029+0000 7f287d800640 1 --2- 172.21.15.90:0/2759259887 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2860004800 0x7f2860004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:44.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.029+0000 7f287d800640 1 -- 172.21.15.90:0/2759259887 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2860003520 con 0x7f28600051c0 2024-04-25T01:18:44.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.029+0000 7f287ce00640 1 --2- 172.21.15.90:0/2759259887 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28600039b0 0x7f2860003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:44.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.029+0000 7f2877e00640 1 --2- 172.21.15.90:0/2759259887 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2860004800 0x7f2860004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:44.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.030+0000 7f287d800640 1 -- 172.21.15.90:0/2759259887 reap_dead start 2024-04-25T01:18:44.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.030+0000 7f287d800640 1 --2- 172.21.15.90:0/2759259887 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28600051c0 0x7f286000b9c0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f287000a860 tx=0x7f2860005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25703018bb0b1a48 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:44.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.030+0000 7f2877400640 1 -- 172.21.15.90:0/2759259887 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f287000a480 con 0x7f28600051c0 2024-04-25T01:18:44.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.030+0000 7f2877400640 1 -- 172.21.15.90:0/2759259887 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f287002f070 con 0x7f28600051c0 2024-04-25T01:18:44.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.030+0000 7f2877400640 1 -- 172.21.15.90:0/2759259887 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f287002bca0 con 0x7f28600051c0 2024-04-25T01:18:44.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.031+0000 7f287e200640 1 -- 172.21.15.90:0/2759259887 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28600051c0 msgr2=0x7f286000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:44.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.031+0000 7f287e200640 1 --2- 172.21.15.90:0/2759259887 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28600051c0 0x7f286000b9c0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f287000a860 tx=0x7f2860005830 comp rx=0 tx=0).stop 2024-04-25T01:18:44.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.031+0000 7f287e200640 1 -- 172.21.15.90:0/2759259887 shutdown_connections 2024-04-25T01:18:44.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.031+0000 7f287e200640 1 --2- 172.21.15.90:0/2759259887 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28600051c0 0x7f286000b9c0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:44.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.031+0000 7f287e200640 1 -- 172.21.15.90:0/2759259887 >> 172.21.15.90:0/2759259887 conn(0x7f28780b1560 msgr2=0x7f28780b1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:44.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.031+0000 7f287e200640 1 -- 172.21.15.90:0/2759259887 shutdown_connections 2024-04-25T01:18:44.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.031+0000 7f287e200640 1 -- 172.21.15.90:0/2759259887 wait complete. 2024-04-25T01:18:44.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.033+0000 7f287e200640 1 Processor -- start 2024-04-25T01:18:44.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.033+0000 7f287e200640 1 -- start start 2024-04-25T01:18:44.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.034+0000 7f287e200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28600051c0 0x7f2878155d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:44.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.034+0000 7f287e200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28781f4450 0x7f28781554f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:44.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.034+0000 7f287e200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28780cca50 0x7f28780cced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:44.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.034+0000 7f287e200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f28780715c0 con 0x7f28781f4450 2024-04-25T01:18:44.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.034+0000 7f287e200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2878071930 con 0x7f28780cca50 2024-04-25T01:18:44.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.034+0000 7f287e200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2878071dc0 con 0x7f28600051c0 2024-04-25T01:18:44.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.034+0000 7f2877e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28781f4450 0x7f28781554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:44.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.034+0000 7f2877e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28781f4450 0x7f28781554f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51726/0 (socket says 172.21.15.90:51726) 2024-04-25T01:18:44.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.034+0000 7f2877e00640 1 -- 172.21.15.90:0/1818447972 learned_addr learned my addr 172.21.15.90:0/1818447972 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:44.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.035+0000 7f287d800640 1 --2- 172.21.15.90:0/1818447972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28780cca50 0x7f28780cced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:44.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.035+0000 7f287ce00640 1 --2- 172.21.15.90:0/1818447972 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28600051c0 0x7f2878155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:44.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.035+0000 7f2877e00640 1 -- 172.21.15.90:0/1818447972 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28600051c0 msgr2=0x7f2878155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:44.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.035+0000 7f2877e00640 1 --2- 172.21.15.90:0/1818447972 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28600051c0 0x7f2878155d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:44.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.035+0000 7f2877e00640 1 -- 172.21.15.90:0/1818447972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28780cca50 msgr2=0x7f28780cced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:44.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.035+0000 7f2877e00640 1 --2- 172.21.15.90:0/1818447972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28780cca50 0x7f28780cced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:44.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.035+0000 7f2877e00640 1 -- 172.21.15.90:0/1818447972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28781563a0 con 0x7f28781f4450 2024-04-25T01:18:44.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.035+0000 7f2877e00640 1 --2- 172.21.15.90:0/1818447972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28781f4450 0x7f28781554f0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f28640108e0 tx=0x7f2864010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:44.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.035+0000 7f2874c00640 1 -- 172.21.15.90:0/1818447972 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f286400f040 con 0x7f28781f4450 2024-04-25T01:18:44.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.035+0000 7f2874c00640 1 -- 172.21.15.90:0/1818447972 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f286400ec20 con 0x7f28781f4450 2024-04-25T01:18:44.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.035+0000 7f287e200640 1 -- 172.21.15.90:0/1818447972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2878156680 con 0x7f28781f4450 2024-04-25T01:18:44.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.035+0000 7f2874c00640 1 -- 172.21.15.90:0/1818447972 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f286401c400 con 0x7f28781f4450 2024-04-25T01:18:44.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.035+0000 7f287e200640 1 -- 172.21.15.90:0/1818447972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2878156c40 con 0x7f28781f4450 2024-04-25T01:18:44.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.037+0000 7f287e200640 1 -- 172.21.15.90:0/1818447972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28780aece0 con 0x7f28781f4450 2024-04-25T01:18:44.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.038+0000 7f2874c00640 1 -- 172.21.15.90:0/1818447972 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f286401c5a0 con 0x7f28781f4450 2024-04-25T01:18:44.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.039+0000 7f2874c00640 1 --2- 172.21.15.90:0/1818447972 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f285806be30 0x7f285806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:44.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.039+0000 7f2874c00640 1 -- 172.21.15.90:0/1818447972 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f286408baa0 con 0x7f28781f4450 2024-04-25T01:18:44.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.040+0000 7f287ce00640 1 --2- 172.21.15.90:0/1818447972 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f285806be30 0x7f285806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:44.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.040+0000 7f287ce00640 1 --2- 172.21.15.90:0/1818447972 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f285806be30 0x7f285806e2f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f286c000f60 tx=0x7f286c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:44.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.044+0000 7f2874c00640 1 -- 172.21.15.90:0/1818447972 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f28780aece0 con 0x7f28781f4450 2024-04-25T01:18:44.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.282+0000 7f287e200640 1 -- 172.21.15.90:0/1818447972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f287804f500 con 0x7f28781f4450 2024-04-25T01:18:44.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.989+0000 7f2874c00640 1 -- 172.21.15.90:0/1818447972 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v29) ==== 161+0+0 (secure 0 0 0) 0x7f2864018030 con 0x7f28781f4450 2024-04-25T01:18:44.973 INFO:teuthology.orchestra.run.smithi090.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T01:18:44.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.994+0000 7f287e200640 1 -- 172.21.15.90:0/1818447972 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f285806be30 msgr2=0x7f285806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:44.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.994+0000 7f287e200640 1 --2- 172.21.15.90:0/1818447972 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f285806be30 0x7f285806e2f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f286c000f60 tx=0x7f286c009040 comp rx=0 tx=0).stop 2024-04-25T01:18:44.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.994+0000 7f287e200640 1 -- 172.21.15.90:0/1818447972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28781f4450 msgr2=0x7f28781554f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:44.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.994+0000 7f287e200640 1 --2- 172.21.15.90:0/1818447972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28781f4450 0x7f28781554f0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f28640108e0 tx=0x7f2864010da0 comp rx=0 tx=0).stop 2024-04-25T01:18:44.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.995+0000 7f287e200640 1 -- 172.21.15.90:0/1818447972 shutdown_connections 2024-04-25T01:18:44.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.995+0000 7f287e200640 1 --2- 172.21.15.90:0/1818447972 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f285806be30 0x7f285806e2f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:44.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.995+0000 7f287e200640 1 --2- 172.21.15.90:0/1818447972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28780cca50 0x7f28780cced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:44.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.995+0000 7f287e200640 1 --2- 172.21.15.90:0/1818447972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28781f4450 0x7f28781554f0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:44.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.995+0000 7f287e200640 1 --2- 172.21.15.90:0/1818447972 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f28600051c0 0x7f2878155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:44.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.995+0000 7f287e200640 1 -- 172.21.15.90:0/1818447972 >> 172.21.15.90:0/1818447972 conn(0x7f28780b1560 msgr2=0x7f287806d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:44.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.995+0000 7f287e200640 1 -- 172.21.15.90:0/1818447972 shutdown_connections 2024-04-25T01:18:44.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:44.995+0000 7f287e200640 1 -- 172.21.15.90:0/1818447972 wait complete. 2024-04-25T01:18:45.000 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T01:18:45.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.219+0000 7f9e13200640 1 Processor -- start 2024-04-25T01:18:45.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.220+0000 7f9e13200640 1 -- start start 2024-04-25T01:18:45.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.221+0000 7f9e13200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e1415ba40 0x7f9e140cdc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:45.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.221+0000 7f9e13200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9e1415be20 0x7f9e140cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:45.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.221+0000 7f9e13200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e1415c340 con 0x7f9e1406e590 2024-04-25T01:18:45.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.221+0000 7f9e13200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f9e140ce1c0 con 0x7f9e1415be20 2024-04-25T01:18:45.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.221+0000 7f9e13200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9e1415cf10 con 0x7f9e1415ba40 2024-04-25T01:18:45.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.222+0000 7f9e0be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9e1415be20 0x7f9e140cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:45.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.222+0000 7f9e0be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9e1415be20 0x7f9e140cfae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47254/0 (socket says 172.21.15.90:47254) 2024-04-25T01:18:45.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.222+0000 7f9e0be00640 1 -- 172.21.15.90:0/734639547 learned_addr learned my addr 172.21.15.90:0/734639547 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:45.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.222+0000 7f9e11e00640 1 --2- 172.21.15.90:0/734639547 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e1415ba40 0x7f9e140cdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:45.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.222+0000 7f9e0be00640 1 -- 172.21.15.90:0/734639547 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e1415ba40 msgr2=0x7f9e140cdc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:45.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.222+0000 7f9e0be00640 1 --2- 172.21.15.90:0/734639547 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e1415ba40 0x7f9e140cdc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:45.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.222+0000 7f9e0be00640 1 -- 172.21.15.90:0/734639547 >> v1:172.21.15.103:6789/0 conn(0x7f9e1406e590 legacy=0x7f9e1415f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:45.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.222+0000 7f9e0be00640 1 -- 172.21.15.90:0/734639547 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e1415c5e0 con 0x7f9e1415be20 2024-04-25T01:18:45.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.223+0000 7f9e0be00640 1 --2- 172.21.15.90:0/734639547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1415be20 0x7f9e140cfae0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc009870 tx=0x7f9dfc02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aee783bf075992cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:45.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.223+0000 7f9e11400640 1 -- 172.21.15.90:0/734639547 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc037070 con 0x7f9e1415be20 2024-04-25T01:18:45.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.223+0000 7f9e11400640 1 -- 172.21.15.90:0/734639547 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dfc032440 con 0x7f9e1415be20 2024-04-25T01:18:45.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.223+0000 7f9e11400640 1 -- 172.21.15.90:0/734639547 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc032810 con 0x7f9e1415be20 2024-04-25T01:18:45.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.224+0000 7f9e13200640 1 -- 172.21.15.90:0/734639547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1415be20 msgr2=0x7f9e140cfae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:45.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.224+0000 7f9e13200640 1 --2- 172.21.15.90:0/734639547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1415be20 0x7f9e140cfae0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc009870 tx=0x7f9dfc02a210 comp rx=0 tx=0).stop 2024-04-25T01:18:45.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.224+0000 7f9e13200640 1 -- 172.21.15.90:0/734639547 shutdown_connections 2024-04-25T01:18:45.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.224+0000 7f9e13200640 1 --2- 172.21.15.90:0/734639547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1415be20 0x7f9e140cfae0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:45.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.224+0000 7f9e13200640 1 --2- 172.21.15.90:0/734639547 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e1415ba40 0x7f9e140cdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:45.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.224+0000 7f9e13200640 1 -- 172.21.15.90:0/734639547 >> 172.21.15.90:0/734639547 conn(0x7f9e1406b350 msgr2=0x7f9e1406b760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:45.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.224+0000 7f9e13200640 1 -- 172.21.15.90:0/734639547 shutdown_connections 2024-04-25T01:18:45.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.224+0000 7f9e13200640 1 -- 172.21.15.90:0/734639547 wait complete. 2024-04-25T01:18:45.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.226+0000 7f9e13200640 1 Processor -- start 2024-04-25T01:18:45.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.226+0000 7f9e13200640 1 -- start start 2024-04-25T01:18:45.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.227+0000 7f9e13200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e1406e590 0x7f9e1414d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:45.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.227+0000 7f9e13200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e1415ba40 0x7f9e141e5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:45.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.227+0000 7f9e13200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1415be20 0x7f9e1414e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:45.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.227+0000 7f9e13200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1406e120 con 0x7f9e1415be20 2024-04-25T01:18:45.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.227+0000 7f9e13200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1415cba0 con 0x7f9e1415ba40 2024-04-25T01:18:45.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.227+0000 7f9e13200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f9e140b4750 con 0x7f9e1406e590 2024-04-25T01:18:45.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.227+0000 7f9e12800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1415be20 0x7f9e1414e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:45.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.228+0000 7f9e12800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1415be20 0x7f9e1414e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47262/0 (socket says 172.21.15.90:47262) 2024-04-25T01:18:45.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.228+0000 7f9e12800640 1 -- 172.21.15.90:0/2759691001 learned_addr learned my addr 172.21.15.90:0/2759691001 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:45.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.228+0000 7f9e0be00640 1 --2- 172.21.15.90:0/2759691001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e1415ba40 0x7f9e141e5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:45.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.228+0000 7f9e11e00640 1 --2- 172.21.15.90:0/2759691001 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e1406e590 0x7f9e1414d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:45.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.228+0000 7f9e12800640 1 -- 172.21.15.90:0/2759691001 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e1406e590 msgr2=0x7f9e1414d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:45.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.228+0000 7f9e12800640 1 --2- 172.21.15.90:0/2759691001 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e1406e590 0x7f9e1414d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:45.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.228+0000 7f9e12800640 1 -- 172.21.15.90:0/2759691001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e1415ba40 msgr2=0x7f9e141e5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:45.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.228+0000 7f9e12800640 1 --2- 172.21.15.90:0/2759691001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e1415ba40 0x7f9e141e5fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:45.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.228+0000 7f9e12800640 1 -- 172.21.15.90:0/2759691001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e1414d8a0 con 0x7f9e1415be20 2024-04-25T01:18:45.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.228+0000 7f9e12800640 1 --2- 172.21.15.90:0/2759691001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1415be20 0x7f9e1414e940 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c00c780 tx=0x7f9e0c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:45.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.229+0000 7f9e0a000640 1 -- 172.21.15.90:0/2759691001 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c004b60 con 0x7f9e1415be20 2024-04-25T01:18:45.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.229+0000 7f9e0a000640 1 -- 172.21.15.90:0/2759691001 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e0c004d00 con 0x7f9e1415be20 2024-04-25T01:18:45.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.229+0000 7f9e0a000640 1 -- 172.21.15.90:0/2759691001 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c00aa90 con 0x7f9e1415be20 2024-04-25T01:18:45.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.229+0000 7f9e13200640 1 -- 172.21.15.90:0/2759691001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e1414db80 con 0x7f9e1415be20 2024-04-25T01:18:45.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.229+0000 7f9e13200640 1 -- 172.21.15.90:0/2759691001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e1414e140 con 0x7f9e1415be20 2024-04-25T01:18:45.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.230+0000 7f9e13200640 1 -- 172.21.15.90:0/2759691001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e1404f500 con 0x7f9e1415be20 2024-04-25T01:18:45.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.231+0000 7f9e0a000640 1 -- 172.21.15.90:0/2759691001 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9e0c00ac30 con 0x7f9e1415be20 2024-04-25T01:18:45.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.231+0000 7f9e0a000640 1 --2- 172.21.15.90:0/2759691001 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9df006be30 0x7f9df006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:45.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.232+0000 7f9e0a000640 1 -- 172.21.15.90:0/2759691001 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f9e0c08b540 con 0x7f9e1415be20 2024-04-25T01:18:45.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.232+0000 7f9e11e00640 1 --2- 172.21.15.90:0/2759691001 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9df006be30 0x7f9df006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:45.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.232+0000 7f9e11e00640 1 --2- 172.21.15.90:0/2759691001 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9df006be30 0x7f9df006e2f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9e04006fd0 tx=0x7f9e04008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:45.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.238+0000 7f9e0a000640 1 -- 172.21.15.90:0/2759691001 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9e0c020070 con 0x7f9e1415be20 2024-04-25T01:18:45.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.475+0000 7f9e13200640 1 -- 172.21.15.90:0/2759691001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f9e1406bbb0 con 0x7f9e1415be20 2024-04-25T01:18:45.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.993+0000 7f9e0a000640 1 -- 172.21.15.90:0/2759691001 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v30) ==== 104+0+0 (secure 0 0 0) 0x7f9e0c059b10 con 0x7f9e1415be20 2024-04-25T01:18:45.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.997+0000 7f9e13200640 1 -- 172.21.15.90:0/2759691001 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9df006be30 msgr2=0x7f9df006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:45.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.997+0000 7f9e13200640 1 --2- 172.21.15.90:0/2759691001 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9df006be30 0x7f9df006e2f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9e04006fd0 tx=0x7f9e04008040 comp rx=0 tx=0).stop 2024-04-25T01:18:45.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.997+0000 7f9e13200640 1 -- 172.21.15.90:0/2759691001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1415be20 msgr2=0x7f9e1414e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:45.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.997+0000 7f9e13200640 1 --2- 172.21.15.90:0/2759691001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1415be20 0x7f9e1414e940 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c00c780 tx=0x7f9e0c00cc40 comp rx=0 tx=0).stop 2024-04-25T01:18:45.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.997+0000 7f9e13200640 1 -- 172.21.15.90:0/2759691001 shutdown_connections 2024-04-25T01:18:45.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.997+0000 7f9e13200640 1 --2- 172.21.15.90:0/2759691001 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f9df006be30 0x7f9df006e2f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:45.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.997+0000 7f9e13200640 1 --2- 172.21.15.90:0/2759691001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e1415be20 0x7f9e1414e940 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:45.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.997+0000 7f9e13200640 1 --2- 172.21.15.90:0/2759691001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e1415ba40 0x7f9e141e5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:45.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.997+0000 7f9e13200640 1 --2- 172.21.15.90:0/2759691001 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f9e1406e590 0x7f9e1414d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:45.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.997+0000 7f9e13200640 1 -- 172.21.15.90:0/2759691001 >> 172.21.15.90:0/2759691001 conn(0x7f9e1406b350 msgr2=0x7f9e14156130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:45.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.998+0000 7f9e13200640 1 -- 172.21.15.90:0/2759691001 shutdown_connections 2024-04-25T01:18:45.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:45.998+0000 7f9e13200640 1 -- 172.21.15.90:0/2759691001 wait complete. 2024-04-25T01:18:46.002 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T01:18:46.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.222+0000 7f3ba5200640 1 Processor -- start 2024-04-25T01:18:46.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.222+0000 7f3ba5200640 1 -- start start 2024-04-25T01:18:46.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.224+0000 7f3ba5200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3ba00ceaf0 0x7f3ba006cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:46.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.224+0000 7f3ba5200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ba006d0c0 0x7f3ba015f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:46.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.224+0000 7f3ba5200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ba00cf040 con 0x7f3ba0145510 2024-04-25T01:18:46.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.224+0000 7f3ba5200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f3ba00cf320 con 0x7f3ba00ceaf0 2024-04-25T01:18:46.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.224+0000 7f3ba5200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3ba0070c70 con 0x7f3ba006d0c0 2024-04-25T01:18:46.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.224+0000 7f3b9d400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3ba00ceaf0 0x7f3ba006cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:46.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.224+0000 7f3b9d400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3ba00ceaf0 0x7f3ba006cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47264/0 (socket says 172.21.15.90:47264) 2024-04-25T01:18:46.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.224+0000 7f3b9d400640 1 -- 172.21.15.90:0/847159402 learned_addr learned my addr 172.21.15.90:0/847159402 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:46.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.225+0000 7f3b9d400640 1 -- 172.21.15.90:0/847159402 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ba006d0c0 msgr2=0x7f3ba015f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:46.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.225+0000 7f3b9e800640 1 --2- 172.21.15.90:0/847159402 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ba006d0c0 0x7f3ba015f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:46.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.225+0000 7f3b9d400640 1 --2- 172.21.15.90:0/847159402 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ba006d0c0 0x7f3ba015f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:46.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.225+0000 7f3b9d400640 1 -- 172.21.15.90:0/847159402 >> v1:172.21.15.103:6789/0 conn(0x7f3ba0145510 legacy=0x7f3ba00728d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:46.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.225+0000 7f3b9d400640 1 -- 172.21.15.90:0/847159402 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ba0070f50 con 0x7f3ba00ceaf0 2024-04-25T01:18:46.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.225+0000 7f3b9e800640 1 --2- 172.21.15.90:0/847159402 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ba006d0c0 0x7f3ba015f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:46.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.226+0000 7f3b9d400640 1 --2- 172.21.15.90:0/847159402 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ba00ceaf0 0x7f3ba006cb80 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f3b88009870 tx=0x7f3b8802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d8e38f3b274aa3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:46.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.226+0000 7f3b9ca00640 1 -- 172.21.15.90:0/847159402 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b88037070 con 0x7f3ba00ceaf0 2024-04-25T01:18:46.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.226+0000 7f3b9ca00640 1 -- 172.21.15.90:0/847159402 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b88032440 con 0x7f3ba00ceaf0 2024-04-25T01:18:46.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.226+0000 7f3b9ca00640 1 -- 172.21.15.90:0/847159402 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b88032810 con 0x7f3ba00ceaf0 2024-04-25T01:18:46.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.227+0000 7f3ba5200640 1 -- 172.21.15.90:0/847159402 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ba00ceaf0 msgr2=0x7f3ba006cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:46.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.227+0000 7f3ba5200640 1 --2- 172.21.15.90:0/847159402 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ba00ceaf0 0x7f3ba006cb80 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f3b88009870 tx=0x7f3b8802a180 comp rx=0 tx=0).stop 2024-04-25T01:18:46.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.227+0000 7f3ba5200640 1 -- 172.21.15.90:0/847159402 shutdown_connections 2024-04-25T01:18:46.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.227+0000 7f3ba5200640 1 --2- 172.21.15.90:0/847159402 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ba006d0c0 0x7f3ba015f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:46.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.227+0000 7f3ba5200640 1 --2- 172.21.15.90:0/847159402 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ba00ceaf0 0x7f3ba006cb80 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:46.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.227+0000 7f3ba5200640 1 -- 172.21.15.90:0/847159402 >> 172.21.15.90:0/847159402 conn(0x7f3ba00b16b0 msgr2=0x7f3ba00b1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:46.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.227+0000 7f3ba5200640 1 -- 172.21.15.90:0/847159402 shutdown_connections 2024-04-25T01:18:46.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.227+0000 7f3ba5200640 1 -- 172.21.15.90:0/847159402 wait complete. 2024-04-25T01:18:46.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.229+0000 7f3ba5200640 1 Processor -- start 2024-04-25T01:18:46.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.229+0000 7f3ba5200640 1 -- start start 2024-04-25T01:18:46.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.230+0000 7f3ba5200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ba006d0c0 0x7f3ba0141a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:46.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.230+0000 7f3ba5200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ba00ceaf0 0x7f3ba00cfcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:46.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.230+0000 7f3ba5200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ba0145510 0x7f3ba0158df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:46.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.230+0000 7f3ba5200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba0145a70 con 0x7f3ba0145510 2024-04-25T01:18:46.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.230+0000 7f3ba5200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba01458f0 con 0x7f3ba00ceaf0 2024-04-25T01:18:46.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.230+0000 7f3ba5200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f3ba015ff30 con 0x7f3ba006d0c0 2024-04-25T01:18:46.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.230+0000 7f3b9e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ba0145510 0x7f3ba0158df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:46.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.230+0000 7f3b9e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ba0145510 0x7f3ba0158df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47266/0 (socket says 172.21.15.90:47266) 2024-04-25T01:18:46.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.230+0000 7f3b9e800640 1 -- 172.21.15.90:0/3456427381 learned_addr learned my addr 172.21.15.90:0/3456427381 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:46.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.231+0000 7f3b9d400640 1 --2- 172.21.15.90:0/3456427381 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ba00ceaf0 0x7f3ba00cfcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:46.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.231+0000 7f3b9de00640 1 --2- 172.21.15.90:0/3456427381 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ba006d0c0 0x7f3ba0141a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:46.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.231+0000 7f3b9e800640 1 -- 172.21.15.90:0/3456427381 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ba006d0c0 msgr2=0x7f3ba0141a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:46.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.231+0000 7f3b9e800640 1 --2- 172.21.15.90:0/3456427381 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ba006d0c0 0x7f3ba0141a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:46.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.231+0000 7f3b9e800640 1 -- 172.21.15.90:0/3456427381 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ba00ceaf0 msgr2=0x7f3ba00cfcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:46.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.231+0000 7f3b9e800640 1 --2- 172.21.15.90:0/3456427381 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ba00ceaf0 0x7f3ba00cfcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:46.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.231+0000 7f3b9e800640 1 -- 172.21.15.90:0/3456427381 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ba0156ed0 con 0x7f3ba0145510 2024-04-25T01:18:46.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.231+0000 7f3b9e800640 1 --2- 172.21.15.90:0/3456427381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ba0145510 0x7f3ba0158df0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f3b9800c7f0 tx=0x7f3b9800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:46.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.231+0000 7f3b96000640 1 -- 172.21.15.90:0/3456427381 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b98015070 con 0x7f3ba0145510 2024-04-25T01:18:46.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.231+0000 7f3b96000640 1 -- 172.21.15.90:0/3456427381 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b98007820 con 0x7f3ba0145510 2024-04-25T01:18:46.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.231+0000 7f3b96000640 1 -- 172.21.15.90:0/3456427381 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b98007c50 con 0x7f3ba0145510 2024-04-25T01:18:46.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.231+0000 7f3ba5200640 1 -- 172.21.15.90:0/3456427381 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ba0157460 con 0x7f3ba0145510 2024-04-25T01:18:46.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.232+0000 7f3ba5200640 1 -- 172.21.15.90:0/3456427381 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ba0157a20 con 0x7f3ba0145510 2024-04-25T01:18:46.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.233+0000 7f3ba5200640 1 -- 172.21.15.90:0/3456427381 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ba0068190 con 0x7f3ba0145510 2024-04-25T01:18:46.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.234+0000 7f3b96000640 1 -- 172.21.15.90:0/3456427381 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3b98018070 con 0x7f3ba0145510 2024-04-25T01:18:46.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.235+0000 7f3b96000640 1 --2- 172.21.15.90:0/3456427381 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f3b7c06be30 0x7f3b7c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:46.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.235+0000 7f3b96000640 1 -- 172.21.15.90:0/3456427381 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6698+0+0 (secure 0 0 0) 0x7f3b9808b190 con 0x7f3ba0145510 2024-04-25T01:18:46.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.235+0000 7f3b9de00640 1 --2- 172.21.15.90:0/3456427381 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f3b7c06be30 0x7f3b7c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:46.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.236+0000 7f3b9de00640 1 --2- 172.21.15.90:0/3456427381 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f3b7c06be30 0x7f3b7c06e2f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3b90007bf0 tx=0x7f3b90007240 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:46.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.239+0000 7f3b96000640 1 -- 172.21.15.90:0/3456427381 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f3b98059760 con 0x7f3ba0145510 2024-04-25T01:18:46.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:46.437+0000 7f3ba5200640 1 -- 172.21.15.90:0/3456427381 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f3ba0156680 con 0x7f3ba0145510 2024-04-25T01:18:46.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.014+0000 7f3b96000640 1 -- 172.21.15.90:0/3456427381 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v51) ==== 181+0+0 (secure 0 0 0) 0x7f3b98059060 con 0x7f3ba0145510 2024-04-25T01:18:46.997 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_metadata' removed 2024-04-25T01:18:47.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.018+0000 7f3ba5200640 1 -- 172.21.15.90:0/3456427381 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f3b7c06be30 msgr2=0x7f3b7c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:47.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.018+0000 7f3ba5200640 1 --2- 172.21.15.90:0/3456427381 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f3b7c06be30 0x7f3b7c06e2f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3b90007bf0 tx=0x7f3b90007240 comp rx=0 tx=0).stop 2024-04-25T01:18:47.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.018+0000 7f3ba5200640 1 -- 172.21.15.90:0/3456427381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ba0145510 msgr2=0x7f3ba0158df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:47.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.018+0000 7f3ba5200640 1 --2- 172.21.15.90:0/3456427381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ba0145510 0x7f3ba0158df0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f3b9800c7f0 tx=0x7f3b9800ccb0 comp rx=0 tx=0).stop 2024-04-25T01:18:47.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.019+0000 7f3ba5200640 1 -- 172.21.15.90:0/3456427381 shutdown_connections 2024-04-25T01:18:47.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.019+0000 7f3ba5200640 1 --2- 172.21.15.90:0/3456427381 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f3b7c06be30 0x7f3b7c06e2f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:47.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.019+0000 7f3ba5200640 1 --2- 172.21.15.90:0/3456427381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ba0145510 0x7f3ba0158df0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:47.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.019+0000 7f3ba5200640 1 --2- 172.21.15.90:0/3456427381 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ba00ceaf0 0x7f3ba00cfcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:47.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.019+0000 7f3ba5200640 1 --2- 172.21.15.90:0/3456427381 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f3ba006d0c0 0x7f3ba0141a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:47.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.019+0000 7f3ba5200640 1 -- 172.21.15.90:0/3456427381 >> 172.21.15.90:0/3456427381 conn(0x7f3ba00b16b0 msgr2=0x7f3ba015cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:47.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.020+0000 7f3ba5200640 1 -- 172.21.15.90:0/3456427381 shutdown_connections 2024-04-25T01:18:47.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.020+0000 7f3ba5200640 1 -- 172.21.15.90:0/3456427381 wait complete. 2024-04-25T01:18:47.029 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T01:18:47.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.251+0000 7f0a82000640 1 Processor -- start 2024-04-25T01:18:47.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.251+0000 7f0a82000640 1 -- start start 2024-04-25T01:18:47.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.252+0000 7f0a82000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0a7c1616f0 0x7f0a7c161b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:47.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.252+0000 7f0a82000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f0a7c072620 0x7f0a7c1620d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:47.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.252+0000 7f0a82000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a7c0af170 con 0x7f0a7c162610 2024-04-25T01:18:47.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.252+0000 7f0a82000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f0a7c071c10 con 0x7f0a7c1616f0 2024-04-25T01:18:47.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.252+0000 7f0a82000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f0a7c071ef0 con 0x7f0a7c072620 2024-04-25T01:18:47.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.252+0000 7f0a7b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0a7c1616f0 0x7f0a7c161b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:47.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.253+0000 7f0a7b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0a7c1616f0 0x7f0a7c161b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47270/0 (socket says 172.21.15.90:47270) 2024-04-25T01:18:47.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.253+0000 7f0a7b400640 1 -- 172.21.15.90:0/590265394 learned_addr learned my addr 172.21.15.90:0/590265394 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:47.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.253+0000 7f0a7b400640 1 -- 172.21.15.90:0/590265394 >> v2:172.21.15.103:3300/0 conn(0x7f0a7c072620 msgr2=0x7f0a7c1620d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:47.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.253+0000 7f0a7b400640 1 --2- 172.21.15.90:0/590265394 >> v2:172.21.15.103:3300/0 conn(0x7f0a7c072620 0x7f0a7c1620d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:47.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.253+0000 7f0a7b400640 1 -- 172.21.15.90:0/590265394 >> v1:172.21.15.103:6789/0 conn(0x7f0a7c162610 legacy=0x7f0a7c0b29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:47.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.253+0000 7f0a7b400640 1 -- 172.21.15.90:0/590265394 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a7c0721d0 con 0x7f0a7c1616f0 2024-04-25T01:18:47.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.254+0000 7f0a7b400640 1 --2- 172.21.15.90:0/590265394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a7c1616f0 0x7f0a7c161b90 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c0af4c0 tx=0x7f0a7402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7658d11da8e7b0c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:47.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.254+0000 7f0a7a000640 1 -- 172.21.15.90:0/590265394 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a74036070 con 0x7f0a7c1616f0 2024-04-25T01:18:47.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.254+0000 7f0a7a000640 1 -- 172.21.15.90:0/590265394 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a74031600 con 0x7f0a7c1616f0 2024-04-25T01:18:47.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.254+0000 7f0a7a000640 1 -- 172.21.15.90:0/590265394 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a740319d0 con 0x7f0a7c1616f0 2024-04-25T01:18:47.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.255+0000 7f0a82000640 1 -- 172.21.15.90:0/590265394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a7c1616f0 msgr2=0x7f0a7c161b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:47.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.255+0000 7f0a82000640 1 --2- 172.21.15.90:0/590265394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a7c1616f0 0x7f0a7c161b90 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c0af4c0 tx=0x7f0a7402a6f0 comp rx=0 tx=0).stop 2024-04-25T01:18:47.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.255+0000 7f0a82000640 1 -- 172.21.15.90:0/590265394 shutdown_connections 2024-04-25T01:18:47.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.255+0000 7f0a82000640 1 --2- 172.21.15.90:0/590265394 >> v2:172.21.15.103:3300/0 conn(0x7f0a7c072620 0x7f0a7c1620d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:47.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.255+0000 7f0a82000640 1 --2- 172.21.15.90:0/590265394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a7c1616f0 0x7f0a7c161b90 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:47.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.255+0000 7f0a82000640 1 -- 172.21.15.90:0/590265394 >> 172.21.15.90:0/590265394 conn(0x7f0a7c0b16b0 msgr2=0x7f0a7c0b1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:47.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.255+0000 7f0a82000640 1 -- 172.21.15.90:0/590265394 shutdown_connections 2024-04-25T01:18:47.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.255+0000 7f0a82000640 1 -- 172.21.15.90:0/590265394 wait complete. 2024-04-25T01:18:47.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.257+0000 7f0a82000640 1 Processor -- start 2024-04-25T01:18:47.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.257+0000 7f0a82000640 1 -- start start 2024-04-25T01:18:47.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.258+0000 7f0a82000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a7c072620 0x7f0a7c0cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:47.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.258+0000 7f0a82000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a7c1616f0 0x7f0a7c146560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:47.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.258+0000 7f0a82000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a7c162610 0x7f0a7c0b2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:47.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.258+0000 7f0a82000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7c0718c0 con 0x7f0a7c162610 2024-04-25T01:18:47.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.258+0000 7f0a82000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7c0adc40 con 0x7f0a7c1616f0 2024-04-25T01:18:47.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.258+0000 7f0a82000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0a7c06cdf0 con 0x7f0a7c072620 2024-04-25T01:18:47.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.259+0000 7f0a7be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a7c162610 0x7f0a7c0b2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:47.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.259+0000 7f0a7be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a7c162610 0x7f0a7c0b2a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47272/0 (socket says 172.21.15.90:47272) 2024-04-25T01:18:47.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.259+0000 7f0a7be00640 1 -- 172.21.15.90:0/2653970095 learned_addr learned my addr 172.21.15.90:0/2653970095 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:47.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.259+0000 7f0a7b400640 1 --2- 172.21.15.90:0/2653970095 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a7c072620 0x7f0a7c0cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:47.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.259+0000 7f0a7aa00640 1 --2- 172.21.15.90:0/2653970095 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a7c1616f0 0x7f0a7c146560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:47.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.259+0000 7f0a7be00640 1 -- 172.21.15.90:0/2653970095 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a7c072620 msgr2=0x7f0a7c0cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:47.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.259+0000 7f0a7be00640 1 --2- 172.21.15.90:0/2653970095 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a7c072620 0x7f0a7c0cdeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:47.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.259+0000 7f0a7be00640 1 -- 172.21.15.90:0/2653970095 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a7c1616f0 msgr2=0x7f0a7c146560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:47.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.259+0000 7f0a7be00640 1 --2- 172.21.15.90:0/2653970095 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a7c1616f0 0x7f0a7c146560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:47.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.259+0000 7f0a7be00640 1 -- 172.21.15.90:0/2653970095 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a7c0cce10 con 0x7f0a7c162610 2024-04-25T01:18:47.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.259+0000 7f0a7aa00640 1 --2- 172.21.15.90:0/2653970095 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a7c1616f0 0x7f0a7c146560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:47.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.259+0000 7f0a7b400640 1 --2- 172.21.15.90:0/2653970095 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0a7c072620 0x7f0a7c0cdeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:47.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.259+0000 7f0a7be00640 1 --2- 172.21.15.90:0/2653970095 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a7c162610 0x7f0a7c0b2a70 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c00b490 tx=0x7f0a6c00b950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:47.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.260+0000 7f0a73400640 1 -- 172.21.15.90:0/2653970095 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c015020 con 0x7f0a7c162610 2024-04-25T01:18:47.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.260+0000 7f0a73400640 1 -- 172.21.15.90:0/2653970095 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a6c011850 con 0x7f0a7c162610 2024-04-25T01:18:47.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.260+0000 7f0a82000640 1 -- 172.21.15.90:0/2653970095 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a7c0cd0f0 con 0x7f0a7c162610 2024-04-25T01:18:47.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.260+0000 7f0a73400640 1 -- 172.21.15.90:0/2653970095 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c011c40 con 0x7f0a7c162610 2024-04-25T01:18:47.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.260+0000 7f0a82000640 1 -- 172.21.15.90:0/2653970095 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a7c0cd6b0 con 0x7f0a7c162610 2024-04-25T01:18:47.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.261+0000 7f0a82000640 1 -- 172.21.15.90:0/2653970095 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a7c068190 con 0x7f0a7c162610 2024-04-25T01:18:47.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.262+0000 7f0a73400640 1 -- 172.21.15.90:0/2653970095 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0a6c018020 con 0x7f0a7c162610 2024-04-25T01:18:47.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.263+0000 7f0a73400640 1 --2- 172.21.15.90:0/2653970095 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0a5806be30 0x7f0a5806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:47.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.264+0000 7f0a73400640 1 -- 172.21.15.90:0/2653970095 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6260+0+0 (secure 0 0 0) 0x7f0a6c08af90 con 0x7f0a7c162610 2024-04-25T01:18:47.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.264+0000 7f0a7b400640 1 --2- 172.21.15.90:0/2653970095 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0a5806be30 0x7f0a5806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:47.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.264+0000 7f0a7b400640 1 --2- 172.21.15.90:0/2653970095 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0a5806be30 0x7f0a5806e2f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0a74033000 tx=0x7f0a74009000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:47.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.268+0000 7f0a73400640 1 -- 172.21.15.90:0/2653970095 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0a6c01f020 con 0x7f0a7c162610 2024-04-25T01:18:47.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.466+0000 7f0a82000640 1 -- 172.21.15.90:0/2653970095 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f0a7c0b2810 con 0x7f0a7c162610 2024-04-25T01:18:47.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.467+0000 7f0a73400640 1 -- 172.21.15.90:0/2653970095 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=-16 pool 'cephfs_data' is in use by CephFS v51) ==== 181+0+0 (secure 0 0 0) 0x7f0a6c0172c0 con 0x7f0a7c162610 2024-04-25T01:18:47.450 INFO:teuthology.orchestra.run.smithi090.stderr:Error EBUSY: pool 'cephfs_data' is in use by CephFS 2024-04-25T01:18:47.469 DEBUG:teuthology.orchestra.run:got remote process result: 16 2024-04-25T01:18:47.470 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:18:47.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.691+0000 7fd381400640 1 Processor -- start 2024-04-25T01:18:47.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.692+0000 7fd381400640 1 -- start start 2024-04-25T01:18:47.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.693+0000 7fd381400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd37c15e5b0 0x7fd37c0b52d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:47.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.693+0000 7fd381400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd37c0b2dc0 0x7fd37c0b3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:47.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.693+0000 7fd381400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd37c0d4bc0 con 0x7fd37c0d06e0 2024-04-25T01:18:47.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.693+0000 7fd381400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fd37c0cf5b0 con 0x7fd37c0b2dc0 2024-04-25T01:18:47.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.693+0000 7fd381400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd37c0d48e0 con 0x7fd37c15e5b0 2024-04-25T01:18:47.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.694+0000 7fd380a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd37c0b2dc0 0x7fd37c0b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:47.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.694+0000 7fd373400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd37c15e5b0 0x7fd37c0b52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:47.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.694+0000 7fd380a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fd37c0b2dc0 0x7fd37c0b3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.90:45322/0 (socket says 172.21.15.90:45322) 2024-04-25T01:18:47.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.694+0000 7fd380a00640 1 -- 172.21.15.90:0/2589753076 learned_addr learned my addr 172.21.15.90:0/2589753076 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:47.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.695+0000 7fd380a00640 1 -- 172.21.15.90:0/2589753076 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd37c15e5b0 msgr2=0x7fd37c0b52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:47.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.695+0000 7fd380a00640 1 --2- 172.21.15.90:0/2589753076 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd37c15e5b0 0x7fd37c0b52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:47.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.695+0000 7fd380a00640 1 -- 172.21.15.90:0/2589753076 >> v1:172.21.15.103:6789/0 conn(0x7fd37c0d06e0 legacy=0x7fd37c15dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:47.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.695+0000 7fd380a00640 1 -- 172.21.15.90:0/2589753076 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd37c0cf850 con 0x7fd37c0b2dc0 2024-04-25T01:18:47.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.695+0000 7fd373400640 1 --2- 172.21.15.90:0/2589753076 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd37c15e5b0 0x7fd37c0b52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:18:47.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.696+0000 7fd380a00640 1 --2- 172.21.15.90:0/2589753076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd37c0b2dc0 0x7fd37c0b3450 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd374009e50 tx=0x7fd37402a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=390301be04c378b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:47.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.696+0000 7fd37b400640 1 -- 172.21.15.90:0/2589753076 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd374036070 con 0x7fd37c0b2dc0 2024-04-25T01:18:47.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.696+0000 7fd37b400640 1 -- 172.21.15.90:0/2589753076 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd374031600 con 0x7fd37c0b2dc0 2024-04-25T01:18:47.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.696+0000 7fd37b400640 1 -- 172.21.15.90:0/2589753076 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3740319d0 con 0x7fd37c0b2dc0 2024-04-25T01:18:47.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.697+0000 7fd381400640 1 -- 172.21.15.90:0/2589753076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd37c0b2dc0 msgr2=0x7fd37c0b3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:47.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.697+0000 7fd381400640 1 --2- 172.21.15.90:0/2589753076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd37c0b2dc0 0x7fd37c0b3450 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd374009e50 tx=0x7fd37402a680 comp rx=0 tx=0).stop 2024-04-25T01:18:47.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.697+0000 7fd381400640 1 -- 172.21.15.90:0/2589753076 shutdown_connections 2024-04-25T01:18:47.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.697+0000 7fd381400640 1 --2- 172.21.15.90:0/2589753076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd37c0b2dc0 0x7fd37c0b3450 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:47.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.697+0000 7fd381400640 1 --2- 172.21.15.90:0/2589753076 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd37c15e5b0 0x7fd37c0b52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:47.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.697+0000 7fd381400640 1 -- 172.21.15.90:0/2589753076 >> 172.21.15.90:0/2589753076 conn(0x7fd37c0d3d20 msgr2=0x7fd37c0d4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:47.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.698+0000 7fd381400640 1 -- 172.21.15.90:0/2589753076 shutdown_connections 2024-04-25T01:18:47.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.698+0000 7fd381400640 1 -- 172.21.15.90:0/2589753076 wait complete. 2024-04-25T01:18:47.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.700+0000 7fd381400640 1 Processor -- start 2024-04-25T01:18:47.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.700+0000 7fd381400640 1 -- start start 2024-04-25T01:18:47.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.701+0000 7fd381400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd37c0b2dc0 0x7fd37c14a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:47.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.701+0000 7fd381400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd37c0d06e0 0x7fd37c14ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:47.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.701+0000 7fd381400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd37c15e5b0 0x7fd37c15bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:47.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.701+0000 7fd381400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd37c0d2e40 con 0x7fd37c0d06e0 2024-04-25T01:18:47.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.701+0000 7fd381400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd37c0d2cc0 con 0x7fd37c0b2dc0 2024-04-25T01:18:47.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.701+0000 7fd381400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd37c0d0ac0 con 0x7fd37c15e5b0 2024-04-25T01:18:47.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.702+0000 7fd373400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd37c0d06e0 0x7fd37c14ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:47.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.702+0000 7fd373400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd37c0d06e0 0x7fd37c14ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47284/0 (socket says 172.21.15.90:47284) 2024-04-25T01:18:47.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.702+0000 7fd373400640 1 -- 172.21.15.90:0/2471360392 learned_addr learned my addr 172.21.15.90:0/2471360392 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:47.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.702+0000 7fd37be00640 1 --2- 172.21.15.90:0/2471360392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd37c0b2dc0 0x7fd37c14a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:47.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.702+0000 7fd380a00640 1 --2- 172.21.15.90:0/2471360392 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd37c15e5b0 0x7fd37c15bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:47.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.702+0000 7fd373400640 1 -- 172.21.15.90:0/2471360392 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd37c15e5b0 msgr2=0x7fd37c15bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:47.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.702+0000 7fd373400640 1 --2- 172.21.15.90:0/2471360392 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd37c15e5b0 0x7fd37c15bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:47.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.702+0000 7fd373400640 1 -- 172.21.15.90:0/2471360392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd37c0b2dc0 msgr2=0x7fd37c14a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:47.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.702+0000 7fd373400640 1 --2- 172.21.15.90:0/2471360392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd37c0b2dc0 0x7fd37c14a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:47.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.702+0000 7fd373400640 1 -- 172.21.15.90:0/2471360392 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd37c0cc730 con 0x7fd37c0d06e0 2024-04-25T01:18:47.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.703+0000 7fd373400640 1 --2- 172.21.15.90:0/2471360392 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd37c0d06e0 0x7fd37c14ab70 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fd36400e990 tx=0x7fd36400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:47.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.703+0000 7fd378c00640 1 -- 172.21.15.90:0/2471360392 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd364017070 con 0x7fd37c0d06e0 2024-04-25T01:18:47.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.703+0000 7fd378c00640 1 -- 172.21.15.90:0/2471360392 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd36400c9e0 con 0x7fd37c0d06e0 2024-04-25T01:18:47.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.703+0000 7fd378c00640 1 -- 172.21.15.90:0/2471360392 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd36400cdb0 con 0x7fd37c0d06e0 2024-04-25T01:18:47.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.703+0000 7fd381400640 1 -- 172.21.15.90:0/2471360392 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd37c0b82a0 con 0x7fd37c0d06e0 2024-04-25T01:18:47.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.703+0000 7fd381400640 1 -- 172.21.15.90:0/2471360392 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd37c0b8860 con 0x7fd37c0d06e0 2024-04-25T01:18:47.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.704+0000 7fd381400640 1 -- 172.21.15.90:0/2471360392 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd37c04f500 con 0x7fd37c0d06e0 2024-04-25T01:18:47.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.705+0000 7fd378c00640 1 -- 172.21.15.90:0/2471360392 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd36401a070 con 0x7fd37c0d06e0 2024-04-25T01:18:47.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.706+0000 7fd378c00640 1 --2- 172.21.15.90:0/2471360392 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd35806be30 0x7fd35806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:47.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.706+0000 7fd378c00640 1 -- 172.21.15.90:0/2471360392 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6260+0+0 (secure 0 0 0) 0x7fd36408b2f0 con 0x7fd37c0d06e0 2024-04-25T01:18:47.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.706+0000 7fd37be00640 1 --2- 172.21.15.90:0/2471360392 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd35806be30 0x7fd35806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:47.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.707+0000 7fd37be00640 1 --2- 172.21.15.90:0/2471360392 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd35806be30 0x7fd35806e2f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd36c007fd0 tx=0x7fd36c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:47.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.711+0000 7fd378c00640 1 -- 172.21.15.90:0/2471360392 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd364059a60 con 0x7fd37c0d06e0 2024-04-25T01:18:47.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.910+0000 7fd381400640 1 -- 172.21.15.90:0/2471360392 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd37c0b1e70 con 0x7fd37c0d06e0 2024-04-25T01:18:47.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.912+0000 7fd378c00640 1 -- 172.21.15.90:0/2471360392 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) ==== 74+0+17080 (secure 0 0 0) 0x7fd364059360 con 0x7fd37c0d06e0 2024-04-25T01:18:47.895 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:47.895 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":51,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:18:47.003583+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T01:18:01.714313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"recovery_fs_metadata","create_time":"2024-04-25T01:18:18.384671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.16","mappings":[{"from":4,"to":2}]},{"pgid":"6.20","mappings":[{"from":4,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:6837/431390375":"2024-04-26T01:18:15.602554+0000","172.21.15.90:6836/431390375":"2024-04-26T01:18:15.602554+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:18:47.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.915+0000 7fd381400640 1 -- 172.21.15.90:0/2471360392 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd35806be30 msgr2=0x7fd35806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:47.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.915+0000 7fd381400640 1 --2- 172.21.15.90:0/2471360392 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd35806be30 0x7fd35806e2f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd36c007fd0 tx=0x7fd36c009040 comp rx=0 tx=0).stop 2024-04-25T01:18:47.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.915+0000 7fd381400640 1 -- 172.21.15.90:0/2471360392 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd37c0d06e0 msgr2=0x7fd37c14ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:47.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.915+0000 7fd381400640 1 --2- 172.21.15.90:0/2471360392 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd37c0d06e0 0x7fd37c14ab70 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fd36400e990 tx=0x7fd36400ee50 comp rx=0 tx=0).stop 2024-04-25T01:18:47.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.915+0000 7fd381400640 1 -- 172.21.15.90:0/2471360392 shutdown_connections 2024-04-25T01:18:47.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.915+0000 7fd381400640 1 --2- 172.21.15.90:0/2471360392 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd35806be30 0x7fd35806e2f0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:47.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.915+0000 7fd381400640 1 --2- 172.21.15.90:0/2471360392 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd37c15e5b0 0x7fd37c15bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:47.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.915+0000 7fd381400640 1 --2- 172.21.15.90:0/2471360392 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd37c0d06e0 0x7fd37c14ab70 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:47.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.915+0000 7fd381400640 1 --2- 172.21.15.90:0/2471360392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd37c0b2dc0 0x7fd37c14a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:47.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.915+0000 7fd381400640 1 -- 172.21.15.90:0/2471360392 >> 172.21.15.90:0/2471360392 conn(0x7fd37c0d3d20 msgr2=0x7fd37c0b3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:47.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.915+0000 7fd381400640 1 -- 172.21.15.90:0/2471360392 shutdown_connections 2024-04-25T01:18:47.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:47.915+0000 7fd381400640 1 -- 172.21.15.90:0/2471360392 wait complete. 2024-04-25T01:18:47.917 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:16:15.894126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:18:01.714313+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'recovery_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'recovery_fs_metadata', 'create_time': '2024-04-25T01:18:18.384671+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'recovery_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:18:47.917 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:18:48.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.131+0000 7fa0fc600640 1 Processor -- start 2024-04-25T01:18:48.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.131+0000 7fa0fc600640 1 -- start start 2024-04-25T01:18:48.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.133+0000 7fa0fc600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa0f415e010 0x7fa0f406dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:48.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.133+0000 7fa0fc600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0f4071520 0x7fa0f40719a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:48.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.133+0000 7fa0fc600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0f415e3f0 con 0x7fa0f406e620 2024-04-25T01:18:48.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.133+0000 7fa0fc600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fa0f406cd20 con 0x7fa0f415e010 2024-04-25T01:18:48.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.133+0000 7fa0fc600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa0f415f440 con 0x7fa0f4071520 2024-04-25T01:18:48.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.134+0000 7fa0f9c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa0f415e010 0x7fa0f406dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:48.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.134+0000 7fa0f9200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0f4071520 0x7fa0f40719a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:48.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.134+0000 7fa0f9c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fa0f415e010 0x7fa0f406dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.90:45342/0 (socket says 172.21.15.90:45342) 2024-04-25T01:18:48.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.134+0000 7fa0f9c00640 1 -- 172.21.15.90:0/592890259 learned_addr learned my addr 172.21.15.90:0/592890259 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:48.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.134+0000 7fa0f9c00640 1 -- 172.21.15.90:0/592890259 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0f4071520 msgr2=0x7fa0f40719a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:48.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.134+0000 7fa0f9c00640 1 --2- 172.21.15.90:0/592890259 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0f4071520 0x7fa0f40719a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:48.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.134+0000 7fa0f9c00640 1 -- 172.21.15.90:0/592890259 >> v1:172.21.15.103:6789/0 conn(0x7fa0f406e620 legacy=0x7fa0f40b3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:48.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.135+0000 7fa0f9c00640 1 -- 172.21.15.90:0/592890259 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0f4071fe0 con 0x7fa0f415e010 2024-04-25T01:18:48.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.136+0000 7fa0f9c00640 1 --2- 172.21.15.90:0/592890259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0f415e010 0x7fa0f406dd80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8009870 tx=0x7fa0e802a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be8e40ad603a4b4e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:48.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.137+0000 7fa0f3e00640 1 -- 172.21.15.90:0/592890259 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e8037070 con 0x7fa0f415e010 2024-04-25T01:18:48.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.137+0000 7fa0f3e00640 1 -- 172.21.15.90:0/592890259 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0e8032440 con 0x7fa0f415e010 2024-04-25T01:18:48.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.137+0000 7fa0f3e00640 1 -- 172.21.15.90:0/592890259 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e8032810 con 0x7fa0f415e010 2024-04-25T01:18:48.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.138+0000 7fa0fc600640 1 -- 172.21.15.90:0/592890259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0f415e010 msgr2=0x7fa0f406dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:48.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.138+0000 7fa0fc600640 1 --2- 172.21.15.90:0/592890259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0f415e010 0x7fa0f406dd80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8009870 tx=0x7fa0e802a250 comp rx=0 tx=0).stop 2024-04-25T01:18:48.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.138+0000 7fa0fc600640 1 -- 172.21.15.90:0/592890259 shutdown_connections 2024-04-25T01:18:48.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.138+0000 7fa0fc600640 1 --2- 172.21.15.90:0/592890259 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0f4071520 0x7fa0f40719a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:48.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.138+0000 7fa0fc600640 1 --2- 172.21.15.90:0/592890259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0f415e010 0x7fa0f406dd80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:48.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.138+0000 7fa0fc600640 1 -- 172.21.15.90:0/592890259 >> 172.21.15.90:0/592890259 conn(0x7fa0f40b17b0 msgr2=0x7fa0f40b1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:48.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.138+0000 7fa0fc600640 1 -- 172.21.15.90:0/592890259 shutdown_connections 2024-04-25T01:18:48.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.138+0000 7fa0fc600640 1 -- 172.21.15.90:0/592890259 wait complete. 2024-04-25T01:18:48.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.140+0000 7fa0fc600640 1 Processor -- start 2024-04-25T01:18:48.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.140+0000 7fa0fc600640 1 -- start start 2024-04-25T01:18:48.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.141+0000 7fa0fc600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0f406e620 0x7fa0f4160370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:48.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.141+0000 7fa0fc600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0f4071520 0x7fa0f4159270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:48.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.141+0000 7fa0fc600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0f415e010 0x7fa0f41f4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:48.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.141+0000 7fa0fc600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f406d580 con 0x7fa0f406e620 2024-04-25T01:18:48.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.141+0000 7fa0fc600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f406d2a0 con 0x7fa0f4071520 2024-04-25T01:18:48.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.141+0000 7fa0fc600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa0f406d860 con 0x7fa0f415e010 2024-04-25T01:18:48.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.141+0000 7fa0f9c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0f406e620 0x7fa0f4160370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:48.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.141+0000 7fa0f9c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0f406e620 0x7fa0f4160370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47286/0 (socket says 172.21.15.90:47286) 2024-04-25T01:18:48.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.141+0000 7fa0f9c00640 1 -- 172.21.15.90:0/762619796 learned_addr learned my addr 172.21.15.90:0/762619796 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:48.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.141+0000 7fa0f9c00640 1 -- 172.21.15.90:0/762619796 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0f415e010 msgr2=0x7fa0f41f4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:48.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.141+0000 7fa0f9c00640 1 --2- 172.21.15.90:0/762619796 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0f415e010 0x7fa0f41f4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:48.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.141+0000 7fa0f9200640 1 --2- 172.21.15.90:0/762619796 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0f4071520 0x7fa0f4159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:48.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.141+0000 7fa0f9c00640 1 -- 172.21.15.90:0/762619796 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0f4071520 msgr2=0x7fa0f4159270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:48.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.141+0000 7fa0f9c00640 1 --2- 172.21.15.90:0/762619796 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0f4071520 0x7fa0f4159270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:48.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.141+0000 7fa0f9c00640 1 -- 172.21.15.90:0/762619796 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0f41e7a20 con 0x7fa0f406e620 2024-04-25T01:18:48.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.142+0000 7fa0f9c00640 1 --2- 172.21.15.90:0/762619796 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0f406e620 0x7fa0f4160370 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8000c00 tx=0x7fa0e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:48.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.142+0000 7fa0f1600640 1 -- 172.21.15.90:0/762619796 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e8037040 con 0x7fa0f406e620 2024-04-25T01:18:48.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.142+0000 7fa0f1600640 1 -- 172.21.15.90:0/762619796 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0e8042070 con 0x7fa0f406e620 2024-04-25T01:18:48.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.142+0000 7fa0f1600640 1 -- 172.21.15.90:0/762619796 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e8030880 con 0x7fa0f406e620 2024-04-25T01:18:48.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.142+0000 7fa0fc600640 1 -- 172.21.15.90:0/762619796 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0f4160aa0 con 0x7fa0f406e620 2024-04-25T01:18:48.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.142+0000 7fa0fc600640 1 -- 172.21.15.90:0/762619796 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0f4161060 con 0x7fa0f406e620 2024-04-25T01:18:48.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.143+0000 7fa0fc600640 1 -- 172.21.15.90:0/762619796 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0f41e7bf0 con 0x7fa0f406e620 2024-04-25T01:18:48.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.144+0000 7fa0f1600640 1 -- 172.21.15.90:0/762619796 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa0e8030a20 con 0x7fa0f406e620 2024-04-25T01:18:48.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.146+0000 7fa0f1600640 1 --2- 172.21.15.90:0/762619796 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa0d806be60 0x7fa0d806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:48.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.146+0000 7fa0f1600640 1 -- 172.21.15.90:0/762619796 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6260+0+0 (secure 0 0 0) 0x7fa0e80ab9d0 con 0x7fa0f406e620 2024-04-25T01:18:48.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.146+0000 7fa0f9200640 1 --2- 172.21.15.90:0/762619796 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa0d806be60 0x7fa0d806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:48.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.147+0000 7fa0f9200640 1 --2- 172.21.15.90:0/762619796 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa0d806be60 0x7fa0d806e320 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa0f40cc770 tx=0x7fa0e4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:48.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.150+0000 7fa0f1600640 1 -- 172.21.15.90:0/762619796 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa0e807a030 con 0x7fa0f406e620 2024-04-25T01:18:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.350+0000 7fa0fc600640 1 -- 172.21.15.90:0/762619796 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa0f4158c40 con 0x7fa0f406e620 2024-04-25T01:18:48.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.351+0000 7fa0f1600640 1 -- 172.21.15.90:0/762619796 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v51) ==== 93+0+10 (secure 0 0 0) 0x7fa0e8079930 con 0x7fa0f406e620 2024-04-25T01:18:48.334 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T01:18:48.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.354+0000 7fa0fc600640 1 -- 172.21.15.90:0/762619796 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa0d806be60 msgr2=0x7fa0d806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:48.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.354+0000 7fa0fc600640 1 --2- 172.21.15.90:0/762619796 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa0d806be60 0x7fa0d806e320 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa0f40cc770 tx=0x7fa0e4008040 comp rx=0 tx=0).stop 2024-04-25T01:18:48.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.354+0000 7fa0fc600640 1 -- 172.21.15.90:0/762619796 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0f406e620 msgr2=0x7fa0f4160370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:48.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.354+0000 7fa0fc600640 1 --2- 172.21.15.90:0/762619796 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0f406e620 0x7fa0f4160370 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8000c00 tx=0x7fa0e80046c0 comp rx=0 tx=0).stop 2024-04-25T01:18:48.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.355+0000 7fa0fc600640 1 -- 172.21.15.90:0/762619796 shutdown_connections 2024-04-25T01:18:48.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.355+0000 7fa0fc600640 1 --2- 172.21.15.90:0/762619796 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa0d806be60 0x7fa0d806e320 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:48.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.355+0000 7fa0fc600640 1 --2- 172.21.15.90:0/762619796 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa0f415e010 0x7fa0f41f4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:48.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.355+0000 7fa0fc600640 1 --2- 172.21.15.90:0/762619796 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0f4071520 0x7fa0f4159270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:48.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.355+0000 7fa0fc600640 1 --2- 172.21.15.90:0/762619796 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0f406e620 0x7fa0f4160370 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:48.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.355+0000 7fa0fc600640 1 -- 172.21.15.90:0/762619796 >> 172.21.15.90:0/762619796 conn(0x7fa0f40b17b0 msgr2=0x7fa0f40b33b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:48.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.355+0000 7fa0fc600640 1 -- 172.21.15.90:0/762619796 shutdown_connections 2024-04-25T01:18:48.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.355+0000 7fa0fc600640 1 -- 172.21.15.90:0/762619796 wait complete. 2024-04-25T01:18:48.356 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:18:48.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.570+0000 7f59c7200640 1 Processor -- start 2024-04-25T01:18:48.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.570+0000 7f59c7200640 1 -- start start 2024-04-25T01:18:48.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.572+0000 7f59c7200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f59c015be40 0x7f59c00b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:48.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.572+0000 7f59c7200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59c00b3250 con 0x7f59c0155460 2024-04-25T01:18:48.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.572+0000 7f59c7200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59c00b3530 con 0x7f59c006ea40 2024-04-25T01:18:48.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.572+0000 7f59c7200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f59c015c2b0 con 0x7f59c015be40 2024-04-25T01:18:48.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.572+0000 7f59bfe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f59c0155460 0x7f59c006e330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52822/0 (socket says 172.21.15.90:52822) 2024-04-25T01:18:48.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.572+0000 7f59bfe00640 1 -- 172.21.15.90:0/2269522176 learned_addr learned my addr 172.21.15.90:0/2269522176 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:48.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.572+0000 7f59c5200640 1 --2- 172.21.15.90:0/2269522176 >> v2:172.21.15.90:3300/0 conn(0x7f59c015be40 0x7f59c00b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:48.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.573+0000 7f59bf400640 1 -- 172.21.15.90:0/2269522176 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1100507468 0 0) 0x7f59c00b3250 con 0x7f59c0155460 2024-04-25T01:18:48.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.573+0000 7f59bf400640 1 -- 172.21.15.90:0/2269522176 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59a4005730 con 0x7f59c0155460 2024-04-25T01:18:48.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.573+0000 7f59c5200640 1 -- 172.21.15.90:0/2269522176 >> v1:172.21.15.103:6789/0 conn(0x7f59c006ea40 legacy=0x7f59c00713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:48.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.573+0000 7f59c5200640 1 -- 172.21.15.90:0/2269522176 >> v1:172.21.15.90:6789/0 conn(0x7f59c0155460 legacy=0x7f59c006e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:48.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.573+0000 7f59c5200640 1 -- 172.21.15.90:0/2269522176 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59c00b3810 con 0x7f59c015be40 2024-04-25T01:18:48.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.574+0000 7f59c5200640 1 --2- 172.21.15.90:0/2269522176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59c015be40 0x7f59c00b2ab0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f59b8009d90 tx=0x7f59b802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7afb03ca7f817e6d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:48.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.574+0000 7f59bf400640 1 -- 172.21.15.90:0/2269522176 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b8007740 con 0x7f59c015be40 2024-04-25T01:18:48.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.574+0000 7f59bf400640 1 -- 172.21.15.90:0/2269522176 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59b80078e0 con 0x7f59c015be40 2024-04-25T01:18:48.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.574+0000 7f59bf400640 1 -- 172.21.15.90:0/2269522176 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b8007cb0 con 0x7f59c015be40 2024-04-25T01:18:48.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.575+0000 7f59c7200640 1 -- 172.21.15.90:0/2269522176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59c015be40 msgr2=0x7f59c00b2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:48.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.575+0000 7f59c7200640 1 --2- 172.21.15.90:0/2269522176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59c015be40 0x7f59c00b2ab0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f59b8009d90 tx=0x7f59b802a6f0 comp rx=0 tx=0).stop 2024-04-25T01:18:48.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.575+0000 7f59c7200640 1 -- 172.21.15.90:0/2269522176 shutdown_connections 2024-04-25T01:18:48.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.575+0000 7f59c7200640 1 --2- 172.21.15.90:0/2269522176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59c015be40 0x7f59c00b2ab0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:48.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.575+0000 7f59c7200640 1 -- 172.21.15.90:0/2269522176 >> 172.21.15.90:0/2269522176 conn(0x7f59c00b17b0 msgr2=0x7f59c00b1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:48.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.575+0000 7f59c7200640 1 -- 172.21.15.90:0/2269522176 shutdown_connections 2024-04-25T01:18:48.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.575+0000 7f59c7200640 1 -- 172.21.15.90:0/2269522176 wait complete. 2024-04-25T01:18:48.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.577+0000 7f59c7200640 1 Processor -- start 2024-04-25T01:18:48.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.577+0000 7f59c7200640 1 -- start start 2024-04-25T01:18:48.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.578+0000 7f59c7200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f59c006ea40 0x7f59c015ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:48.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.578+0000 7f59c7200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59c0155460 0x7f59c015c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:48.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.578+0000 7f59c7200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59c015be40 0x7f59c015ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:48.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.578+0000 7f59c7200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f59c00b3af0 con 0x7f59c0155460 2024-04-25T01:18:48.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.578+0000 7f59c7200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f59c00b2ff0 con 0x7f59c015be40 2024-04-25T01:18:48.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.578+0000 7f59c7200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f59c015c480 con 0x7f59c006ea40 2024-04-25T01:18:48.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.578+0000 7f59b7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59c0155460 0x7f59c015c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:48.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.578+0000 7f59b7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59c0155460 0x7f59c015c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47292/0 (socket says 172.21.15.90:47292) 2024-04-25T01:18:48.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.578+0000 7f59b7e00640 1 -- 172.21.15.90:0/3776214584 learned_addr learned my addr 172.21.15.90:0/3776214584 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:48.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.579+0000 7f59c5200640 1 --2- 172.21.15.90:0/3776214584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59c015be40 0x7f59c015ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:48.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.579+0000 7f59bfe00640 1 --2- 172.21.15.90:0/3776214584 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f59c006ea40 0x7f59c015ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:48.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.579+0000 7f59b7e00640 1 -- 172.21.15.90:0/3776214584 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f59c006ea40 msgr2=0x7f59c015ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:48.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.579+0000 7f59b7e00640 1 --2- 172.21.15.90:0/3776214584 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f59c006ea40 0x7f59c015ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:48.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.579+0000 7f59b7e00640 1 -- 172.21.15.90:0/3776214584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59c015be40 msgr2=0x7f59c015ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:48.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.579+0000 7f59b7e00640 1 --2- 172.21.15.90:0/3776214584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59c015be40 0x7f59c015ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:48.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.579+0000 7f59b7e00640 1 -- 172.21.15.90:0/3776214584 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59c0141aa0 con 0x7f59c0155460 2024-04-25T01:18:48.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.579+0000 7f59c5200640 1 --2- 172.21.15.90:0/3776214584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59c015be40 0x7f59c015ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:48.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.579+0000 7f59b7e00640 1 --2- 172.21.15.90:0/3776214584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59c0155460 0x7f59c015c780 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f59a800c3c0 tx=0x7f59a800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:48.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.579+0000 7f59bcc00640 1 -- 172.21.15.90:0/3776214584 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a8017070 con 0x7f59c0155460 2024-04-25T01:18:48.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.579+0000 7f59bcc00640 1 -- 172.21.15.90:0/3776214584 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59a80052d0 con 0x7f59c0155460 2024-04-25T01:18:48.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.579+0000 7f59bcc00640 1 -- 172.21.15.90:0/3776214584 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a8003830 con 0x7f59c0155460 2024-04-25T01:18:48.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.579+0000 7f59c7200640 1 -- 172.21.15.90:0/3776214584 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59c015ef40 con 0x7f59c0155460 2024-04-25T01:18:48.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.580+0000 7f59c7200640 1 -- 172.21.15.90:0/3776214584 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59c015d5d0 con 0x7f59c0155460 2024-04-25T01:18:48.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.581+0000 7f59c7200640 1 -- 172.21.15.90:0/3776214584 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59c006ce50 con 0x7f59c0155460 2024-04-25T01:18:48.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.582+0000 7f59bcc00640 1 -- 172.21.15.90:0/3776214584 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f59a801a070 con 0x7f59c0155460 2024-04-25T01:18:48.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.582+0000 7f59bcc00640 1 --2- 172.21.15.90:0/3776214584 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f599c06be30 0x7f599c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:48.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.583+0000 7f59bcc00640 1 -- 172.21.15.90:0/3776214584 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6260+0+0 (secure 0 0 0) 0x7f59a808a3a0 con 0x7f59c0155460 2024-04-25T01:18:48.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.583+0000 7f59bfe00640 1 --2- 172.21.15.90:0/3776214584 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f599c06be30 0x7f599c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:48.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.583+0000 7f59bfe00640 1 --2- 172.21.15.90:0/3776214584 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f599c06be30 0x7f599c06e2f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f59b00011e0 tx=0x7f59b0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:48.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.587+0000 7f59bcc00640 1 -- 172.21.15.90:0/3776214584 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f59a8058b20 con 0x7f59c0155460 2024-04-25T01:18:48.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.786+0000 7f59c7200640 1 -- 172.21.15.90:0/3776214584 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f59c0151230 con 0x7f59c0155460 2024-04-25T01:18:48.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.786+0000 7f59bcc00640 1 -- 172.21.15.90:0/3776214584 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v51) ==== 100+0+11 (secure 0 0 0) 0x7f59a8058420 con 0x7f59c0155460 2024-04-25T01:18:48.769 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-25T01:18:48.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.789+0000 7f59c7200640 1 -- 172.21.15.90:0/3776214584 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f599c06be30 msgr2=0x7f599c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:48.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.789+0000 7f59c7200640 1 --2- 172.21.15.90:0/3776214584 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f599c06be30 0x7f599c06e2f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f59b00011e0 tx=0x7f59b0009040 comp rx=0 tx=0).stop 2024-04-25T01:18:48.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.789+0000 7f59c7200640 1 -- 172.21.15.90:0/3776214584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59c0155460 msgr2=0x7f59c015c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:48.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.789+0000 7f59c7200640 1 --2- 172.21.15.90:0/3776214584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59c0155460 0x7f59c015c780 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f59a800c3c0 tx=0x7f59a800c880 comp rx=0 tx=0).stop 2024-04-25T01:18:48.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.789+0000 7f59c7200640 1 -- 172.21.15.90:0/3776214584 shutdown_connections 2024-04-25T01:18:48.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.789+0000 7f59c7200640 1 --2- 172.21.15.90:0/3776214584 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f599c06be30 0x7f599c06e2f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:48.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.789+0000 7f59c7200640 1 --2- 172.21.15.90:0/3776214584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59c015be40 0x7f59c015ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:48.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.789+0000 7f59c7200640 1 --2- 172.21.15.90:0/3776214584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59c0155460 0x7f59c015c780 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:48.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.789+0000 7f59c7200640 1 --2- 172.21.15.90:0/3776214584 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f59c006ea40 0x7f59c015ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:48.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.789+0000 7f59c7200640 1 -- 172.21.15.90:0/3776214584 >> 172.21.15.90:0/3776214584 conn(0x7f59c00b17b0 msgr2=0x7f59c00b1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:48.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.790+0000 7f59c7200640 1 -- 172.21.15.90:0/3776214584 shutdown_connections 2024-04-25T01:18:48.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:48.790+0000 7f59c7200640 1 -- 172.21.15.90:0/3776214584 wait complete. 2024-04-25T01:18:48.791 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get recovery_fs_metadata pg_num 2024-04-25T01:18:48.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.006+0000 7ff184600640 1 Processor -- start 2024-04-25T01:18:48.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.006+0000 7ff184600640 1 -- start start 2024-04-25T01:18:48.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.008+0000 7ff184600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff17c06d0a0 0x7ff17c162910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:48.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.008+0000 7ff184600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff17c072360 con 0x7ff17c06ea00 2024-04-25T01:18:48.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.008+0000 7ff184600640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff17c0b2c50 con 0x7ff17c14be80 2024-04-25T01:18:48.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.008+0000 7ff184600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff17c072640 con 0x7ff17c06d0a0 2024-04-25T01:18:48.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.008+0000 7ff181200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff17c06ea00 0x7ff17c06c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52836/0 (socket says 172.21.15.90:52836) 2024-04-25T01:18:48.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.008+0000 7ff181200640 1 -- 172.21.15.90:0/1676108374 learned_addr learned my addr 172.21.15.90:0/1676108374 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:48.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.008+0000 7ff182600640 1 --2- 172.21.15.90:0/1676108374 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff17c06d0a0 0x7ff17c162910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:48.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.009+0000 7ff17be00640 1 -- 172.21.15.90:0/1676108374 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 466578726 0 0) 0x7ff17c072360 con 0x7ff17c06ea00 2024-04-25T01:18:48.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.009+0000 7ff17be00640 1 -- 172.21.15.90:0/1676108374 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff164003520 con 0x7ff17c06ea00 2024-04-25T01:18:48.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.009+0000 7ff17be00640 1 -- 172.21.15.90:0/1676108374 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3524558217 0 0) 0x7ff164003520 con 0x7ff17c06ea00 2024-04-25T01:18:48.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.009+0000 7ff17be00640 1 -- 172.21.15.90:0/1676108374 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff17c06d0a0 msgr2=0x7ff17c162910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:48.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.009+0000 7ff17be00640 1 --2- 172.21.15.90:0/1676108374 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff17c06d0a0 0x7ff17c162910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:48.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.009+0000 7ff17be00640 1 -- 172.21.15.90:0/1676108374 >> v1:172.21.15.103:6789/0 conn(0x7ff17c14be80 legacy=0x7ff17c071b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:48.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.009+0000 7ff17be00640 1 -- 172.21.15.90:0/1676108374 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff17c0b2f30 con 0x7ff17c06ea00 2024-04-25T01:18:48.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.009+0000 7ff17be00640 1 -- 172.21.15.90:0/1676108374 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7ff168001670 con 0x7ff17c06ea00 2024-04-25T01:18:48.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.009+0000 7ff17be00640 1 -- 172.21.15.90:0/1676108374 >> v1:172.21.15.90:6789/0 conn(0x7ff17c06ea00 legacy=0x7ff17c06c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:48.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.009+0000 7ff17be00640 1 --2- 172.21.15.90:0/1676108374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1640039e0 0x7ff164003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:48.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.009+0000 7ff17be00640 1 --2- 172.21.15.90:0/1676108374 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff17c06d0a0 0x7ff164004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:48.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.009+0000 7ff17be00640 1 --2- 172.21.15.90:0/1676108374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff164004d40 0x7ff1640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:48.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.009+0000 7ff17be00640 1 -- 172.21.15.90:0/1676108374 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff17c072640 con 0x7ff164004d40 2024-04-25T01:18:48.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.009+0000 7ff17be00640 1 -- 172.21.15.90:0/1676108374 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff16400baa0 con 0x7ff1640039e0 2024-04-25T01:18:48.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.009+0000 7ff17be00640 1 -- 172.21.15.90:0/1676108374 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff16400bca0 con 0x7ff17c06d0a0 2024-04-25T01:18:48.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.009+0000 7ff182600640 1 --2- 172.21.15.90:0/1676108374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff164004d40 0x7ff1640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:48.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.010+0000 7ff181c00640 1 --2- 172.21.15.90:0/1676108374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1640039e0 0x7ff164003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:48.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.010+0000 7ff181200640 1 --2- 172.21.15.90:0/1676108374 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff17c06d0a0 0x7ff164004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:48.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.010+0000 7ff182600640 1 -- 172.21.15.90:0/1676108374 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff17c06d0a0 msgr2=0x7ff164004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:48.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.010+0000 7ff182600640 1 --2- 172.21.15.90:0/1676108374 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff17c06d0a0 0x7ff164004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:48.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.010+0000 7ff182600640 1 -- 172.21.15.90:0/1676108374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1640039e0 msgr2=0x7ff164003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:48.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.010+0000 7ff182600640 1 --2- 172.21.15.90:0/1676108374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1640039e0 0x7ff164003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:48.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.010+0000 7ff182600640 1 -- 172.21.15.90:0/1676108374 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff164003520 con 0x7ff164004d40 2024-04-25T01:18:48.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.010+0000 7ff181c00640 1 --2- 172.21.15.90:0/1676108374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1640039e0 0x7ff164003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:18:48.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.010+0000 7ff181200640 1 --2- 172.21.15.90:0/1676108374 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff17c06d0a0 0x7ff164004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:48.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.011+0000 7ff182600640 1 --2- 172.21.15.90:0/1676108374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff164004d40 0x7ff1640051c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7ff17c070bc0 tx=0x7ff164005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae36e552733a413d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:48.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.011+0000 7ff17be00640 1 -- 172.21.15.90:0/1676108374 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff174037070 con 0x7ff164004d40 2024-04-25T01:18:48.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.011+0000 7ff17be00640 1 -- 172.21.15.90:0/1676108374 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1740325f0 con 0x7ff164004d40 2024-04-25T01:18:48.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.011+0000 7ff17be00640 1 -- 172.21.15.90:0/1676108374 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff174009c10 con 0x7ff164004d40 2024-04-25T01:18:48.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.012+0000 7ff184600640 1 -- 172.21.15.90:0/1676108374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff164004d40 msgr2=0x7ff1640051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:48.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.012+0000 7ff184600640 1 --2- 172.21.15.90:0/1676108374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff164004d40 0x7ff1640051c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7ff17c070bc0 tx=0x7ff164005830 comp rx=0 tx=0).stop 2024-04-25T01:18:48.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.012+0000 7ff182600640 1 -- 172.21.15.90:0/1676108374 reap_dead start 2024-04-25T01:18:48.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.012+0000 7ff184600640 1 -- 172.21.15.90:0/1676108374 shutdown_connections 2024-04-25T01:18:48.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.012+0000 7ff184600640 1 -- 172.21.15.90:0/1676108374 >> 172.21.15.90:0/1676108374 conn(0x7ff17c0b17b0 msgr2=0x7ff17c0b1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:48.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.012+0000 7ff184600640 1 -- 172.21.15.90:0/1676108374 shutdown_connections 2024-04-25T01:18:48.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.012+0000 7ff184600640 1 -- 172.21.15.90:0/1676108374 wait complete. 2024-04-25T01:18:48.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.014+0000 7ff184600640 1 Processor -- start 2024-04-25T01:18:48.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.014+0000 7ff184600640 1 -- start start 2024-04-25T01:18:48.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.015+0000 7ff184600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff164004d40 0x7ff17c161870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:48.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.015+0000 7ff184600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff17c0ada00 0x7ff17c0ade80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:48.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.015+0000 7ff184600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff17c0b3630 0x7ff17c0b3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:48.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.015+0000 7ff184600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff17c162ca0 con 0x7ff17c0b3630 2024-04-25T01:18:48.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.015+0000 7ff184600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff17c162b20 con 0x7ff164004d40 2024-04-25T01:18:48.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.015+0000 7ff184600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff17c14c260 con 0x7ff17c0ada00 2024-04-25T01:18:48.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.015+0000 7ff182600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff17c0b3630 0x7ff17c0b3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:48.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.015+0000 7ff182600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff17c0b3630 0x7ff17c0b3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47318/0 (socket says 172.21.15.90:47318) 2024-04-25T01:18:48.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.015+0000 7ff182600640 1 -- 172.21.15.90:0/937229137 learned_addr learned my addr 172.21.15.90:0/937229137 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:49.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.016+0000 7ff181c00640 1 --2- 172.21.15.90:0/937229137 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff164004d40 0x7ff17c161870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:49.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.016+0000 7ff182600640 1 -- 172.21.15.90:0/937229137 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff17c0ada00 msgr2=0x7ff17c0ade80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:49.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.016+0000 7ff182600640 1 --2- 172.21.15.90:0/937229137 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff17c0ada00 0x7ff17c0ade80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:49.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.016+0000 7ff182600640 1 -- 172.21.15.90:0/937229137 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff164004d40 msgr2=0x7ff17c161870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:49.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.016+0000 7ff182600640 1 --2- 172.21.15.90:0/937229137 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff164004d40 0x7ff17c161870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:49.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.016+0000 7ff182600640 1 -- 172.21.15.90:0/937229137 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff17c0ae4e0 con 0x7ff17c0b3630 2024-04-25T01:18:49.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.016+0000 7ff181c00640 1 --2- 172.21.15.90:0/937229137 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff164004d40 0x7ff17c161870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:49.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.016+0000 7ff182600640 1 --2- 172.21.15.90:0/937229137 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff17c0b3630 0x7ff17c0b3ab0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7ff17400e200 tx=0x7ff174009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:49.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.016+0000 7ff179600640 1 -- 172.21.15.90:0/937229137 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff174037040 con 0x7ff17c0b3630 2024-04-25T01:18:49.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.016+0000 7ff179600640 1 -- 172.21.15.90:0/937229137 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff174044070 con 0x7ff17c0b3630 2024-04-25T01:18:49.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.016+0000 7ff179600640 1 -- 172.21.15.90:0/937229137 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff174009c10 con 0x7ff17c0b3630 2024-04-25T01:18:49.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.016+0000 7ff184600640 1 -- 172.21.15.90:0/937229137 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff17c0ae7c0 con 0x7ff17c0b3630 2024-04-25T01:18:49.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.017+0000 7ff184600640 1 -- 172.21.15.90:0/937229137 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff17c0aed80 con 0x7ff17c0b3630 2024-04-25T01:18:49.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.018+0000 7ff184600640 1 -- 172.21.15.90:0/937229137 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff17c04f5a0 con 0x7ff17c0b3630 2024-04-25T01:18:49.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.019+0000 7ff179600640 1 -- 172.21.15.90:0/937229137 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff174005d10 con 0x7ff17c0b3630 2024-04-25T01:18:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.019+0000 7ff179600640 1 --2- 172.21.15.90:0/937229137 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff15c06be30 0x7ff15c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.019+0000 7ff179600640 1 -- 172.21.15.90:0/937229137 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6260+0+0 (secure 0 0 0) 0x7ff1740ab960 con 0x7ff17c0b3630 2024-04-25T01:18:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.020+0000 7ff181c00640 1 --2- 172.21.15.90:0/937229137 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff15c06be30 0x7ff15c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:49.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.020+0000 7ff181c00640 1 --2- 172.21.15.90:0/937229137 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff15c06be30 0x7ff15c06e2f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff1700016b0 tx=0x7ff170009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:49.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.024+0000 7ff179600640 1 -- 172.21.15.90:0/937229137 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff17404e020 con 0x7ff17c0b3630 2024-04-25T01:18:49.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.222+0000 7ff184600640 1 -- 172.21.15.90:0/937229137 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "recovery_fs_metadata", "var": "pg_num"} v 0) -- 0x7ff17c0b03b0 con 0x7ff17c0b3630 2024-04-25T01:18:49.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.223+0000 7ff179600640 1 -- 172.21.15.90:0/937229137 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "recovery_fs_metadata", "var": "pg_num"}]=0 v51) ==== 109+0+11 (secure 0 0 0) 0x7ff17402f070 con 0x7ff17c0b3630 2024-04-25T01:18:49.206 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T01:18:49.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.226+0000 7ff184600640 1 -- 172.21.15.90:0/937229137 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff15c06be30 msgr2=0x7ff15c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:49.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.226+0000 7ff184600640 1 --2- 172.21.15.90:0/937229137 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff15c06be30 0x7ff15c06e2f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff1700016b0 tx=0x7ff170009040 comp rx=0 tx=0).stop 2024-04-25T01:18:49.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.226+0000 7ff184600640 1 -- 172.21.15.90:0/937229137 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff17c0b3630 msgr2=0x7ff17c0b3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:49.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.226+0000 7ff184600640 1 --2- 172.21.15.90:0/937229137 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff17c0b3630 0x7ff17c0b3ab0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7ff17400e200 tx=0x7ff174009750 comp rx=0 tx=0).stop 2024-04-25T01:18:49.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.226+0000 7ff184600640 1 -- 172.21.15.90:0/937229137 shutdown_connections 2024-04-25T01:18:49.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.226+0000 7ff184600640 1 --2- 172.21.15.90:0/937229137 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff15c06be30 0x7ff15c06e2f0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:49.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.226+0000 7ff184600640 1 --2- 172.21.15.90:0/937229137 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff17c0b3630 0x7ff17c0b3ab0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:49.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.226+0000 7ff184600640 1 --2- 172.21.15.90:0/937229137 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff17c0ada00 0x7ff17c0ade80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:49.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.226+0000 7ff184600640 1 --2- 172.21.15.90:0/937229137 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff164004d40 0x7ff17c161870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:49.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.226+0000 7ff184600640 1 -- 172.21.15.90:0/937229137 >> 172.21.15.90:0/937229137 conn(0x7ff17c0b17b0 msgr2=0x7ff17c0cecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:49.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.226+0000 7ff184600640 1 -- 172.21.15.90:0/937229137 shutdown_connections 2024-04-25T01:18:49.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.226+0000 7ff184600640 1 -- 172.21.15.90:0/937229137 wait complete. 2024-04-25T01:18:49.228 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:18:49.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.445+0000 7f7ebb200640 1 Processor -- start 2024-04-25T01:18:49.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.445+0000 7f7ebb200640 1 -- start start 2024-04-25T01:18:49.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.447+0000 7f7ebb200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ebc15a980 0x7f7ebc06d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:49.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.447+0000 7f7ebb200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7ebc0d0240 0x7f7ebc0ccfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:49.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.447+0000 7f7ebb200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ebc0ccb10 con 0x7f7ebc071dd0 2024-04-25T01:18:49.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.447+0000 7f7ebb200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f7ebc072230 con 0x7f7ebc0d0240 2024-04-25T01:18:49.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.447+0000 7f7ebb200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7ebc0aede0 con 0x7f7ebc15a980 2024-04-25T01:18:49.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.447+0000 7f7eb9400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f7ebc071dd0 0x7f7ebc0cfb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52842/0 (socket says 172.21.15.90:52842) 2024-04-25T01:18:49.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.447+0000 7f7eb9400640 1 -- 172.21.15.90:0/3799537592 learned_addr learned my addr 172.21.15.90:0/3799537592 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:49.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.447+0000 7f7eb9e00640 1 --2- 172.21.15.90:0/3799537592 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ebc15a980 0x7f7ebc06d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:49.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.447+0000 7f7eba800640 1 --2- 172.21.15.90:0/3799537592 >> v2:172.21.15.103:3300/0 conn(0x7f7ebc0d0240 0x7f7ebc0ccfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:49.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.447+0000 7f7eb8a00640 1 -- 172.21.15.90:0/3799537592 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 837677361 0 0) 0x7f7ea8001110 con 0x7f7ebc071dd0 2024-04-25T01:18:49.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.448+0000 7f7eb8a00640 1 -- 172.21.15.90:0/3799537592 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ea0003520 con 0x7f7ebc071dd0 2024-04-25T01:18:49.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.448+0000 7f7eb8a00640 1 -- 172.21.15.90:0/3799537592 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 794153177 0 0) 0x7f7ea0003520 con 0x7f7ebc071dd0 2024-04-25T01:18:49.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.448+0000 7f7eb8a00640 1 -- 172.21.15.90:0/3799537592 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ebc15a980 msgr2=0x7f7ebc06d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:49.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.448+0000 7f7eb8a00640 1 --2- 172.21.15.90:0/3799537592 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ebc15a980 0x7f7ebc06d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:49.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.448+0000 7f7eb8a00640 1 -- 172.21.15.90:0/3799537592 >> v2:172.21.15.103:3300/0 conn(0x7f7ebc0d0240 msgr2=0x7f7ebc0ccfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:49.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.448+0000 7f7eb8a00640 1 --2- 172.21.15.90:0/3799537592 >> v2:172.21.15.103:3300/0 conn(0x7f7ebc0d0240 0x7f7ebc0ccfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:49.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.448+0000 7f7eb8a00640 1 -- 172.21.15.90:0/3799537592 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ebc06ddc0 con 0x7f7ebc071dd0 2024-04-25T01:18:49.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.448+0000 7f7eb8a00640 1 -- 172.21.15.90:0/3799537592 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f7ea8001840 con 0x7f7ebc071dd0 2024-04-25T01:18:49.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.448+0000 7f7eb8a00640 1 -- 172.21.15.90:0/3799537592 >> v1:172.21.15.90:6789/0 conn(0x7f7ebc071dd0 legacy=0x7f7ebc0cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:49.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.448+0000 7f7eb8a00640 1 --2- 172.21.15.90:0/3799537592 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ebc15a980 0x7f7ea00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:49.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.448+0000 7f7eb8a00640 1 --2- 172.21.15.90:0/3799537592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ea0003f30 0x7f7ea0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:49.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.448+0000 7f7eb8a00640 1 --2- 172.21.15.90:0/3799537592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ea0004d40 0x7f7ea00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:49.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.448+0000 7f7eb8a00640 1 -- 172.21.15.90:0/3799537592 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7ebc072230 con 0x7f7ea0003f30 2024-04-25T01:18:49.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.448+0000 7f7eb8a00640 1 -- 172.21.15.90:0/3799537592 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7ebc0aede0 con 0x7f7ea0004d40 2024-04-25T01:18:49.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.448+0000 7f7eb8a00640 1 -- 172.21.15.90:0/3799537592 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7ea000dc20 con 0x7f7ebc15a980 2024-04-25T01:18:49.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.448+0000 7f7eb9400640 1 --2- 172.21.15.90:0/3799537592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ea0003f30 0x7f7ea0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:49.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.449+0000 7f7eb9400640 1 -- 172.21.15.90:0/3799537592 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ebc15a980 msgr2=0x7f7ea00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:49.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.449+0000 7f7eb9400640 1 --2- 172.21.15.90:0/3799537592 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ebc15a980 0x7f7ea00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:49.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.449+0000 7f7eb9400640 1 -- 172.21.15.90:0/3799537592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ea0004d40 msgr2=0x7f7ea00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:49.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.449+0000 7f7eba800640 1 --2- 172.21.15.90:0/3799537592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ea0004d40 0x7f7ea00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:49.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.449+0000 7f7eb9400640 1 --2- 172.21.15.90:0/3799537592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ea0004d40 0x7f7ea00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:49.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.449+0000 7f7eb9400640 1 -- 172.21.15.90:0/3799537592 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ea0003520 con 0x7f7ea0003f30 2024-04-25T01:18:49.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.449+0000 7f7eba800640 1 --2- 172.21.15.90:0/3799537592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ea0004d40 0x7f7ea00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:49.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.450+0000 7f7eb9400640 1 --2- 172.21.15.90:0/3799537592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ea0003f30 0x7f7ea0004800 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f7ea800c4f0 tx=0x7f7ea0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b239cca3186f541f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:49.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.450+0000 7f7eb8a00640 1 -- 172.21.15.90:0/3799537592 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ea800c0e0 con 0x7f7ea0003f30 2024-04-25T01:18:49.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.450+0000 7f7eb8a00640 1 -- 172.21.15.90:0/3799537592 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ea8034070 con 0x7f7ea0003f30 2024-04-25T01:18:49.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.450+0000 7f7eb8a00640 1 -- 172.21.15.90:0/3799537592 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ea8005d70 con 0x7f7ea0003f30 2024-04-25T01:18:49.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.451+0000 7f7ebb200640 1 -- 172.21.15.90:0/3799537592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ea0003f30 msgr2=0x7f7ea0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:49.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.451+0000 7f7ebb200640 1 --2- 172.21.15.90:0/3799537592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ea0003f30 0x7f7ea0004800 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f7ea800c4f0 tx=0x7f7ea0005830 comp rx=0 tx=0).stop 2024-04-25T01:18:49.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.451+0000 7f7eba800640 1 -- 172.21.15.90:0/3799537592 reap_dead start 2024-04-25T01:18:49.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.451+0000 7f7ebb200640 1 -- 172.21.15.90:0/3799537592 shutdown_connections 2024-04-25T01:18:49.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.451+0000 7f7ebb200640 1 -- 172.21.15.90:0/3799537592 >> 172.21.15.90:0/3799537592 conn(0x7f7ebc06b320 msgr2=0x7f7ebc06b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:49.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.451+0000 7f7ebb200640 1 -- 172.21.15.90:0/3799537592 shutdown_connections 2024-04-25T01:18:49.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.451+0000 7f7ebb200640 1 -- 172.21.15.90:0/3799537592 wait complete. 2024-04-25T01:18:49.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.453+0000 7f7ebb200640 1 Processor -- start 2024-04-25T01:18:49.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.453+0000 7f7ebb200640 1 -- start start 2024-04-25T01:18:49.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.454+0000 7f7ebb200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ea0003f30 0x7f7ebc144f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:49.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.454+0000 7f7ebb200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ebc1e32c0 0x7f7ebc15b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:49.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.454+0000 7f7ebb200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ebc1e3740 0x7f7ebc147770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:49.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.454+0000 7f7ebb200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7ebc06e660 con 0x7f7ea0003f30 2024-04-25T01:18:49.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.454+0000 7f7ebb200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7ebc06e380 con 0x7f7ebc1e3740 2024-04-25T01:18:49.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.454+0000 7f7ebb200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7ebc15ad60 con 0x7f7ebc1e32c0 2024-04-25T01:18:49.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.454+0000 7f7eb9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ea0003f30 0x7f7ebc144f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:49.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.454+0000 7f7eb9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ea0003f30 0x7f7ebc144f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47344/0 (socket says 172.21.15.90:47344) 2024-04-25T01:18:49.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.454+0000 7f7eb9e00640 1 -- 172.21.15.90:0/2375253172 learned_addr learned my addr 172.21.15.90:0/2375253172 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:49.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.455+0000 7f7eb9400640 1 --2- 172.21.15.90:0/2375253172 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ebc1e32c0 0x7f7ebc15b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:49.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.455+0000 7f7eb9e00640 1 -- 172.21.15.90:0/2375253172 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ebc1e32c0 msgr2=0x7f7ebc15b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:49.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.455+0000 7f7eba800640 1 --2- 172.21.15.90:0/2375253172 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ebc1e3740 0x7f7ebc147770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:49.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.455+0000 7f7eb9e00640 1 --2- 172.21.15.90:0/2375253172 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ebc1e32c0 0x7f7ebc15b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:49.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.455+0000 7f7eb9e00640 1 -- 172.21.15.90:0/2375253172 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ebc1e3740 msgr2=0x7f7ebc147770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:49.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.455+0000 7f7eb9e00640 1 --2- 172.21.15.90:0/2375253172 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ebc1e3740 0x7f7ebc147770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:49.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.455+0000 7f7eb9e00640 1 -- 172.21.15.90:0/2375253172 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ebc145720 con 0x7f7ea0003f30 2024-04-25T01:18:49.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.455+0000 7f7eb9400640 1 --2- 172.21.15.90:0/2375253172 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ebc1e32c0 0x7f7ebc15b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:49.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.455+0000 7f7eb9e00640 1 --2- 172.21.15.90:0/2375253172 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ea0003f30 0x7f7ebc144f90 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f7eac00e7c0 tx=0x7f7eac00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:49.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.455+0000 7f7eb2000640 1 -- 172.21.15.90:0/2375253172 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eac017070 con 0x7f7ea0003f30 2024-04-25T01:18:49.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.455+0000 7f7eb2000640 1 -- 172.21.15.90:0/2375253172 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7eac00c940 con 0x7f7ea0003f30 2024-04-25T01:18:49.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.455+0000 7f7ebb200640 1 -- 172.21.15.90:0/2375253172 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ebc145a00 con 0x7f7ea0003f30 2024-04-25T01:18:49.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.455+0000 7f7eb2000640 1 -- 172.21.15.90:0/2375253172 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eac00cd10 con 0x7f7ea0003f30 2024-04-25T01:18:49.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.455+0000 7f7ebb200640 1 -- 172.21.15.90:0/2375253172 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ebc145fc0 con 0x7f7ea0003f30 2024-04-25T01:18:49.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.456+0000 7f7ebb200640 1 -- 172.21.15.90:0/2375253172 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ebc04f500 con 0x7f7ea0003f30 2024-04-25T01:18:49.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.457+0000 7f7eb2000640 1 -- 172.21.15.90:0/2375253172 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7eac01a070 con 0x7f7ea0003f30 2024-04-25T01:18:49.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.458+0000 7f7eb2000640 1 --2- 172.21.15.90:0/2375253172 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7e9c06be30 0x7f7e9c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:49.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.458+0000 7f7eb2000640 1 -- 172.21.15.90:0/2375253172 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6260+0+0 (secure 0 0 0) 0x7f7eac08b220 con 0x7f7ea0003f30 2024-04-25T01:18:49.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.459+0000 7f7eb9400640 1 --2- 172.21.15.90:0/2375253172 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7e9c06be30 0x7f7e9c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:49.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.459+0000 7f7eb9400640 1 --2- 172.21.15.90:0/2375253172 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7e9c06be30 0x7f7e9c06e2f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8033000 tx=0x7f7ea8009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:49.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.463+0000 7f7eb2000640 1 -- 172.21.15.90:0/2375253172 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7eac059990 con 0x7f7ea0003f30 2024-04-25T01:18:49.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.694+0000 7f7ebb200640 1 -- 172.21.15.90:0/2375253172 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ebc0ae930 con 0x7f7ea0003f30 2024-04-25T01:18:49.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.695+0000 7f7eb2000640 1 -- 172.21.15.90:0/2375253172 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+5193 (secure 0 0 0) 0x7f7eac059290 con 0x7f7ea0003f30 2024-04-25T01:18:49.678 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:49.678 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":30,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1873497946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6833","nonce":1873497946},{"type":"v1","addr":"172.21.15.103:6835","nonce":1873497946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/3206865830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":3206865830},{"type":"v1","addr":"172.21.15.90:6835","nonce":3206865830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5598,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2208594231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2208594231},{"type":"v1","addr":"172.21.15.90:6837","nonce":2208594231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:18:19.371011+0000","modified":"2024-04-25T01:18:30.533899+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5601":{"gid":5601,"name":"d","rank":0,"incarnation":25,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6834/2850373997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":2850373997},{"type":"v1","addr":"172.21.15.103:6834","nonce":2850373997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":7,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5601,"qdb_cluster":[5601]},"id":3}]} 2024-04-25T01:18:49.678 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 30 2024-04-25T01:18:49.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.697+0000 7f7ebb200640 1 -- 172.21.15.90:0/2375253172 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7e9c06be30 msgr2=0x7f7e9c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:49.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.697+0000 7f7ebb200640 1 --2- 172.21.15.90:0/2375253172 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7e9c06be30 0x7f7e9c06e2f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8033000 tx=0x7f7ea8009030 comp rx=0 tx=0).stop 2024-04-25T01:18:49.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.697+0000 7f7ebb200640 1 -- 172.21.15.90:0/2375253172 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ea0003f30 msgr2=0x7f7ebc144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:49.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.697+0000 7f7ebb200640 1 --2- 172.21.15.90:0/2375253172 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ea0003f30 0x7f7ebc144f90 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f7eac00e7c0 tx=0x7f7eac00ec80 comp rx=0 tx=0).stop 2024-04-25T01:18:49.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.698+0000 7f7ebb200640 1 -- 172.21.15.90:0/2375253172 shutdown_connections 2024-04-25T01:18:49.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.698+0000 7f7ebb200640 1 --2- 172.21.15.90:0/2375253172 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7e9c06be30 0x7f7e9c06e2f0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:49.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.698+0000 7f7ebb200640 1 --2- 172.21.15.90:0/2375253172 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ebc1e3740 0x7f7ebc147770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:49.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.698+0000 7f7ebb200640 1 --2- 172.21.15.90:0/2375253172 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ebc1e32c0 0x7f7ebc15b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:49.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.698+0000 7f7ebb200640 1 --2- 172.21.15.90:0/2375253172 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ea0003f30 0x7f7ebc144f90 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:49.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.698+0000 7f7ebb200640 1 -- 172.21.15.90:0/2375253172 >> 172.21.15.90:0/2375253172 conn(0x7f7ebc06b320 msgr2=0x7f7ebc159310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:49.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.698+0000 7f7ebb200640 1 -- 172.21.15.90:0/2375253172 shutdown_connections 2024-04-25T01:18:49.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.698+0000 7f7ebb200640 1 -- 172.21.15.90:0/2375253172 wait complete. 2024-04-25T01:18:49.699 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:18:49.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.902+0000 7f7c13000640 1 Processor -- start 2024-04-25T01:18:49.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.902+0000 7f7c13000640 1 -- start start 2024-04-25T01:18:49.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.904+0000 7f7c13000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c0c145640 0x7f7c0c15ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:49.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.904+0000 7f7c13000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c0c0cdf70 con 0x7f7c0c0d03a0 2024-04-25T01:18:49.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.904+0000 7f7c13000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c0c15d360 con 0x7f7c0c0b2a20 2024-04-25T01:18:49.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.904+0000 7f7c13000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7c0c145a20 con 0x7f7c0c145640 2024-04-25T01:18:49.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.904+0000 7f7c11000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f7c0c0d03a0 0x7f7c0c0ce450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52846/0 (socket says 172.21.15.90:52846) 2024-04-25T01:18:49.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.904+0000 7f7c11000640 1 -- 172.21.15.90:0/2427991134 learned_addr learned my addr 172.21.15.90:0/2427991134 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:49.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.905+0000 7f7c0be00640 1 --2- 172.21.15.90:0/2427991134 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c0c145640 0x7f7c0c15ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:49.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.905+0000 7f7c0b400640 1 -- 172.21.15.90:0/2427991134 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 460752416 0 0) 0x7f7c0c0cdf70 con 0x7f7c0c0d03a0 2024-04-25T01:18:49.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.905+0000 7f7c0b400640 1 -- 172.21.15.90:0/2427991134 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7bf0003520 con 0x7f7c0c0d03a0 2024-04-25T01:18:49.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.905+0000 7f7c0b400640 1 -- 172.21.15.90:0/2427991134 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1547113843 0 0) 0x7f7bf0003520 con 0x7f7c0c0d03a0 2024-04-25T01:18:49.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.905+0000 7f7c0b400640 1 -- 172.21.15.90:0/2427991134 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c0c145640 msgr2=0x7f7c0c15ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:49.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.905+0000 7f7c0b400640 1 --2- 172.21.15.90:0/2427991134 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c0c145640 0x7f7c0c15ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:49.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.905+0000 7f7c0b400640 1 -- 172.21.15.90:0/2427991134 >> v1:172.21.15.103:6789/0 conn(0x7f7c0c0b2a20 legacy=0x7f7c0c0cfc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:49.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.905+0000 7f7c0b400640 1 -- 172.21.15.90:0/2427991134 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c0c06d140 con 0x7f7c0c0d03a0 2024-04-25T01:18:49.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.905+0000 7f7c0b400640 1 -- 172.21.15.90:0/2427991134 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f7c040019f0 con 0x7f7c0c0d03a0 2024-04-25T01:18:49.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.905+0000 7f7c0be00640 1 --2- 172.21.15.90:0/2427991134 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c0c145640 0x7f7c0c15ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:18:49.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.905+0000 7f7c0b400640 1 -- 172.21.15.90:0/2427991134 >> v1:172.21.15.90:6789/0 conn(0x7f7c0c0d03a0 legacy=0x7f7c0c0ce450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:49.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.905+0000 7f7c0b400640 1 --2- 172.21.15.90:0/2427991134 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c0c145640 0x7f7bf00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:49.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.905+0000 7f7c0b400640 1 --2- 172.21.15.90:0/2427991134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bf0003fd0 0x7f7bf0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:49.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.905+0000 7f7c0b400640 1 --2- 172.21.15.90:0/2427991134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7bf0004d40 0x7f7bf00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:49.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.905+0000 7f7c0b400640 1 -- 172.21.15.90:0/2427991134 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c145a20 con 0x7f7bf0004d40 2024-04-25T01:18:49.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.905+0000 7f7c0b400640 1 -- 172.21.15.90:0/2427991134 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf000bb20 con 0x7f7bf0003fd0 2024-04-25T01:18:49.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.905+0000 7f7c0b400640 1 -- 172.21.15.90:0/2427991134 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7bf000bd20 con 0x7f7c0c145640 2024-04-25T01:18:49.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.906+0000 7f7c11000640 1 --2- 172.21.15.90:0/2427991134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7bf0004d40 0x7f7bf00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:49.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.906+0000 7f7c0be00640 1 --2- 172.21.15.90:0/2427991134 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c0c145640 0x7f7bf00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:49.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.906+0000 7f7c11000640 1 -- 172.21.15.90:0/2427991134 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c0c145640 msgr2=0x7f7bf00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:49.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.906+0000 7f7c11000640 1 --2- 172.21.15.90:0/2427991134 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c0c145640 0x7f7bf00039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:49.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.906+0000 7f7c11000640 1 -- 172.21.15.90:0/2427991134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bf0003fd0 msgr2=0x7f7bf0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:49.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.906+0000 7f7c11000640 1 --2- 172.21.15.90:0/2427991134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bf0003fd0 0x7f7bf0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:49.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.906+0000 7f7c11000640 1 -- 172.21.15.90:0/2427991134 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bf0003520 con 0x7f7bf0004d40 2024-04-25T01:18:49.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.907+0000 7f7c11000640 1 --2- 172.21.15.90:0/2427991134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7bf0004d40 0x7f7bf00051c0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f7c0400a830 tx=0x7f7bf0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edaf932d8d172a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:49.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.907+0000 7f7c0b400640 1 -- 172.21.15.90:0/2427991134 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0400a480 con 0x7f7bf0004d40 2024-04-25T01:18:49.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.907+0000 7f7c0b400640 1 -- 172.21.15.90:0/2427991134 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c0402f070 con 0x7f7bf0004d40 2024-04-25T01:18:49.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.907+0000 7f7c0b400640 1 -- 172.21.15.90:0/2427991134 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0402bd00 con 0x7f7bf0004d40 2024-04-25T01:18:49.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.908+0000 7f7c13000640 1 -- 172.21.15.90:0/2427991134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7bf0004d40 msgr2=0x7f7bf00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:49.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.908+0000 7f7c13000640 1 --2- 172.21.15.90:0/2427991134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7bf0004d40 0x7f7bf00051c0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f7c0400a830 tx=0x7f7bf0005830 comp rx=0 tx=0).stop 2024-04-25T01:18:49.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.908+0000 7f7c11000640 1 -- 172.21.15.90:0/2427991134 reap_dead start 2024-04-25T01:18:49.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.908+0000 7f7c13000640 1 -- 172.21.15.90:0/2427991134 shutdown_connections 2024-04-25T01:18:49.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.908+0000 7f7c13000640 1 -- 172.21.15.90:0/2427991134 >> 172.21.15.90:0/2427991134 conn(0x7f7c0c06b320 msgr2=0x7f7c0c06b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:49.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.908+0000 7f7c13000640 1 -- 172.21.15.90:0/2427991134 shutdown_connections 2024-04-25T01:18:49.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.908+0000 7f7c13000640 1 -- 172.21.15.90:0/2427991134 wait complete. 2024-04-25T01:18:49.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.909+0000 7f7c13000640 1 Processor -- start 2024-04-25T01:18:49.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.910+0000 7f7c13000640 1 -- start start 2024-04-25T01:18:49.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.910+0000 7f7c13000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bf0004d40 0x7f7c0c143b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:49.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.910+0000 7f7c13000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c0c15bea0 0x7f7c0c15c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:49.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.910+0000 7f7c13000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c0c15c8a0 0x7f7c0c13f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:49.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.910+0000 7f7c13000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c06d9e0 con 0x7f7c0c15c8a0 2024-04-25T01:18:49.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.910+0000 7f7c13000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c06d700 con 0x7f7bf0004d40 2024-04-25T01:18:49.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.910+0000 7f7c13000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7c0c06cbc0 con 0x7f7c0c15bea0 2024-04-25T01:18:49.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.911+0000 7f7c11000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c0c15c8a0 0x7f7c0c13f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:49.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.911+0000 7f7c11000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c0c15c8a0 0x7f7c0c13f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47368/0 (socket says 172.21.15.90:47368) 2024-04-25T01:18:49.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.911+0000 7f7c11000640 1 -- 172.21.15.90:0/1200705709 learned_addr learned my addr 172.21.15.90:0/1200705709 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:49.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.911+0000 7f7c0be00640 1 --2- 172.21.15.90:0/1200705709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bf0004d40 0x7f7c0c143b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:49.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.911+0000 7f7c03400640 1 --2- 172.21.15.90:0/1200705709 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c0c15bea0 0x7f7c0c15c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:49.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.911+0000 7f7c11000640 1 -- 172.21.15.90:0/1200705709 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c0c15bea0 msgr2=0x7f7c0c15c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:49.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.911+0000 7f7c11000640 1 --2- 172.21.15.90:0/1200705709 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c0c15bea0 0x7f7c0c15c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:49.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.911+0000 7f7c11000640 1 -- 172.21.15.90:0/1200705709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bf0004d40 msgr2=0x7f7c0c143b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:49.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.911+0000 7f7c11000640 1 --2- 172.21.15.90:0/1200705709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bf0004d40 0x7f7c0c143b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:49.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.911+0000 7f7c11000640 1 -- 172.21.15.90:0/1200705709 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c0c142a70 con 0x7f7c0c15c8a0 2024-04-25T01:18:49.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.911+0000 7f7c03400640 1 --2- 172.21.15.90:0/1200705709 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c0c15bea0 0x7f7c0c15c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:49.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.911+0000 7f7c11000640 1 --2- 172.21.15.90:0/1200705709 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c0c15c8a0 0x7f7c0c13f8d0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f7c0400a950 tx=0x7f7c0c06d310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:49.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.912+0000 7f7c08c00640 1 -- 172.21.15.90:0/1200705709 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0400a480 con 0x7f7c0c15c8a0 2024-04-25T01:18:49.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.912+0000 7f7c08c00640 1 -- 172.21.15.90:0/1200705709 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c04004b60 con 0x7f7c0c15c8a0 2024-04-25T01:18:49.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.912+0000 7f7c08c00640 1 -- 172.21.15.90:0/1200705709 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c04034470 con 0x7f7c0c15c8a0 2024-04-25T01:18:49.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.912+0000 7f7c13000640 1 -- 172.21.15.90:0/1200705709 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c0c142d50 con 0x7f7c0c15c8a0 2024-04-25T01:18:49.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.912+0000 7f7c13000640 1 -- 172.21.15.90:0/1200705709 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c0c143310 con 0x7f7c0c15c8a0 2024-04-25T01:18:49.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.913+0000 7f7c13000640 1 -- 172.21.15.90:0/1200705709 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c0c068190 con 0x7f7c0c15c8a0 2024-04-25T01:18:49.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.914+0000 7f7c08c00640 1 -- 172.21.15.90:0/1200705709 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7c04034610 con 0x7f7c0c15c8a0 2024-04-25T01:18:49.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.915+0000 7f7c08c00640 1 --2- 172.21.15.90:0/1200705709 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7be806be60 0x7f7be806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:49.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.915+0000 7f7c08c00640 1 -- 172.21.15.90:0/1200705709 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6260+0+0 (secure 0 0 0) 0x7f7c040aac70 con 0x7f7c0c15c8a0 2024-04-25T01:18:49.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.915+0000 7f7c0be00640 1 --2- 172.21.15.90:0/1200705709 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7be806be60 0x7f7be806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:49.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.916+0000 7f7c0be00640 1 --2- 172.21.15.90:0/1200705709 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7be806be60 0x7f7be806e320 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc006fd0 tx=0x7f7bfc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:49.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:49.919+0000 7f7c08c00640 1 -- 172.21.15.90:0/1200705709 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7c040792c0 con 0x7f7c0c15c8a0 2024-04-25T01:18:50.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.109+0000 7f7c13000640 1 -- 172.21.15.90:0/1200705709 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7c0c0b4d10 con 0x7f7c0c15c8a0 2024-04-25T01:18:50.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.111+0000 7f7c08c00640 1 -- 172.21.15.90:0/1200705709 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) ==== 74+0+17080 (secure 0 0 0) 0x7f7c04078bc0 con 0x7f7c0c15c8a0 2024-04-25T01:18:50.094 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:50.095 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":51,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:18:47.003583+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T01:18:01.714313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"recovery_fs_metadata","create_time":"2024-04-25T01:18:18.384671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.16","mappings":[{"from":4,"to":2}]},{"pgid":"6.20","mappings":[{"from":4,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:6837/431390375":"2024-04-26T01:18:15.602554+0000","172.21.15.90:6836/431390375":"2024-04-26T01:18:15.602554+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:18:50.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.114+0000 7f7c13000640 1 -- 172.21.15.90:0/1200705709 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7be806be60 msgr2=0x7f7be806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:50.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.114+0000 7f7c13000640 1 --2- 172.21.15.90:0/1200705709 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7be806be60 0x7f7be806e320 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc006fd0 tx=0x7f7bfc008040 comp rx=0 tx=0).stop 2024-04-25T01:18:50.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.114+0000 7f7c13000640 1 -- 172.21.15.90:0/1200705709 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c0c15c8a0 msgr2=0x7f7c0c13f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:50.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.114+0000 7f7c13000640 1 --2- 172.21.15.90:0/1200705709 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c0c15c8a0 0x7f7c0c13f8d0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f7c0400a950 tx=0x7f7c0c06d310 comp rx=0 tx=0).stop 2024-04-25T01:18:50.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.114+0000 7f7c13000640 1 -- 172.21.15.90:0/1200705709 shutdown_connections 2024-04-25T01:18:50.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.114+0000 7f7c13000640 1 --2- 172.21.15.90:0/1200705709 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7be806be60 0x7f7be806e320 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:50.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.114+0000 7f7c13000640 1 --2- 172.21.15.90:0/1200705709 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c0c15c8a0 0x7f7c0c13f8d0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:50.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.114+0000 7f7c13000640 1 --2- 172.21.15.90:0/1200705709 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c0c15bea0 0x7f7c0c15c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:50.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.114+0000 7f7c13000640 1 --2- 172.21.15.90:0/1200705709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bf0004d40 0x7f7c0c143b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:50.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.114+0000 7f7c13000640 1 -- 172.21.15.90:0/1200705709 >> 172.21.15.90:0/1200705709 conn(0x7f7c0c06b320 msgr2=0x7f7c0c0b3690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:50.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.114+0000 7f7c13000640 1 -- 172.21.15.90:0/1200705709 shutdown_connections 2024-04-25T01:18:50.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.114+0000 7f7c13000640 1 -- 172.21.15.90:0/1200705709 wait complete. 2024-04-25T01:18:50.106 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:18:50.114 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:18:50.115 INFO:tasks.cephfs.filesystem:Destroying file system recovery_fs and related pools 2024-04-25T01:18:50.115 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:18:50.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.361+0000 7f64f3e00640 1 Processor -- start 2024-04-25T01:18:50.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.361+0000 7f64f3e00640 1 -- start start 2024-04-25T01:18:50.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.363+0000 7f64f3e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f64f415ddd0 0x7f64f4070b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:50.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.363+0000 7f64f3e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f64f40d0240 0x7f64f40b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:50.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.363+0000 7f64f3e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64f415e1b0 con 0x7f64f4072920 2024-04-25T01:18:50.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.363+0000 7f64f3e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f64f40713b0 con 0x7f64f40d0240 2024-04-25T01:18:50.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.363+0000 7f64f3e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f64f4071690 con 0x7f64f415ddd0 2024-04-25T01:18:50.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.363+0000 7f64f3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f64f40d0240 0x7f64f40b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:50.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.363+0000 7f64f2000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f64f4072920 0x7f64f40cfb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52856/0 (socket says 172.21.15.90:52856) 2024-04-25T01:18:50.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.363+0000 7f64f2000640 1 -- 172.21.15.90:0/1262067858 learned_addr learned my addr 172.21.15.90:0/1262067858 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:50.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.363+0000 7f64f2a00640 1 --2- 172.21.15.90:0/1262067858 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f64f415ddd0 0x7f64f4070b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:50.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.364+0000 7f64f3400640 1 -- 172.21.15.90:0/1262067858 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f64f415ddd0 msgr2=0x7f64f4070b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:50.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.364+0000 7f64f3400640 1 --2- 172.21.15.90:0/1262067858 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f64f415ddd0 0x7f64f4070b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:50.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.364+0000 7f64f1600640 1 -- 172.21.15.90:0/1262067858 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 280919471 0 0) 0x7f64f415e1b0 con 0x7f64f4072920 2024-04-25T01:18:50.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.364+0000 7f64f3400640 1 -- 172.21.15.90:0/1262067858 >> v1:172.21.15.90:6789/0 conn(0x7f64f4072920 legacy=0x7f64f40cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:50.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.364+0000 7f64f3400640 1 -- 172.21.15.90:0/1262067858 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64f4071970 con 0x7f64f40d0240 2024-04-25T01:18:50.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.364+0000 7f64f2a00640 1 --2- 172.21.15.90:0/1262067858 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f64f415ddd0 0x7f64f4070b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:50.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.365+0000 7f64f3400640 1 --2- 172.21.15.90:0/1262067858 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64f40d0240 0x7f64f40b2c00 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f64ec009dc0 tx=0x7f64ec02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7962cfbca706c4a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:50.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.365+0000 7f64f1600640 1 -- 172.21.15.90:0/1262067858 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ec02ae60 con 0x7f64f40d0240 2024-04-25T01:18:50.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.365+0000 7f64f1600640 1 -- 172.21.15.90:0/1262067858 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64ec007740 con 0x7f64f40d0240 2024-04-25T01:18:50.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.365+0000 7f64f1600640 1 -- 172.21.15.90:0/1262067858 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ec007b10 con 0x7f64f40d0240 2024-04-25T01:18:50.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.365+0000 7f64f3e00640 1 -- 172.21.15.90:0/1262067858 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64f40d0240 msgr2=0x7f64f40b2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:50.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.365+0000 7f64f3e00640 1 --2- 172.21.15.90:0/1262067858 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64f40d0240 0x7f64f40b2c00 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f64ec009dc0 tx=0x7f64ec02a680 comp rx=0 tx=0).stop 2024-04-25T01:18:50.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.366+0000 7f64f3e00640 1 -- 172.21.15.90:0/1262067858 shutdown_connections 2024-04-25T01:18:50.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.366+0000 7f64f3e00640 1 --2- 172.21.15.90:0/1262067858 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64f40d0240 0x7f64f40b2c00 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:50.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.366+0000 7f64f3e00640 1 --2- 172.21.15.90:0/1262067858 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f64f415ddd0 0x7f64f4070b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:50.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.366+0000 7f64f3e00640 1 -- 172.21.15.90:0/1262067858 >> 172.21.15.90:0/1262067858 conn(0x7f64f406b320 msgr2=0x7f64f406b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:50.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.366+0000 7f64f3e00640 1 -- 172.21.15.90:0/1262067858 shutdown_connections 2024-04-25T01:18:50.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.366+0000 7f64f3e00640 1 -- 172.21.15.90:0/1262067858 wait complete. 2024-04-25T01:18:50.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.367+0000 7f64f3e00640 1 Processor -- start 2024-04-25T01:18:50.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.367+0000 7f64f3e00640 1 -- start start 2024-04-25T01:18:50.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.368+0000 7f64f3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64f4072920 0x7f64f4155120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:50.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.368+0000 7f64f3e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f64f40d0240 0x7f64f415fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:50.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.368+0000 7f64f3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64f415ddd0 0x7f64f41463f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:50.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.368+0000 7f64f3e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f64f40b3140 con 0x7f64f415ddd0 2024-04-25T01:18:50.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.368+0000 7f64f3e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f64f40710a0 con 0x7f64f4072920 2024-04-25T01:18:50.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.368+0000 7f64f3e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f64f40b32c0 con 0x7f64f40d0240 2024-04-25T01:18:50.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.369+0000 7f64f3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64f415ddd0 0x7f64f41463f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:50.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.369+0000 7f64f3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64f415ddd0 0x7f64f41463f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47386/0 (socket says 172.21.15.90:47386) 2024-04-25T01:18:50.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.369+0000 7f64f3400640 1 -- 172.21.15.90:0/1927604813 learned_addr learned my addr 172.21.15.90:0/1927604813 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:50.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.369+0000 7f64f2000640 1 --2- 172.21.15.90:0/1927604813 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f64f40d0240 0x7f64f415fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:50.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.369+0000 7f64f2a00640 1 --2- 172.21.15.90:0/1927604813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64f4072920 0x7f64f4155120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:50.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.369+0000 7f64f3400640 1 -- 172.21.15.90:0/1927604813 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f64f40d0240 msgr2=0x7f64f415fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:50.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.369+0000 7f64f3400640 1 --2- 172.21.15.90:0/1927604813 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f64f40d0240 0x7f64f415fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:50.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.369+0000 7f64f3400640 1 -- 172.21.15.90:0/1927604813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64f4072920 msgr2=0x7f64f4155120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:50.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.369+0000 7f64f3400640 1 --2- 172.21.15.90:0/1927604813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64f4072920 0x7f64f4155120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:50.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.369+0000 7f64f3400640 1 -- 172.21.15.90:0/1927604813 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64f41540c0 con 0x7f64f415ddd0 2024-04-25T01:18:50.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.369+0000 7f64f2a00640 1 --2- 172.21.15.90:0/1927604813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64f4072920 0x7f64f4155120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:50.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.369+0000 7f64f3400640 1 --2- 172.21.15.90:0/1927604813 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64f415ddd0 0x7f64f41463f0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f64ec009ee0 tx=0x7f64ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:50.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.370+0000 7f64eb400640 1 -- 172.21.15.90:0/1927604813 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ec040070 con 0x7f64f415ddd0 2024-04-25T01:18:50.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.370+0000 7f64eb400640 1 -- 172.21.15.90:0/1927604813 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64ec02ae60 con 0x7f64f415ddd0 2024-04-25T01:18:50.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.370+0000 7f64f3e00640 1 -- 172.21.15.90:0/1927604813 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64f41603d0 con 0x7f64f415ddd0 2024-04-25T01:18:50.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.370+0000 7f64eb400640 1 -- 172.21.15.90:0/1927604813 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ec045460 con 0x7f64f415ddd0 2024-04-25T01:18:50.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.370+0000 7f64f3e00640 1 -- 172.21.15.90:0/1927604813 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64f4154640 con 0x7f64f415ddd0 2024-04-25T01:18:50.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.371+0000 7f64f3e00640 1 -- 172.21.15.90:0/1927604813 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64f404f500 con 0x7f64f415ddd0 2024-04-25T01:18:50.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.372+0000 7f64eb400640 1 -- 172.21.15.90:0/1927604813 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f64ec039040 con 0x7f64f415ddd0 2024-04-25T01:18:50.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.373+0000 7f64eb400640 1 --2- 172.21.15.90:0/1927604813 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f64d406be30 0x7f64d406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:50.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.374+0000 7f64eb400640 1 -- 172.21.15.90:0/1927604813 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6260+0+0 (secure 0 0 0) 0x7f64ec0ac900 con 0x7f64f415ddd0 2024-04-25T01:18:50.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.374+0000 7f64f2a00640 1 --2- 172.21.15.90:0/1927604813 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f64d406be30 0x7f64d406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:50.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.374+0000 7f64f2a00640 1 --2- 172.21.15.90:0/1927604813 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f64d406be30 0x7f64d406e2f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f64e4006fd0 tx=0x7f64e4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:50.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.378+0000 7f64eb400640 1 -- 172.21.15.90:0/1927604813 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f64ec07abe0 con 0x7f64f415ddd0 2024-04-25T01:18:50.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.619+0000 7f64f3e00640 1 -- 172.21.15.90:0/1927604813 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f64f40008d0 con 0x7f64f415ddd0 2024-04-25T01:18:50.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.619+0000 7f64eb400640 1 -- 172.21.15.90:0/1927604813 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+5193 (secure 0 0 0) 0x7f64ec07eee0 con 0x7f64f415ddd0 2024-04-25T01:18:50.602 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:50.603 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":30,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1873497946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6833","nonce":1873497946},{"type":"v1","addr":"172.21.15.103:6835","nonce":1873497946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/3206865830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":3206865830},{"type":"v1","addr":"172.21.15.90:6835","nonce":3206865830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5598,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2208594231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2208594231},{"type":"v1","addr":"172.21.15.90:6837","nonce":2208594231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:18:19.371011+0000","modified":"2024-04-25T01:18:30.533899+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5601":{"gid":5601,"name":"d","rank":0,"incarnation":25,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6834/2850373997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":2850373997},{"type":"v1","addr":"172.21.15.103:6834","nonce":2850373997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":7,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5601,"qdb_cluster":[5601]},"id":3}]} 2024-04-25T01:18:50.603 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 30 2024-04-25T01:18:50.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.622+0000 7f64f3e00640 1 -- 172.21.15.90:0/1927604813 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f64d406be30 msgr2=0x7f64d406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:50.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.622+0000 7f64f3e00640 1 --2- 172.21.15.90:0/1927604813 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f64d406be30 0x7f64d406e2f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f64e4006fd0 tx=0x7f64e4008040 comp rx=0 tx=0).stop 2024-04-25T01:18:50.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.622+0000 7f64f3e00640 1 -- 172.21.15.90:0/1927604813 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64f415ddd0 msgr2=0x7f64f41463f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:50.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.622+0000 7f64f3e00640 1 --2- 172.21.15.90:0/1927604813 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64f415ddd0 0x7f64f41463f0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f64ec009ee0 tx=0x7f64ec004a40 comp rx=0 tx=0).stop 2024-04-25T01:18:50.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.623+0000 7f64f3e00640 1 -- 172.21.15.90:0/1927604813 shutdown_connections 2024-04-25T01:18:50.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.623+0000 7f64f3e00640 1 --2- 172.21.15.90:0/1927604813 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f64d406be30 0x7f64d406e2f0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:50.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.623+0000 7f64f3e00640 1 --2- 172.21.15.90:0/1927604813 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64f415ddd0 0x7f64f41463f0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:50.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.623+0000 7f64f3e00640 1 --2- 172.21.15.90:0/1927604813 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f64f40d0240 0x7f64f415fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:50.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.623+0000 7f64f3e00640 1 --2- 172.21.15.90:0/1927604813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64f4072920 0x7f64f4155120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:50.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.623+0000 7f64f3e00640 1 -- 172.21.15.90:0/1927604813 >> 172.21.15.90:0/1927604813 conn(0x7f64f406b320 msgr2=0x7f64f40cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:50.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.623+0000 7f64f3e00640 1 -- 172.21.15.90:0/1927604813 shutdown_connections 2024-04-25T01:18:50.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.623+0000 7f64f3e00640 1 -- 172.21.15.90:0/1927604813 wait complete. 2024-04-25T01:18:50.626 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:18:50.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.838+0000 7fd59f800640 1 Processor -- start 2024-04-25T01:18:50.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.838+0000 7fd59f800640 1 -- start start 2024-04-25T01:18:50.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.840+0000 7fd59f800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd59806cb60 0x7fd59806d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:50.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.840+0000 7fd59f800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd5980d2870 0x7fd5980b2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:50.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.840+0000 7fd59f800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5980d4b60 con 0x7fd598161800 2024-04-25T01:18:50.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.840+0000 7fd59f800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fd5980d4e40 con 0x7fd59806cb60 2024-04-25T01:18:50.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.840+0000 7fd59f800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd5980b1b70 con 0x7fd5980d2870 2024-04-25T01:18:50.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.840+0000 7fd597e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd59806cb60 0x7fd59806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:50.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.840+0000 7fd597e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd59806cb60 0x7fd59806d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47388/0 (socket says 172.21.15.90:47388) 2024-04-25T01:18:50.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.840+0000 7fd597e00640 1 -- 172.21.15.90:0/2442451189 learned_addr learned my addr 172.21.15.90:0/2442451189 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:50.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.841+0000 7fd59d800640 1 --2- 172.21.15.90:0/2442451189 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd5980d2870 0x7fd5980b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:50.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.841+0000 7fd597e00640 1 -- 172.21.15.90:0/2442451189 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd5980d2870 msgr2=0x7fd5980b2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:50.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.841+0000 7fd597e00640 1 --2- 172.21.15.90:0/2442451189 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd5980d2870 0x7fd5980b2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:50.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.841+0000 7fd597e00640 1 -- 172.21.15.90:0/2442451189 >> v1:172.21.15.103:6789/0 conn(0x7fd598161800 legacy=0x7fd5980d2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:50.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.841+0000 7fd597e00640 1 -- 172.21.15.90:0/2442451189 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5980b1800 con 0x7fd59806cb60 2024-04-25T01:18:50.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.841+0000 7fd59d800640 1 --2- 172.21.15.90:0/2442451189 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd5980d2870 0x7fd5980b2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:50.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.842+0000 7fd597e00640 1 --2- 172.21.15.90:0/2442451189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd59806cb60 0x7fd59806d790 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fd588009870 tx=0x7fd58802a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a86869661c23774 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:50.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.842+0000 7fd597400640 1 -- 172.21.15.90:0/2442451189 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd588037070 con 0x7fd59806cb60 2024-04-25T01:18:50.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.842+0000 7fd597400640 1 -- 172.21.15.90:0/2442451189 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd588032440 con 0x7fd59806cb60 2024-04-25T01:18:50.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.842+0000 7fd597400640 1 -- 172.21.15.90:0/2442451189 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd588032810 con 0x7fd59806cb60 2024-04-25T01:18:50.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.843+0000 7fd59f800640 1 -- 172.21.15.90:0/2442451189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd59806cb60 msgr2=0x7fd59806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:50.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.843+0000 7fd59f800640 1 --2- 172.21.15.90:0/2442451189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd59806cb60 0x7fd59806d790 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fd588009870 tx=0x7fd58802a2a0 comp rx=0 tx=0).stop 2024-04-25T01:18:50.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.843+0000 7fd59f800640 1 -- 172.21.15.90:0/2442451189 shutdown_connections 2024-04-25T01:18:50.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.843+0000 7fd59f800640 1 --2- 172.21.15.90:0/2442451189 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd5980d2870 0x7fd5980b2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:50.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.843+0000 7fd59f800640 1 --2- 172.21.15.90:0/2442451189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd59806cb60 0x7fd59806d790 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:50.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.843+0000 7fd59f800640 1 -- 172.21.15.90:0/2442451189 >> 172.21.15.90:0/2442451189 conn(0x7fd5980d3ea0 msgr2=0x7fd5980d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:50.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.843+0000 7fd59f800640 1 -- 172.21.15.90:0/2442451189 shutdown_connections 2024-04-25T01:18:50.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.843+0000 7fd59f800640 1 -- 172.21.15.90:0/2442451189 wait complete. 2024-04-25T01:18:50.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.844+0000 7fd59f800640 1 Processor -- start 2024-04-25T01:18:50.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.845+0000 7fd59f800640 1 -- start start 2024-04-25T01:18:50.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.845+0000 7fd59f800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd59806cb60 0x7fd5981f4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:50.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.846+0000 7fd59f800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5980d2870 0x7fd5981f6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:50.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.846+0000 7fd59f800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd598161800 0x7fd5980cb980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:50.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.846+0000 7fd59f800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd59806d130 con 0x7fd598161800 2024-04-25T01:18:50.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.846+0000 7fd59f800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd59806cfb0 con 0x7fd5980d2870 2024-04-25T01:18:50.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.846+0000 7fd59f800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd5980b2540 con 0x7fd59806cb60 2024-04-25T01:18:50.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.846+0000 7fd59d800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd598161800 0x7fd5980cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:50.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.846+0000 7fd59d800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd598161800 0x7fd5980cb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47390/0 (socket says 172.21.15.90:47390) 2024-04-25T01:18:50.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.846+0000 7fd59d800640 1 -- 172.21.15.90:0/2187978045 learned_addr learned my addr 172.21.15.90:0/2187978045 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:50.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.846+0000 7fd59ce00640 1 --2- 172.21.15.90:0/2187978045 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd59806cb60 0x7fd5981f4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:50.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.846+0000 7fd597e00640 1 --2- 172.21.15.90:0/2187978045 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5980d2870 0x7fd5981f6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:50.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.846+0000 7fd59d800640 1 -- 172.21.15.90:0/2187978045 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd59806cb60 msgr2=0x7fd5981f4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:50.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.846+0000 7fd59d800640 1 --2- 172.21.15.90:0/2187978045 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd59806cb60 0x7fd5981f4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:50.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.846+0000 7fd59d800640 1 -- 172.21.15.90:0/2187978045 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5980d2870 msgr2=0x7fd5981f6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:50.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.846+0000 7fd59d800640 1 --2- 172.21.15.90:0/2187978045 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5980d2870 0x7fd5981f6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:50.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.846+0000 7fd59d800640 1 -- 172.21.15.90:0/2187978045 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5980cc0f0 con 0x7fd598161800 2024-04-25T01:18:50.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.846+0000 7fd59ce00640 1 --2- 172.21.15.90:0/2187978045 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd59806cb60 0x7fd5981f4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:50.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.846+0000 7fd597e00640 1 --2- 172.21.15.90:0/2187978045 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5980d2870 0x7fd5981f6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:50.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.847+0000 7fd59d800640 1 --2- 172.21.15.90:0/2187978045 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd598161800 0x7fd5980cb980 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fd59000ce40 tx=0x7fd59000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:50.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.847+0000 7fd594c00640 1 -- 172.21.15.90:0/2187978045 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd590003b90 con 0x7fd598161800 2024-04-25T01:18:50.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.847+0000 7fd594c00640 1 -- 172.21.15.90:0/2187978045 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd590003d50 con 0x7fd598161800 2024-04-25T01:18:50.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.847+0000 7fd59f800640 1 -- 172.21.15.90:0/2187978045 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5980c7c20 con 0x7fd598161800 2024-04-25T01:18:50.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.847+0000 7fd594c00640 1 -- 172.21.15.90:0/2187978045 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd590004260 con 0x7fd598161800 2024-04-25T01:18:50.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.847+0000 7fd59f800640 1 -- 172.21.15.90:0/2187978045 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5980c81e0 con 0x7fd598161800 2024-04-25T01:18:50.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.848+0000 7fd59f800640 1 -- 172.21.15.90:0/2187978045 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd598068190 con 0x7fd598161800 2024-04-25T01:18:50.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.849+0000 7fd594c00640 1 -- 172.21.15.90:0/2187978045 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd590004400 con 0x7fd598161800 2024-04-25T01:18:50.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.851+0000 7fd594c00640 1 --2- 172.21.15.90:0/2187978045 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd57c06be30 0x7fd57c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:50.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.851+0000 7fd594c00640 1 -- 172.21.15.90:0/2187978045 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6260+0+0 (secure 0 0 0) 0x7fd59008ba40 con 0x7fd598161800 2024-04-25T01:18:50.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.851+0000 7fd59ce00640 1 --2- 172.21.15.90:0/2187978045 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd57c06be30 0x7fd57c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:50.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.852+0000 7fd59ce00640 1 --2- 172.21.15.90:0/2187978045 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd57c06be30 0x7fd57c06e2f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd59815fd70 tx=0x7fd58c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:50.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:50.855+0000 7fd594c00640 1 -- 172.21.15.90:0/2187978045 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd59005a1b0 con 0x7fd598161800 2024-04-25T01:18:51.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.091+0000 7fd59f800640 1 -- 172.21.15.90:0/2187978045 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd59808dfc0 con 0x7fd598161800 2024-04-25T01:18:51.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.093+0000 7fd594c00640 1 -- 172.21.15.90:0/2187978045 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+5193 (secure 0 0 0) 0x7fd590059ab0 con 0x7fd598161800 2024-04-25T01:18:51.076 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:51.076 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":30,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1873497946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6833","nonce":1873497946},{"type":"v1","addr":"172.21.15.103:6835","nonce":1873497946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/3206865830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":3206865830},{"type":"v1","addr":"172.21.15.90:6835","nonce":3206865830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5598,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2208594231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2208594231},{"type":"v1","addr":"172.21.15.90:6837","nonce":2208594231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:18:19.371011+0000","modified":"2024-04-25T01:18:30.533899+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5601":{"gid":5601,"name":"d","rank":0,"incarnation":25,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6834/2850373997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":2850373997},{"type":"v1","addr":"172.21.15.103:6834","nonce":2850373997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":7,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5601,"qdb_cluster":[5601]},"id":3}]} 2024-04-25T01:18:51.076 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 30 2024-04-25T01:18:51.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.095+0000 7fd59f800640 1 -- 172.21.15.90:0/2187978045 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd57c06be30 msgr2=0x7fd57c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:51.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.095+0000 7fd59f800640 1 --2- 172.21.15.90:0/2187978045 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd57c06be30 0x7fd57c06e2f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd59815fd70 tx=0x7fd58c008040 comp rx=0 tx=0).stop 2024-04-25T01:18:51.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.095+0000 7fd59f800640 1 -- 172.21.15.90:0/2187978045 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd598161800 msgr2=0x7fd5980cb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:51.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.095+0000 7fd59f800640 1 --2- 172.21.15.90:0/2187978045 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd598161800 0x7fd5980cb980 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fd59000ce40 tx=0x7fd59000a710 comp rx=0 tx=0).stop 2024-04-25T01:18:51.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.096+0000 7fd59f800640 1 -- 172.21.15.90:0/2187978045 shutdown_connections 2024-04-25T01:18:51.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.096+0000 7fd59f800640 1 --2- 172.21.15.90:0/2187978045 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd57c06be30 0x7fd57c06e2f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:51.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.096+0000 7fd59f800640 1 --2- 172.21.15.90:0/2187978045 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd598161800 0x7fd5980cb980 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:51.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.096+0000 7fd59f800640 1 --2- 172.21.15.90:0/2187978045 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5980d2870 0x7fd5981f6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:51.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.096+0000 7fd59f800640 1 --2- 172.21.15.90:0/2187978045 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd59806cb60 0x7fd5981f4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:51.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.096+0000 7fd59f800640 1 -- 172.21.15.90:0/2187978045 >> 172.21.15.90:0/2187978045 conn(0x7fd5980d3ea0 msgr2=0x7fd5980cf4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:51.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.096+0000 7fd59f800640 1 -- 172.21.15.90:0/2187978045 shutdown_connections 2024-04-25T01:18:51.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.096+0000 7fd59f800640 1 -- 172.21.15.90:0/2187978045 wait complete. 2024-04-25T01:18:51.096 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:18:51.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.309+0000 7f9183a00640 1 Processor -- start 2024-04-25T01:18:51.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.309+0000 7f9183a00640 1 -- start start 2024-04-25T01:18:51.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.310+0000 7f9183a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f917c1445e0 0x7f917c0cfc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:51.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.310+0000 7f9183a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f917c0728c0 0x7f917c0d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:51.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.311+0000 7f9183a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f917c1449c0 con 0x7f917c070c10 2024-04-25T01:18:51.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.311+0000 7f9183a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f917c072430 con 0x7f917c0728c0 2024-04-25T01:18:51.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.311+0000 7f9183a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f917c0b2d70 con 0x7f917c1445e0 2024-04-25T01:18:51.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.311+0000 7f9173e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f917c0728c0 0x7f917c0d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:51.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.311+0000 7f9173e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f917c0728c0 0x7f917c0d01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47402/0 (socket says 172.21.15.90:47402) 2024-04-25T01:18:51.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.311+0000 7f9173e00640 1 -- 172.21.15.90:0/192053402 learned_addr learned my addr 172.21.15.90:0/192053402 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:51.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.311+0000 7f9181000640 1 --2- 172.21.15.90:0/192053402 >> v2:172.21.15.103:3300/0 conn(0x7f917c1445e0 0x7f917c0cfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:51.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.311+0000 7f9173e00640 1 -- 172.21.15.90:0/192053402 >> v2:172.21.15.103:3300/0 conn(0x7f917c1445e0 msgr2=0x7f917c0cfc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:51.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.311+0000 7f9173e00640 1 --2- 172.21.15.90:0/192053402 >> v2:172.21.15.103:3300/0 conn(0x7f917c1445e0 0x7f917c0cfc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:51.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.311+0000 7f9173e00640 1 -- 172.21.15.90:0/192053402 >> v1:172.21.15.103:6789/0 conn(0x7f917c070c10 legacy=0x7f917c15e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:51.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.311+0000 7f9173e00640 1 -- 172.21.15.90:0/192053402 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f917c071820 con 0x7f917c0728c0 2024-04-25T01:18:51.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.312+0000 7f9173e00640 1 --2- 172.21.15.90:0/192053402 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f917c0728c0 0x7f917c0d01d0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f917c0d08a0 tx=0x7f916402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2a7e9f9f148093f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:51.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.313+0000 7f917be00640 1 -- 172.21.15.90:0/192053402 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9164037070 con 0x7f917c0728c0 2024-04-25T01:18:51.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.313+0000 7f917be00640 1 -- 172.21.15.90:0/192053402 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9164032440 con 0x7f917c0728c0 2024-04-25T01:18:51.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.313+0000 7f917be00640 1 -- 172.21.15.90:0/192053402 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9164032810 con 0x7f917c0728c0 2024-04-25T01:18:51.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.313+0000 7f9183a00640 1 -- 172.21.15.90:0/192053402 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f917c0728c0 msgr2=0x7f917c0d01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:51.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.313+0000 7f9183a00640 1 --2- 172.21.15.90:0/192053402 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f917c0728c0 0x7f917c0d01d0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f917c0d08a0 tx=0x7f916402a230 comp rx=0 tx=0).stop 2024-04-25T01:18:51.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.313+0000 7f9183a00640 1 -- 172.21.15.90:0/192053402 shutdown_connections 2024-04-25T01:18:51.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.313+0000 7f9183a00640 1 --2- 172.21.15.90:0/192053402 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f917c0728c0 0x7f917c0d01d0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:51.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.313+0000 7f9183a00640 1 --2- 172.21.15.90:0/192053402 >> v2:172.21.15.103:3300/0 conn(0x7f917c1445e0 0x7f917c0cfc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:51.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.313+0000 7f9183a00640 1 -- 172.21.15.90:0/192053402 >> 172.21.15.90:0/192053402 conn(0x7f917c06b320 msgr2=0x7f917c06b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:51.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.313+0000 7f9183a00640 1 -- 172.21.15.90:0/192053402 shutdown_connections 2024-04-25T01:18:51.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.313+0000 7f9183a00640 1 -- 172.21.15.90:0/192053402 wait complete. 2024-04-25T01:18:51.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.315+0000 7f9183a00640 1 Processor -- start 2024-04-25T01:18:51.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.315+0000 7f9183a00640 1 -- start start 2024-04-25T01:18:51.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.316+0000 7f9183a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f917c070c10 0x7f917c15dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:51.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.316+0000 7f9183a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f917c0728c0 0x7f917c0cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:51.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.316+0000 7f9183a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f917c1445e0 0x7f917c13ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:51.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.316+0000 7f9183a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f917c0720c0 con 0x7f917c070c10 2024-04-25T01:18:51.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.316+0000 7f9183a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f917c071de0 con 0x7f917c0728c0 2024-04-25T01:18:51.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.316+0000 7f9183a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f917c071080 con 0x7f917c1445e0 2024-04-25T01:18:51.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.316+0000 7f9181000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f917c070c10 0x7f917c15dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:51.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.317+0000 7f9181000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f917c070c10 0x7f917c15dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47414/0 (socket says 172.21.15.90:47414) 2024-04-25T01:18:51.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.317+0000 7f9181000640 1 -- 172.21.15.90:0/2449868716 learned_addr learned my addr 172.21.15.90:0/2449868716 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:51.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.317+0000 7f9173e00640 1 --2- 172.21.15.90:0/2449868716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f917c0728c0 0x7f917c0cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:51.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.317+0000 7f9181a00640 1 --2- 172.21.15.90:0/2449868716 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f917c1445e0 0x7f917c13ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:51.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.317+0000 7f9181000640 1 -- 172.21.15.90:0/2449868716 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f917c1445e0 msgr2=0x7f917c13ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:51.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.317+0000 7f9181000640 1 --2- 172.21.15.90:0/2449868716 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f917c1445e0 0x7f917c13ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:51.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.317+0000 7f9181000640 1 -- 172.21.15.90:0/2449868716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f917c0728c0 msgr2=0x7f917c0cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:51.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.317+0000 7f9181000640 1 --2- 172.21.15.90:0/2449868716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f917c0728c0 0x7f917c0cdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:51.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.317+0000 7f9181000640 1 -- 172.21.15.90:0/2449868716 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f917c0cf850 con 0x7f917c070c10 2024-04-25T01:18:51.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.317+0000 7f9173e00640 1 --2- 172.21.15.90:0/2449868716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f917c0728c0 0x7f917c0cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:51.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.317+0000 7f9181000640 1 --2- 172.21.15.90:0/2449868716 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f917c070c10 0x7f917c15dac0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f916c00d620 tx=0x7f916c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:51.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.318+0000 7f9179600640 1 -- 172.21.15.90:0/2449868716 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f916c00b620 con 0x7f917c070c10 2024-04-25T01:18:51.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.318+0000 7f9183a00640 1 -- 172.21.15.90:0/2449868716 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f917c0ce7b0 con 0x7f917c070c10 2024-04-25T01:18:51.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.318+0000 7f9183a00640 1 -- 172.21.15.90:0/2449868716 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f917c0ced70 con 0x7f917c070c10 2024-04-25T01:18:51.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.318+0000 7f9179600640 1 -- 172.21.15.90:0/2449868716 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f916c019070 con 0x7f917c070c10 2024-04-25T01:18:51.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.318+0000 7f9179600640 1 -- 172.21.15.90:0/2449868716 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f916c00ba00 con 0x7f917c070c10 2024-04-25T01:18:51.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.319+0000 7f9183a00640 1 -- 172.21.15.90:0/2449868716 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9154005290 con 0x7f917c070c10 2024-04-25T01:18:51.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.320+0000 7f9179600640 1 -- 172.21.15.90:0/2449868716 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f916c00bba0 con 0x7f917c070c10 2024-04-25T01:18:51.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.321+0000 7f9179600640 1 --2- 172.21.15.90:0/2449868716 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f915806be30 0x7f915806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:51.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.321+0000 7f9179600640 1 -- 172.21.15.90:0/2449868716 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6260+0+0 (secure 0 0 0) 0x7f916c08b470 con 0x7f917c070c10 2024-04-25T01:18:51.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.322+0000 7f9173e00640 1 --2- 172.21.15.90:0/2449868716 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f915806be30 0x7f915806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:51.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.322+0000 7f9173e00640 1 --2- 172.21.15.90:0/2449868716 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f915806be30 0x7f915806e2f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9164034000 tx=0x7f916402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:51.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.326+0000 7f9179600640 1 -- 172.21.15.90:0/2449868716 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f916c059be0 con 0x7f917c070c10 2024-04-25T01:18:51.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.515+0000 7f9183a00640 1 -- 172.21.15.90:0/2449868716 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9154005580 con 0x7f917c070c10 2024-04-25T01:18:51.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.518+0000 7f9179600640 1 -- 172.21.15.90:0/2449868716 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) ==== 74+0+17080 (secure 0 0 0) 0x7f916c0594e0 con 0x7f917c070c10 2024-04-25T01:18:51.501 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:51.501 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":51,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:18:47.003583+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T01:18:01.714313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"recovery_fs_metadata","create_time":"2024-04-25T01:18:18.384671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.16","mappings":[{"from":4,"to":2}]},{"pgid":"6.20","mappings":[{"from":4,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:6837/431390375":"2024-04-26T01:18:15.602554+0000","172.21.15.90:6836/431390375":"2024-04-26T01:18:15.602554+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:18:51.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.520+0000 7f9183a00640 1 -- 172.21.15.90:0/2449868716 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f915806be30 msgr2=0x7f915806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:51.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.520+0000 7f9183a00640 1 --2- 172.21.15.90:0/2449868716 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f915806be30 0x7f915806e2f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9164034000 tx=0x7f916402e000 comp rx=0 tx=0).stop 2024-04-25T01:18:51.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.520+0000 7f9183a00640 1 -- 172.21.15.90:0/2449868716 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f917c070c10 msgr2=0x7f917c15dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:51.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.520+0000 7f9183a00640 1 --2- 172.21.15.90:0/2449868716 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f917c070c10 0x7f917c15dac0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f916c00d620 tx=0x7f916c00dae0 comp rx=0 tx=0).stop 2024-04-25T01:18:51.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.521+0000 7f9183a00640 1 -- 172.21.15.90:0/2449868716 shutdown_connections 2024-04-25T01:18:51.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.521+0000 7f9183a00640 1 --2- 172.21.15.90:0/2449868716 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f915806be30 0x7f915806e2f0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:51.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.521+0000 7f9183a00640 1 --2- 172.21.15.90:0/2449868716 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f917c1445e0 0x7f917c13ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:51.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.521+0000 7f9183a00640 1 --2- 172.21.15.90:0/2449868716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f917c0728c0 0x7f917c0cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:51.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.521+0000 7f9183a00640 1 --2- 172.21.15.90:0/2449868716 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f917c070c10 0x7f917c15dac0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:51.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.521+0000 7f9183a00640 1 -- 172.21.15.90:0/2449868716 >> 172.21.15.90:0/2449868716 conn(0x7f917c06b320 msgr2=0x7f917c0afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:51.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.521+0000 7f9183a00640 1 -- 172.21.15.90:0/2449868716 shutdown_connections 2024-04-25T01:18:51.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.521+0000 7f9183a00640 1 -- 172.21.15.90:0/2449868716 wait complete. 2024-04-25T01:18:51.523 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail recovery_fs 2024-04-25T01:18:51.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.738+0000 7ff2bd800640 1 Processor -- start 2024-04-25T01:18:51.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.738+0000 7ff2bd800640 1 -- start start 2024-04-25T01:18:51.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.739+0000 7ff2bd800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff2b815bdc0 0x7ff2b8072690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:51.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.740+0000 7ff2bd800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff2b815cd80 0x7ff2b80718c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:51.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.740+0000 7ff2bd800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2b815c1a0 con 0x7ff2b80ae830 2024-04-25T01:18:51.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.740+0000 7ff2bd800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7ff2b80cc5d0 con 0x7ff2b815bdc0 2024-04-25T01:18:51.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.740+0000 7ff2bd800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff2b80cf6b0 con 0x7ff2b815cd80 2024-04-25T01:18:51.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.740+0000 7ff2b7e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff2b815bdc0 0x7ff2b8072690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:51.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.740+0000 7ff2b7e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff2b815bdc0 0x7ff2b8072690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47430/0 (socket says 172.21.15.90:47430) 2024-04-25T01:18:51.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.740+0000 7ff2bce00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff2b80ae830 0x7ff2b80cdc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52872/0 (socket says 172.21.15.90:52872) 2024-04-25T01:18:51.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.740+0000 7ff2b7e00640 1 -- 172.21.15.90:0/795220241 learned_addr learned my addr 172.21.15.90:0/795220241 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:51.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.740+0000 7ff2b7400640 1 --2- 172.21.15.90:0/795220241 >> v2:172.21.15.103:3300/0 conn(0x7ff2b815cd80 0x7ff2b80718c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:51.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.740+0000 7ff2b7e00640 1 -- 172.21.15.90:0/795220241 >> v2:172.21.15.103:3300/0 conn(0x7ff2b815cd80 msgr2=0x7ff2b80718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:51.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.740+0000 7ff2b7e00640 1 --2- 172.21.15.90:0/795220241 >> v2:172.21.15.103:3300/0 conn(0x7ff2b815cd80 0x7ff2b80718c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:51.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.740+0000 7ff2b7e00640 1 -- 172.21.15.90:0/795220241 >> v1:172.21.15.90:6789/0 conn(0x7ff2b80ae830 legacy=0x7ff2b80cdc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:51.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.740+0000 7ff2b7e00640 1 -- 172.21.15.90:0/795220241 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2b815d210 con 0x7ff2b815bdc0 2024-04-25T01:18:51.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.741+0000 7ff2b7e00640 1 --2- 172.21.15.90:0/795220241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2b815bdc0 0x7ff2b8072690 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7ff2b80cc1c0 tx=0x7ff2a802a1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6658ab62f58448a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:51.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.742+0000 7ff2b6a00640 1 -- 172.21.15.90:0/795220241 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a8037070 con 0x7ff2b815bdc0 2024-04-25T01:18:51.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.742+0000 7ff2b6a00640 1 -- 172.21.15.90:0/795220241 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2a8032440 con 0x7ff2b815bdc0 2024-04-25T01:18:51.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.742+0000 7ff2b6a00640 1 -- 172.21.15.90:0/795220241 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a8032810 con 0x7ff2b815bdc0 2024-04-25T01:18:51.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.742+0000 7ff2bd800640 1 -- 172.21.15.90:0/795220241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2b815bdc0 msgr2=0x7ff2b8072690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:51.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.742+0000 7ff2bd800640 1 --2- 172.21.15.90:0/795220241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2b815bdc0 0x7ff2b8072690 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7ff2b80cc1c0 tx=0x7ff2a802a1f0 comp rx=0 tx=0).stop 2024-04-25T01:18:51.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.742+0000 7ff2bd800640 1 -- 172.21.15.90:0/795220241 shutdown_connections 2024-04-25T01:18:51.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.742+0000 7ff2bd800640 1 --2- 172.21.15.90:0/795220241 >> v2:172.21.15.103:3300/0 conn(0x7ff2b815cd80 0x7ff2b80718c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:51.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.742+0000 7ff2bd800640 1 --2- 172.21.15.90:0/795220241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2b815bdc0 0x7ff2b8072690 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:51.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.742+0000 7ff2bd800640 1 -- 172.21.15.90:0/795220241 >> 172.21.15.90:0/795220241 conn(0x7ff2b80b1560 msgr2=0x7ff2b80b1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:51.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.743+0000 7ff2bd800640 1 -- 172.21.15.90:0/795220241 shutdown_connections 2024-04-25T01:18:51.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.743+0000 7ff2bd800640 1 -- 172.21.15.90:0/795220241 wait complete. 2024-04-25T01:18:51.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.744+0000 7ff2bd800640 1 Processor -- start 2024-04-25T01:18:51.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.744+0000 7ff2bd800640 1 -- start start 2024-04-25T01:18:51.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.745+0000 7ff2bd800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2b80ae830 0x7ff2b8153f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:51.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.745+0000 7ff2bd800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2b815bdc0 0x7ff2b8154460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:51.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.745+0000 7ff2bd800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2b815cd80 0x7ff2b8154c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:51.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.745+0000 7ff2bd800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff2b8070b00 con 0x7ff2b80ae830 2024-04-25T01:18:51.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.745+0000 7ff2bd800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff2b8071e00 con 0x7ff2b815cd80 2024-04-25T01:18:51.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.745+0000 7ff2bd800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff2b8070de0 con 0x7ff2b815bdc0 2024-04-25T01:18:51.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.746+0000 7ff2b7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2b80ae830 0x7ff2b8153f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:51.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.746+0000 7ff2b7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2b80ae830 0x7ff2b8153f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47432/0 (socket says 172.21.15.90:47432) 2024-04-25T01:18:51.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.746+0000 7ff2b7e00640 1 -- 172.21.15.90:0/3471126161 learned_addr learned my addr 172.21.15.90:0/3471126161 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:51.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.746+0000 7ff2bce00640 1 --2- 172.21.15.90:0/3471126161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2b815cd80 0x7ff2b8154c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:51.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.746+0000 7ff2b7e00640 1 -- 172.21.15.90:0/3471126161 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2b815bdc0 msgr2=0x7ff2b8154460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:51.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.746+0000 7ff2b7e00640 1 --2- 172.21.15.90:0/3471126161 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2b815bdc0 0x7ff2b8154460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:51.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.746+0000 7ff2b7e00640 1 -- 172.21.15.90:0/3471126161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2b815cd80 msgr2=0x7ff2b8154c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:51.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.746+0000 7ff2b7e00640 1 --2- 172.21.15.90:0/3471126161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2b815cd80 0x7ff2b8154c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:51.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.746+0000 7ff2b7e00640 1 -- 172.21.15.90:0/3471126161 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2b81f4000 con 0x7ff2b80ae830 2024-04-25T01:18:51.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.747+0000 7ff2b7e00640 1 --2- 172.21.15.90:0/3471126161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2b80ae830 0x7ff2b8153f20 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7ff2a802a230 tx=0x7ff2a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:51.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.747+0000 7ff2afe00640 1 -- 172.21.15.90:0/3471126161 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a8037040 con 0x7ff2b80ae830 2024-04-25T01:18:51.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.747+0000 7ff2afe00640 1 -- 172.21.15.90:0/3471126161 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2a8042070 con 0x7ff2b80ae830 2024-04-25T01:18:51.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.747+0000 7ff2afe00640 1 -- 172.21.15.90:0/3471126161 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a8030bb0 con 0x7ff2b80ae830 2024-04-25T01:18:51.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.747+0000 7ff2bd800640 1 -- 172.21.15.90:0/3471126161 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2b81f42e0 con 0x7ff2b80ae830 2024-04-25T01:18:51.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.747+0000 7ff2bd800640 1 -- 172.21.15.90:0/3471126161 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2b81f3cf0 con 0x7ff2b80ae830 2024-04-25T01:18:51.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.748+0000 7ff2bd800640 1 -- 172.21.15.90:0/3471126161 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2b804f500 con 0x7ff2b80ae830 2024-04-25T01:18:51.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.749+0000 7ff2afe00640 1 -- 172.21.15.90:0/3471126161 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff2a8047460 con 0x7ff2b80ae830 2024-04-25T01:18:51.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.750+0000 7ff2afe00640 1 --2- 172.21.15.90:0/3471126161 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff29806be30 0x7ff29806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:51.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.750+0000 7ff2afe00640 1 -- 172.21.15.90:0/3471126161 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6260+0+0 (secure 0 0 0) 0x7ff2a80abdc0 con 0x7ff2b80ae830 2024-04-25T01:18:51.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.750+0000 7ff2b7400640 1 --2- 172.21.15.90:0/3471126161 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff29806be30 0x7ff29806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:51.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.751+0000 7ff2b7400640 1 --2- 172.21.15.90:0/3471126161 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff29806be30 0x7ff29806e2f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff2b81598c0 tx=0x7ff2a4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:51.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.754+0000 7ff2afe00640 1 -- 172.21.15.90:0/3471126161 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff2a8045360 con 0x7ff2b80ae830 2024-04-25T01:18:51.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.995+0000 7ff2bd800640 1 -- 172.21.15.90:0/3471126161 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "recovery_fs"} v 0) -- 0x7ff2b806e570 con 0x7ff2b80ae830 2024-04-25T01:18:51.978 INFO:tasks.ceph.mon.a.smithi090.stderr:2024-04-25T01:18:51.995+0000 7f0aa5400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T01:18:51.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:51.999+0000 7ff2afe00640 1 -- 172.21.15.90:0/3471126161 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "recovery_fs"}]=0 recovery_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v31) ==== 171+0+0 (secure 0 0 0) 0x7ff2a807a4a0 con 0x7ff2b80ae830 2024-04-25T01:18:51.982 INFO:teuthology.orchestra.run.smithi090.stderr:recovery_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T01:18:51.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.002+0000 7ff2bd800640 1 -- 172.21.15.90:0/3471126161 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff29806be30 msgr2=0x7ff29806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:51.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.002+0000 7ff2bd800640 1 --2- 172.21.15.90:0/3471126161 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff29806be30 0x7ff29806e2f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff2b81598c0 tx=0x7ff2a4008040 comp rx=0 tx=0).stop 2024-04-25T01:18:51.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.002+0000 7ff2bd800640 1 -- 172.21.15.90:0/3471126161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2b80ae830 msgr2=0x7ff2b8153f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:51.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.002+0000 7ff2bd800640 1 --2- 172.21.15.90:0/3471126161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2b80ae830 0x7ff2b8153f20 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7ff2a802a230 tx=0x7ff2a80046c0 comp rx=0 tx=0).stop 2024-04-25T01:18:51.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.002+0000 7ff2bd800640 1 -- 172.21.15.90:0/3471126161 shutdown_connections 2024-04-25T01:18:51.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.002+0000 7ff2bd800640 1 --2- 172.21.15.90:0/3471126161 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff29806be30 0x7ff29806e2f0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:51.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.002+0000 7ff2bd800640 1 --2- 172.21.15.90:0/3471126161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2b815cd80 0x7ff2b8154c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:51.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.002+0000 7ff2bd800640 1 --2- 172.21.15.90:0/3471126161 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2b815bdc0 0x7ff2b8154460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:51.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.002+0000 7ff2bd800640 1 --2- 172.21.15.90:0/3471126161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2b80ae830 0x7ff2b8153f20 secure :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7ff2a802a230 tx=0x7ff2a80046c0 comp rx=0 tx=0).stop 2024-04-25T01:18:51.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.002+0000 7ff2bd800640 1 -- 172.21.15.90:0/3471126161 >> 172.21.15.90:0/3471126161 conn(0x7ff2b80b1560 msgr2=0x7ff2b80ad8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:51.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.003+0000 7ff2bd800640 1 -- 172.21.15.90:0/3471126161 shutdown_connections 2024-04-25T01:18:51.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.004+0000 7ff2bd800640 1 -- 172.21.15.90:0/3471126161 wait complete. 2024-04-25T01:18:52.006 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm recovery_fs --yes-i-really-mean-it 2024-04-25T01:18:52.008 INFO:tasks.ceph.mds.d.smithi103.stderr: -5279> 2024-04-25T01:18:27.512+0000 7f4f9e600640 -1 mds.0.openfiles _load_finish got (2) No such file or directory 2024-04-25T01:18:52.018 INFO:tasks.ceph.mds.d.smithi103.stderr: -3818> 2024-04-25T01:18:31.626+0000 7f4fa3600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T01:18:52.019 INFO:tasks.ceph.mds.d.smithi103.stderr: -3604> 2024-04-25T01:18:33.186+0000 7f4f9e600640 -1 log_channel(cluster) log [ERR] : bad backtrace on directory inode 0x10000000000 2024-04-25T01:18:52.020 INFO:tasks.ceph.mds.d.smithi103.stderr: -3431> 2024-04-25T01:18:33.189+0000 7f4f9e600640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000000 [...2,head] /subdir/ auth v1 pv4 ap=3 DIRTYPARENT f(v0 1=1+0) n() (inest lock dirty) (ifile lock->sync w=1 flushing) | dirtyscattered=2 lock=1 dirfrag=1 dirtyparent=1 waiter=1 authpin=1 scrubqueue=0 0x55d17e6b5080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(7)0x10000000000:[<0x1/subdir v1>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 1=1+0)","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 1=1+0)","memory_value.rstat":"n()","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-61} 2024-04-25T01:18:52.021 INFO:tasks.ceph.mds.d.smithi103.stderr: -3377> 2024-04-25T01:18:33.190+0000 7f4f9e600640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /subdir/sixmegs auth v1 ap=1 DIRTYPARENT s=6291456 n() | dirtyparent=1 authpin=1 scrubqueue=0 0x55d17e7ce580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(6)0x10000000001:[<0x10000000000/sixmegs v10>,<0x1/subdir v12>]//[]","memoryvalue":"(6)0x10000000001:[<0x10000000000/sixmegs v1>,<0x1/subdir v1>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-25T01:18:52.024 INFO:tasks.ceph.mds.d.smithi103.stderr: -3003> 2024-04-25T01:18:33.203+0000 7f4f9e600640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x1 [...2,head] / auth v4 pv6 ap=5 snaprealm=0x55d17e6c6900 DIRTYPARENT f(v1 1=0+1) n(v0 1=0+1)/n() (inest mix->lock(2) w=1 flushing) (ifile mix->sync w=1 flushing) | dirtyscattered=2 lock=2 dirfrag=1 openingsnapparents=0 dirtyparent=1 dirty=1 waiter=1 authpin=1 scrubqueue=0 0x55d17e6b6680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(7)0x1:[]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 1=0+1)","ondisk_value.rstat":"n(v0 2=0+2)","memory_value.dirstat":"f(v1 1=0+1)","memory_value.rstat":"n(v0 1=0+1)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-61} 2024-04-25T01:18:52.109 INFO:tasks.ceph.mds.d.smithi103.stdout:starting mds.d at 2024-04-25T01:18:52.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.208+0000 7f30c4e00640 1 Processor -- start 2024-04-25T01:18:52.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.208+0000 7f30c4e00640 1 -- start start 2024-04-25T01:18:52.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.210+0000 7f30c4e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f30c015dbf0 0x7f30c006d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:52.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.210+0000 7f30c4e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f30c015dfd0 0x7f30c00cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:52.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.210+0000 7f30c4e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30c00719e0 con 0x7f30c00d0020 2024-04-25T01:18:52.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.210+0000 7f30c4e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f30c006dd90 con 0x7f30c015dfd0 2024-04-25T01:18:52.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.210+0000 7f30c4e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f30c00d0470 con 0x7f30c015dbf0 2024-04-25T01:18:52.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.210+0000 7f30bfe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f30c00d0020 0x7f30c0070c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52888/0 (socket says 172.21.15.90:52888) 2024-04-25T01:18:52.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.210+0000 7f30bfe00640 1 -- 172.21.15.90:0/2792379053 learned_addr learned my addr 172.21.15.90:0/2792379053 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:52.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.210+0000 7f30bea00640 1 --2- 172.21.15.90:0/2792379053 >> v2:172.21.15.90:3300/0 conn(0x7f30c015dfd0 0x7f30c00cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:52.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.211+0000 7f30bf400640 1 --2- 172.21.15.90:0/2792379053 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f30c015dbf0 0x7f30c006d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:52.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.211+0000 7f30bea00640 1 -- 172.21.15.90:0/2792379053 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f30c015dbf0 msgr2=0x7f30c006d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:52.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.211+0000 7f30bea00640 1 --2- 172.21.15.90:0/2792379053 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f30c015dbf0 0x7f30c006d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:52.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.211+0000 7f30be000640 1 -- 172.21.15.90:0/2792379053 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1033799151 0 0) 0x7f30c00719e0 con 0x7f30c00d0020 2024-04-25T01:18:52.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.211+0000 7f30bea00640 1 -- 172.21.15.90:0/2792379053 >> v1:172.21.15.90:6789/0 conn(0x7f30c00d0020 legacy=0x7f30c0070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:52.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.211+0000 7f30bea00640 1 -- 172.21.15.90:0/2792379053 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30c006cd70 con 0x7f30c015dfd0 2024-04-25T01:18:52.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.212+0000 7f30bea00640 1 --2- 172.21.15.90:0/2792379053 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30c015dfd0 0x7f30c00cfae0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f30b00097a0 tx=0x7f30b002a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=391d55f28576ad7b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:52.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.212+0000 7f30be000640 1 -- 172.21.15.90:0/2792379053 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b0037070 con 0x7f30c015dfd0 2024-04-25T01:18:52.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.212+0000 7f30be000640 1 -- 172.21.15.90:0/2792379053 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30b002ae50 con 0x7f30c015dfd0 2024-04-25T01:18:52.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.212+0000 7f30be000640 1 -- 172.21.15.90:0/2792379053 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b0032660 con 0x7f30c015dfd0 2024-04-25T01:18:52.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.213+0000 7f30c4e00640 1 -- 172.21.15.90:0/2792379053 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30c015dfd0 msgr2=0x7f30c00cfae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:52.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.213+0000 7f30c4e00640 1 --2- 172.21.15.90:0/2792379053 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30c015dfd0 0x7f30c00cfae0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f30b00097a0 tx=0x7f30b002a100 comp rx=0 tx=0).stop 2024-04-25T01:18:52.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.213+0000 7f30c4e00640 1 -- 172.21.15.90:0/2792379053 shutdown_connections 2024-04-25T01:18:52.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.213+0000 7f30c4e00640 1 --2- 172.21.15.90:0/2792379053 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30c015dfd0 0x7f30c00cfae0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:52.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.213+0000 7f30c4e00640 1 --2- 172.21.15.90:0/2792379053 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f30c015dbf0 0x7f30c006d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:52.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.213+0000 7f30c4e00640 1 -- 172.21.15.90:0/2792379053 >> 172.21.15.90:0/2792379053 conn(0x7f30c006b350 msgr2=0x7f30c006b760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:52.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.213+0000 7f30c4e00640 1 -- 172.21.15.90:0/2792379053 shutdown_connections 2024-04-25T01:18:52.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.213+0000 7f30c4e00640 1 -- 172.21.15.90:0/2792379053 wait complete. 2024-04-25T01:18:52.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.215+0000 7f30c4e00640 1 Processor -- start 2024-04-25T01:18:52.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.215+0000 7f30c4e00640 1 -- start start 2024-04-25T01:18:52.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.216+0000 7f30c4e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30c00d0020 0x7f30c0149660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:52.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.216+0000 7f30c4e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f30c015dbf0 0x7f30c01f3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:52.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.216+0000 7f30c4e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30c015dfd0 0x7f30c00ad8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:52.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.216+0000 7f30c4e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f30c006d330 con 0x7f30c015dfd0 2024-04-25T01:18:52.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.216+0000 7f30c4e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f30c006cb20 con 0x7f30c00d0020 2024-04-25T01:18:52.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.216+0000 7f30c4e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f30c0071380 con 0x7f30c015dbf0 2024-04-25T01:18:52.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.216+0000 7f30bfe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30c015dfd0 0x7f30c00ad8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:52.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.216+0000 7f30bfe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30c015dfd0 0x7f30c00ad8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47444/0 (socket says 172.21.15.90:47444) 2024-04-25T01:18:52.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.216+0000 7f30bfe00640 1 -- 172.21.15.90:0/3695965861 learned_addr learned my addr 172.21.15.90:0/3695965861 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:52.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.216+0000 7f30bea00640 1 --2- 172.21.15.90:0/3695965861 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f30c015dbf0 0x7f30c01f3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:52.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.216+0000 7f30bf400640 1 --2- 172.21.15.90:0/3695965861 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30c00d0020 0x7f30c0149660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:52.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.217+0000 7f30bfe00640 1 -- 172.21.15.90:0/3695965861 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f30c015dbf0 msgr2=0x7f30c01f3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:52.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.217+0000 7f30bfe00640 1 --2- 172.21.15.90:0/3695965861 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f30c015dbf0 0x7f30c01f3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:52.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.217+0000 7f30bfe00640 1 -- 172.21.15.90:0/3695965861 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30c00d0020 msgr2=0x7f30c0149660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:52.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.217+0000 7f30bfe00640 1 --2- 172.21.15.90:0/3695965861 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30c00d0020 0x7f30c0149660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:52.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.217+0000 7f30bfe00640 1 -- 172.21.15.90:0/3695965861 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30c01f4510 con 0x7f30c015dfd0 2024-04-25T01:18:52.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.217+0000 7f30bf400640 1 --2- 172.21.15.90:0/3695965861 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30c00d0020 0x7f30c0149660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:52.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.217+0000 7f30bfe00640 1 --2- 172.21.15.90:0/3695965861 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30c015dfd0 0x7f30c00ad8c0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f30b800c7a0 tx=0x7f30b800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:52.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.217+0000 7f30b7400640 1 -- 172.21.15.90:0/3695965861 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b8018070 con 0x7f30c015dfd0 2024-04-25T01:18:52.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.217+0000 7f30b7400640 1 -- 172.21.15.90:0/3695965861 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30b800a640 con 0x7f30c015dfd0 2024-04-25T01:18:52.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.217+0000 7f30c4e00640 1 -- 172.21.15.90:0/3695965861 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30c01f47f0 con 0x7f30c015dfd0 2024-04-25T01:18:52.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.217+0000 7f30b7400640 1 -- 172.21.15.90:0/3695965861 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b800aab0 con 0x7f30c015dfd0 2024-04-25T01:18:52.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.217+0000 7f30c4e00640 1 -- 172.21.15.90:0/3695965861 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30c00ccd60 con 0x7f30c015dfd0 2024-04-25T01:18:52.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.218+0000 7f30c4e00640 1 -- 172.21.15.90:0/3695965861 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30c004f500 con 0x7f30c015dfd0 2024-04-25T01:18:52.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.219+0000 7f30b7400640 1 -- 172.21.15.90:0/3695965861 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f30b801b070 con 0x7f30c015dfd0 2024-04-25T01:18:52.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.220+0000 7f30b7400640 1 --2- 172.21.15.90:0/3695965861 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f309c06be30 0x7f309c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:52.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.220+0000 7f30b7400640 1 -- 172.21.15.90:0/3695965861 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6346+0+0 (secure 0 0 0) 0x7f30b808a730 con 0x7f30c015dfd0 2024-04-25T01:18:52.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.220+0000 7f30bf400640 1 --2- 172.21.15.90:0/3695965861 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f309c06be30 0x7f309c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:52.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.221+0000 7f30bf400640 1 --2- 172.21.15.90:0/3695965861 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f309c06be30 0x7f309c06e2f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f30a8006fd0 tx=0x7f30a8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:52.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.224+0000 7f30b7400640 1 -- 172.21.15.90:0/3695965861 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f30b8058e50 con 0x7f30c015dfd0 2024-04-25T01:18:52.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:52.463+0000 7f30c4e00640 1 -- 172.21.15.90:0/3695965861 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "recovery_fs", "yes_i_really_mean_it": true} v 0) -- 0x7f30c00cee20 con 0x7f30c015dfd0 2024-04-25T01:18:53.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.059+0000 7f30b7400640 1 -- 172.21.15.90:0/3695965861 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "recovery_fs", "yes_i_really_mean_it": true}]=0 v32) ==== 109+0+0 (secure 0 0 0) 0x7f30b8058750 con 0x7f30c015dfd0 2024-04-25T01:18:53.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.063+0000 7f30c4e00640 1 -- 172.21.15.90:0/3695965861 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f309c06be30 msgr2=0x7f309c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:53.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.063+0000 7f30c4e00640 1 --2- 172.21.15.90:0/3695965861 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f309c06be30 0x7f309c06e2f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f30a8006fd0 tx=0x7f30a8008040 comp rx=0 tx=0).stop 2024-04-25T01:18:53.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.063+0000 7f30c4e00640 1 -- 172.21.15.90:0/3695965861 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30c015dfd0 msgr2=0x7f30c00ad8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:53.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.063+0000 7f30c4e00640 1 --2- 172.21.15.90:0/3695965861 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30c015dfd0 0x7f30c00ad8c0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f30b800c7a0 tx=0x7f30b800cc60 comp rx=0 tx=0).stop 2024-04-25T01:18:53.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.064+0000 7f30c4e00640 1 -- 172.21.15.90:0/3695965861 shutdown_connections 2024-04-25T01:18:53.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.064+0000 7f30c4e00640 1 --2- 172.21.15.90:0/3695965861 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f309c06be30 0x7f309c06e2f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:53.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.064+0000 7f30c4e00640 1 --2- 172.21.15.90:0/3695965861 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30c015dfd0 0x7f30c00ad8c0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:53.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.064+0000 7f30c4e00640 1 --2- 172.21.15.90:0/3695965861 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f30c015dbf0 0x7f30c01f3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:53.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.064+0000 7f30c4e00640 1 --2- 172.21.15.90:0/3695965861 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30c00d0020 0x7f30c0149660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:53.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.064+0000 7f30c4e00640 1 -- 172.21.15.90:0/3695965861 >> 172.21.15.90:0/3695965861 conn(0x7f30c006b350 msgr2=0x7f30c0158450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:53.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.064+0000 7f30c4e00640 1 -- 172.21.15.90:0/3695965861 shutdown_connections 2024-04-25T01:18:53.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.064+0000 7f30c4e00640 1 -- 172.21.15.90:0/3695965861 wait complete. 2024-04-25T01:18:53.068 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm recovery_fs_metadata recovery_fs_metadata --yes-i-really-really-mean-it 2024-04-25T01:18:53.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.291+0000 7f69bf200640 1 Processor -- start 2024-04-25T01:18:53.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.291+0000 7f69bf200640 1 -- start start 2024-04-25T01:18:53.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.293+0000 7f69bf200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f69b8162db0 0x7f69b80aeb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:53.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.293+0000 7f69bf200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69b806c8c0 con 0x7f69b8162780 2024-04-25T01:18:53.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.293+0000 7f69bf200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69b806fa50 con 0x7f69b81623a0 2024-04-25T01:18:53.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.293+0000 7f69bf200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f69b80702f0 con 0x7f69b8162db0 2024-04-25T01:18:53.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.294+0000 7f69bd200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f69b8162780 0x7f69b80d0c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52904/0 (socket says 172.21.15.90:52904) 2024-04-25T01:18:53.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.294+0000 7f69bd200640 1 -- 172.21.15.90:0/1053856413 learned_addr learned my addr 172.21.15.90:0/1053856413 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:53.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.294+0000 7f69b7400640 1 --2- 172.21.15.90:0/1053856413 >> v2:172.21.15.103:3300/0 conn(0x7f69b8162db0 0x7f69b80aeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:53.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.294+0000 7f69b6a00640 1 -- 172.21.15.90:0/1053856413 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2996999834 0 0) 0x7f69b0001110 con 0x7f69b8162780 2024-04-25T01:18:53.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.294+0000 7f69b6a00640 1 -- 172.21.15.90:0/1053856413 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f699c003520 con 0x7f69b8162780 2024-04-25T01:18:53.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.294+0000 7f69b7400640 1 -- 172.21.15.90:0/1053856413 >> v1:172.21.15.103:6789/0 conn(0x7f69b81623a0 legacy=0x7f69b815def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:53.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.294+0000 7f69b7400640 1 -- 172.21.15.90:0/1053856413 >> v1:172.21.15.90:6789/0 conn(0x7f69b8162780 legacy=0x7f69b80d0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:53.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.294+0000 7f69b6a00640 1 -- 172.21.15.90:0/1053856413 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2499470416 0 0) 0x7f699c003520 con 0x7f69b8162780 2024-04-25T01:18:53.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.295+0000 7f69b7400640 1 -- 172.21.15.90:0/1053856413 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69b806fd30 con 0x7f69b8162db0 2024-04-25T01:18:53.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.296+0000 7f69b7400640 1 --2- 172.21.15.90:0/1053856413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69b8162db0 0x7f69b80aeb40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f69a00097a0 tx=0x7f69a002a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ce560c13ad368ea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:53.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.296+0000 7f69b6a00640 1 -- 172.21.15.90:0/1053856413 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69a0037070 con 0x7f69b8162db0 2024-04-25T01:18:53.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.296+0000 7f69b6a00640 1 -- 172.21.15.90:0/1053856413 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69a002ac50 con 0x7f69b8162db0 2024-04-25T01:18:53.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.296+0000 7f69b6a00640 1 -- 172.21.15.90:0/1053856413 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b0001840 con 0x7f69b8162db0 2024-04-25T01:18:53.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.297+0000 7f69bf200640 1 -- 172.21.15.90:0/1053856413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69b8162db0 msgr2=0x7f69b80aeb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:53.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.297+0000 7f69bf200640 1 --2- 172.21.15.90:0/1053856413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69b8162db0 0x7f69b80aeb40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f69a00097a0 tx=0x7f69a002a1c0 comp rx=0 tx=0).stop 2024-04-25T01:18:53.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.297+0000 7f69bf200640 1 -- 172.21.15.90:0/1053856413 shutdown_connections 2024-04-25T01:18:53.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.297+0000 7f69bf200640 1 --2- 172.21.15.90:0/1053856413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69b8162db0 0x7f69b80aeb40 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:53.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.297+0000 7f69bf200640 1 -- 172.21.15.90:0/1053856413 >> 172.21.15.90:0/1053856413 conn(0x7f69b80b3860 msgr2=0x7f69b80b3c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:53.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.297+0000 7f69bf200640 1 -- 172.21.15.90:0/1053856413 shutdown_connections 2024-04-25T01:18:53.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.297+0000 7f69bf200640 1 -- 172.21.15.90:0/1053856413 wait complete. 2024-04-25T01:18:53.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.300+0000 7f69bf200640 1 Processor -- start 2024-04-25T01:18:53.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.300+0000 7f69bf200640 1 -- start start 2024-04-25T01:18:53.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.301+0000 7f69bf200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69b81623a0 0x7f69b80b67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:53.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.301+0000 7f69bf200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f69b8162780 0x7f69b80c7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:53.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.301+0000 7f69bf200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69b8162db0 0x7f69b80c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:53.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.301+0000 7f69bf200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f69b815e600 con 0x7f69b8162db0 2024-04-25T01:18:53.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.301+0000 7f69bf200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f69b806cb40 con 0x7f69b81623a0 2024-04-25T01:18:53.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.301+0000 7f69bf200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f69b815e780 con 0x7f69b8162780 2024-04-25T01:18:53.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.301+0000 7f69bd200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69b8162db0 0x7f69b80c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:53.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.301+0000 7f69bd200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69b8162db0 0x7f69b80c7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47448/0 (socket says 172.21.15.90:47448) 2024-04-25T01:18:53.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.302+0000 7f69bd200640 1 -- 172.21.15.90:0/2631421323 learned_addr learned my addr 172.21.15.90:0/2631421323 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:53.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.302+0000 7f69b7e00640 1 --2- 172.21.15.90:0/2631421323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69b81623a0 0x7f69b80b67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:53.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.302+0000 7f69b7400640 1 --2- 172.21.15.90:0/2631421323 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f69b8162780 0x7f69b80c7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:53.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.302+0000 7f69bd200640 1 -- 172.21.15.90:0/2631421323 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f69b8162780 msgr2=0x7f69b80c7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:53.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.302+0000 7f69bd200640 1 --2- 172.21.15.90:0/2631421323 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f69b8162780 0x7f69b80c7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:53.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.302+0000 7f69bd200640 1 -- 172.21.15.90:0/2631421323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69b81623a0 msgr2=0x7f69b80b67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:53.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.302+0000 7f69bd200640 1 --2- 172.21.15.90:0/2631421323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69b81623a0 0x7f69b80b67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:53.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.302+0000 7f69bd200640 1 -- 172.21.15.90:0/2631421323 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69b81f4da0 con 0x7f69b8162db0 2024-04-25T01:18:53.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.303+0000 7f69bd200640 1 --2- 172.21.15.90:0/2631421323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69b8162db0 0x7f69b80c7f70 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f69b000c750 tx=0x7f69b000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:53.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.303+0000 7f69afe00640 1 -- 172.21.15.90:0/2631421323 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b0017070 con 0x7f69b8162db0 2024-04-25T01:18:53.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.303+0000 7f69afe00640 1 -- 172.21.15.90:0/2631421323 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69b0005ad0 con 0x7f69b8162db0 2024-04-25T01:18:53.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.303+0000 7f69bf200640 1 -- 172.21.15.90:0/2631421323 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69b80b47a0 con 0x7f69b8162db0 2024-04-25T01:18:53.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.303+0000 7f69afe00640 1 -- 172.21.15.90:0/2631421323 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b0007070 con 0x7f69b8162db0 2024-04-25T01:18:53.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.303+0000 7f69bf200640 1 -- 172.21.15.90:0/2631421323 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69b80b4d60 con 0x7f69b8162db0 2024-04-25T01:18:53.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.304+0000 7f69bf200640 1 -- 172.21.15.90:0/2631421323 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69b80688a0 con 0x7f69b8162db0 2024-04-25T01:18:53.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.306+0000 7f69afe00640 1 -- 172.21.15.90:0/2631421323 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f69b001a070 con 0x7f69b8162db0 2024-04-25T01:18:53.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.306+0000 7f69afe00640 1 --2- 172.21.15.90:0/2631421323 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f699406be60 0x7f699406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:53.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.307+0000 7f69afe00640 1 -- 172.21.15.90:0/2631421323 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6346+0+0 (secure 0 0 0) 0x7f69b008a560 con 0x7f69b8162db0 2024-04-25T01:18:53.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.307+0000 7f69b7e00640 1 --2- 172.21.15.90:0/2631421323 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f699406be60 0x7f699406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:53.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.308+0000 7f69b7e00640 1 --2- 172.21.15.90:0/2631421323 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f699406be60 0x7f699406e320 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f69a80012e0 tx=0x7f69a8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:53.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.311+0000 7f69afe00640 1 -- 172.21.15.90:0/2631421323 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f69b806c8c0 con 0x7f69b8162db0 2024-04-25T01:18:53.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:53.508+0000 7f69bf200640 1 -- 172.21.15.90:0/2631421323 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "recovery_fs_metadata", "pool2": "recovery_fs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f69b80b7cb0 con 0x7f69b8162db0 2024-04-25T01:18:54.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.069+0000 7f69afe00640 1 -- 172.21.15.90:0/2631421323 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "recovery_fs_metadata", "pool2": "recovery_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'recovery_fs_metadata' removed v53) ==== 196+0+0 (secure 0 0 0) 0x7f69b0058630 con 0x7f69b8162db0 2024-04-25T01:18:54.053 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'recovery_fs_metadata' removed 2024-04-25T01:18:54.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.073+0000 7f69bf200640 1 -- 172.21.15.90:0/2631421323 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f699406be60 msgr2=0x7f699406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:54.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.073+0000 7f69bf200640 1 --2- 172.21.15.90:0/2631421323 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f699406be60 0x7f699406e320 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f69a80012e0 tx=0x7f69a8009040 comp rx=0 tx=0).stop 2024-04-25T01:18:54.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.073+0000 7f69bf200640 1 -- 172.21.15.90:0/2631421323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69b8162db0 msgr2=0x7f69b80c7f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:54.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.073+0000 7f69bf200640 1 --2- 172.21.15.90:0/2631421323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69b8162db0 0x7f69b80c7f70 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f69b000c750 tx=0x7f69b000cc10 comp rx=0 tx=0).stop 2024-04-25T01:18:54.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.074+0000 7f69bf200640 1 -- 172.21.15.90:0/2631421323 shutdown_connections 2024-04-25T01:18:54.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.074+0000 7f69bf200640 1 --2- 172.21.15.90:0/2631421323 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f699406be60 0x7f699406e320 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:54.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.074+0000 7f69bf200640 1 --2- 172.21.15.90:0/2631421323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69b8162db0 0x7f69b80c7f70 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:54.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.074+0000 7f69bf200640 1 --2- 172.21.15.90:0/2631421323 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f69b8162780 0x7f69b80c7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:54.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.074+0000 7f69bf200640 1 --2- 172.21.15.90:0/2631421323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69b81623a0 0x7f69b80b67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:54.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.074+0000 7f69bf200640 1 -- 172.21.15.90:0/2631421323 >> 172.21.15.90:0/2631421323 conn(0x7f69b80b3860 msgr2=0x7f69b80b3c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:54.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.074+0000 7f69bf200640 1 -- 172.21.15.90:0/2631421323 shutdown_connections 2024-04-25T01:18:54.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.074+0000 7f69bf200640 1 -- 172.21.15.90:0/2631421323 wait complete. 2024-04-25T01:18:54.079 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T01:18:54.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.295+0000 7f814e800640 1 Processor -- start 2024-04-25T01:18:54.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.296+0000 7f814e800640 1 -- start start 2024-04-25T01:18:54.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.297+0000 7f814e800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8148145510 0x7f81480728d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:54.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.297+0000 7f814e800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f814815f9f0 0x7f814815fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:54.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.297+0000 7f814e800640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81480cf040 con 0x7f81480ceaf0 2024-04-25T01:18:54.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.297+0000 7f814e800640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f81481603a0 con 0x7f814815f9f0 2024-04-25T01:18:54.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.297+0000 7f814e800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f81480cf320 con 0x7f8148145510 2024-04-25T01:18:54.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.298+0000 7f8147400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8148145510 0x7f81480728d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:54.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.298+0000 7f8147e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f814815f9f0 0x7f814815fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:54.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.298+0000 7f813ea00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f81480ceaf0 0x7f814806cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.90:38068/0 (socket says 172.21.15.90:38068) 2024-04-25T01:18:54.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.298+0000 7f813ea00640 1 -- 172.21.15.90:0/420636983 learned_addr learned my addr 172.21.15.90:0/420636983 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:54.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.298+0000 7f8147e00640 1 --2- 172.21.15.90:0/420636983 >> v2:172.21.15.103:3300/0 conn(0x7f814815f9f0 0x7f814815fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.90:45528/0 (socket says 172.21.15.90:45528) 2024-04-25T01:18:54.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.299+0000 7f8147e00640 1 -- 172.21.15.90:0/420636983 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8148145510 msgr2=0x7f81480728d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:54.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.299+0000 7f8147e00640 1 --2- 172.21.15.90:0/420636983 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8148145510 0x7f81480728d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:54.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.299+0000 7f8147e00640 1 -- 172.21.15.90:0/420636983 >> v1:172.21.15.103:6789/0 conn(0x7f81480ceaf0 legacy=0x7f814806cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:54.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.299+0000 7f8147e00640 1 -- 172.21.15.90:0/420636983 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8148070c70 con 0x7f814815f9f0 2024-04-25T01:18:54.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.301+0000 7f8147e00640 1 --2- 172.21.15.90:0/420636983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f814815f9f0 0x7f814815fe60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8140009d90 tx=0x7f814002a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca28b6a5453e9a8c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:54.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.302+0000 7f8146a00640 1 -- 172.21.15.90:0/420636983 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8140031600 con 0x7f814815f9f0 2024-04-25T01:18:54.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.302+0000 7f8146a00640 1 -- 172.21.15.90:0/420636983 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81400317a0 con 0x7f814815f9f0 2024-04-25T01:18:54.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.302+0000 7f8146a00640 1 -- 172.21.15.90:0/420636983 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8140031b70 con 0x7f814815f9f0 2024-04-25T01:18:54.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.302+0000 7f814e800640 1 -- 172.21.15.90:0/420636983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f814815f9f0 msgr2=0x7f814815fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:54.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.302+0000 7f814e800640 1 --2- 172.21.15.90:0/420636983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f814815f9f0 0x7f814815fe60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8140009d90 tx=0x7f814002a5d0 comp rx=0 tx=0).stop 2024-04-25T01:18:54.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.302+0000 7f814e800640 1 -- 172.21.15.90:0/420636983 shutdown_connections 2024-04-25T01:18:54.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.302+0000 7f814e800640 1 --2- 172.21.15.90:0/420636983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f814815f9f0 0x7f814815fe60 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:54.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.302+0000 7f814e800640 1 --2- 172.21.15.90:0/420636983 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f8148145510 0x7f81480728d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:54.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.302+0000 7f814e800640 1 -- 172.21.15.90:0/420636983 >> 172.21.15.90:0/420636983 conn(0x7f81480b16b0 msgr2=0x7f81480b1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:54.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.302+0000 7f814e800640 1 -- 172.21.15.90:0/420636983 shutdown_connections 2024-04-25T01:18:54.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.302+0000 7f814e800640 1 -- 172.21.15.90:0/420636983 wait complete. 2024-04-25T01:18:54.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.304+0000 7f814e800640 1 Processor -- start 2024-04-25T01:18:54.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.304+0000 7f814e800640 1 -- start start 2024-04-25T01:18:54.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.305+0000 7f814e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81480ceaf0 0x7f814813d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:54.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.305+0000 7f814e800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8148145510 0x7f81481436f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:54.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.305+0000 7f814e800640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f814815f9f0 0x7f8148143c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:54.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.305+0000 7f814e800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f814806d290 con 0x7f81480ceaf0 2024-04-25T01:18:54.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.305+0000 7f814e800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8148071230 con 0x7f8148145510 2024-04-25T01:18:54.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.305+0000 7f814e800640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f8148072e10 con 0x7f814815f9f0 2024-04-25T01:18:54.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.306+0000 7f8147400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81480ceaf0 0x7f814813d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:54.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.306+0000 7f8147400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81480ceaf0 0x7f814813d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47460/0 (socket says 172.21.15.90:47460) 2024-04-25T01:18:54.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.306+0000 7f8147400640 1 -- 172.21.15.90:0/1096952886 learned_addr learned my addr 172.21.15.90:0/1096952886 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:54.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.306+0000 7f813ea00640 1 --2- 172.21.15.90:0/1096952886 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8148145510 0x7f81481436f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:54.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.306+0000 7f8147e00640 1 --2- 172.21.15.90:0/1096952886 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f814815f9f0 0x7f8148143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:54.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.306+0000 7f8147400640 1 -- 172.21.15.90:0/1096952886 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f814815f9f0 msgr2=0x7f8148143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:54.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.306+0000 7f8147400640 1 --2- 172.21.15.90:0/1096952886 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f814815f9f0 0x7f8148143c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:54.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.306+0000 7f8147400640 1 -- 172.21.15.90:0/1096952886 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8148145510 msgr2=0x7f81481436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:54.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.306+0000 7f8147400640 1 --2- 172.21.15.90:0/1096952886 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8148145510 0x7f81481436f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:54.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.306+0000 7f8147400640 1 -- 172.21.15.90:0/1096952886 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81481f7be0 con 0x7f81480ceaf0 2024-04-25T01:18:54.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.307+0000 7f8147400640 1 --2- 172.21.15.90:0/1096952886 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81480ceaf0 0x7f814813d4d0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f813800e9b0 tx=0x7f813800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:54.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.307+0000 7f813fe00640 1 -- 172.21.15.90:0/1096952886 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8138017070 con 0x7f81480ceaf0 2024-04-25T01:18:54.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.307+0000 7f813fe00640 1 -- 172.21.15.90:0/1096952886 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f813800c9a0 con 0x7f81480ceaf0 2024-04-25T01:18:54.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.307+0000 7f814e800640 1 -- 172.21.15.90:0/1096952886 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81481f7ec0 con 0x7f81480ceaf0 2024-04-25T01:18:54.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.307+0000 7f813fe00640 1 -- 172.21.15.90:0/1096952886 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813800cd90 con 0x7f81480ceaf0 2024-04-25T01:18:54.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.307+0000 7f814e800640 1 -- 172.21.15.90:0/1096952886 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f814813da10 con 0x7f81480ceaf0 2024-04-25T01:18:54.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.308+0000 7f814e800640 1 -- 172.21.15.90:0/1096952886 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81480cdbc0 con 0x7f81480ceaf0 2024-04-25T01:18:54.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.309+0000 7f813fe00640 1 -- 172.21.15.90:0/1096952886 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f813801a070 con 0x7f81480ceaf0 2024-04-25T01:18:54.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.310+0000 7f813fe00640 1 --2- 172.21.15.90:0/1096952886 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f812406be30 0x7f812406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:54.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.310+0000 7f813fe00640 1 -- 172.21.15.90:0/1096952886 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5898+0+0 (secure 0 0 0) 0x7f813808b2b0 con 0x7f81480ceaf0 2024-04-25T01:18:54.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.310+0000 7f813ea00640 1 --2- 172.21.15.90:0/1096952886 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f812406be30 0x7f812406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:54.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.311+0000 7f813ea00640 1 --2- 172.21.15.90:0/1096952886 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f812406be30 0x7f812406e2f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f81480b4870 tx=0x7f8130009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:54.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.315+0000 7f813fe00640 1 -- 172.21.15.90:0/1096952886 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f81480cdbc0 con 0x7f81480ceaf0 2024-04-25T01:18:54.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:54.512+0000 7f814e800640 1 -- 172.21.15.90:0/1096952886 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8148068190 con 0x7f81480ceaf0 2024-04-25T01:18:55.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.073+0000 7f813fe00640 1 -- 172.21.15.90:0/1096952886 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v54) ==== 169+0+0 (secure 0 0 0) 0x7f8138059b90 con 0x7f81480ceaf0 2024-04-25T01:18:55.056 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data' removed 2024-04-25T01:18:55.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.077+0000 7f814e800640 1 -- 172.21.15.90:0/1096952886 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f812406be30 msgr2=0x7f812406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:55.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.077+0000 7f814e800640 1 --2- 172.21.15.90:0/1096952886 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f812406be30 0x7f812406e2f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f81480b4870 tx=0x7f8130009040 comp rx=0 tx=0).stop 2024-04-25T01:18:55.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.077+0000 7f814e800640 1 -- 172.21.15.90:0/1096952886 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81480ceaf0 msgr2=0x7f814813d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:55.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.077+0000 7f814e800640 1 --2- 172.21.15.90:0/1096952886 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81480ceaf0 0x7f814813d4d0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f813800e9b0 tx=0x7f813800ee70 comp rx=0 tx=0).stop 2024-04-25T01:18:55.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.078+0000 7f814e800640 1 -- 172.21.15.90:0/1096952886 shutdown_connections 2024-04-25T01:18:55.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.078+0000 7f814e800640 1 --2- 172.21.15.90:0/1096952886 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f812406be30 0x7f812406e2f0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:55.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.078+0000 7f814e800640 1 --2- 172.21.15.90:0/1096952886 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f814815f9f0 0x7f8148143c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:55.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.078+0000 7f814e800640 1 --2- 172.21.15.90:0/1096952886 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8148145510 0x7f81481436f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:55.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.078+0000 7f814e800640 1 --2- 172.21.15.90:0/1096952886 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81480ceaf0 0x7f814813d4d0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:55.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.078+0000 7f814e800640 1 -- 172.21.15.90:0/1096952886 >> 172.21.15.90:0/1096952886 conn(0x7f81480b16b0 msgr2=0x7f81480b1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:55.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.078+0000 7f814e800640 1 -- 172.21.15.90:0/1096952886 shutdown_connections 2024-04-25T01:18:55.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.078+0000 7f814e800640 1 -- 172.21.15.90:0/1096952886 wait complete. 2024-04-25T01:18:55.083 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-25T01:18:55.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.306+0000 7f01eb200640 1 Processor -- start 2024-04-25T01:18:55.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.306+0000 7f01eb200640 1 -- start start 2024-04-25T01:18:55.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.307+0000 7f01eb200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01ec14ba80 0x7f01ec0b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:55.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.307+0000 7f01eb200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f01ec0d01f0 0x7f01ec0cc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:55.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.307+0000 7f01eb200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01ec070f70 con 0x7f01ec0709d0 2024-04-25T01:18:55.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.307+0000 7f01eb200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f01ec071250 con 0x7f01ec0d01f0 2024-04-25T01:18:55.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.307+0000 7f01eb200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f01ec071530 con 0x7f01ec14ba80 2024-04-25T01:18:55.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.308+0000 7f01ea800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f01ec0d01f0 0x7f01ec0cc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:55.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.308+0000 7f01ea800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f01ec0d01f0 0x7f01ec0cc2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45180/0 (socket says 172.21.15.90:45180) 2024-04-25T01:18:55.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.308+0000 7f01ea800640 1 -- 172.21.15.90:0/3412647741 learned_addr learned my addr 172.21.15.90:0/3412647741 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:55.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.308+0000 7f01e9e00640 1 --2- 172.21.15.90:0/3412647741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01ec14ba80 0x7f01ec0b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:55.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.308+0000 7f01ea800640 1 -- 172.21.15.90:0/3412647741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01ec14ba80 msgr2=0x7f01ec0b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:55.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.308+0000 7f01ea800640 1 --2- 172.21.15.90:0/3412647741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01ec14ba80 0x7f01ec0b46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:55.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.308+0000 7f01ea800640 1 -- 172.21.15.90:0/3412647741 >> v1:172.21.15.103:6789/0 conn(0x7f01ec0709d0 legacy=0x7f01ec0cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:55.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.308+0000 7f01ea800640 1 -- 172.21.15.90:0/3412647741 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01ec071810 con 0x7f01ec0d01f0 2024-04-25T01:18:55.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.308+0000 7f01e9e00640 1 --2- 172.21.15.90:0/3412647741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01ec14ba80 0x7f01ec0b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:55.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.309+0000 7f01ea800640 1 --2- 172.21.15.90:0/3412647741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01ec0d01f0 0x7f01ec0cc2b0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f01e4009e50 tx=0x7f01e402a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5e4e270940bbd60 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:55.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.310+0000 7f01e9400640 1 -- 172.21.15.90:0/3412647741 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e4036070 con 0x7f01ec0d01f0 2024-04-25T01:18:55.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.310+0000 7f01e9400640 1 -- 172.21.15.90:0/3412647741 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01e4031600 con 0x7f01ec0d01f0 2024-04-25T01:18:55.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.310+0000 7f01e9400640 1 -- 172.21.15.90:0/3412647741 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e40319d0 con 0x7f01ec0d01f0 2024-04-25T01:18:55.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.310+0000 7f01eb200640 1 -- 172.21.15.90:0/3412647741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01ec0d01f0 msgr2=0x7f01ec0cc2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:55.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.310+0000 7f01eb200640 1 --2- 172.21.15.90:0/3412647741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01ec0d01f0 0x7f01ec0cc2b0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f01e4009e50 tx=0x7f01e402a6c0 comp rx=0 tx=0).stop 2024-04-25T01:18:55.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.310+0000 7f01eb200640 1 -- 172.21.15.90:0/3412647741 shutdown_connections 2024-04-25T01:18:55.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.310+0000 7f01eb200640 1 --2- 172.21.15.90:0/3412647741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01ec0d01f0 0x7f01ec0cc2b0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:55.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.310+0000 7f01eb200640 1 --2- 172.21.15.90:0/3412647741 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01ec14ba80 0x7f01ec0b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:55.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.310+0000 7f01eb200640 1 -- 172.21.15.90:0/3412647741 >> 172.21.15.90:0/3412647741 conn(0x7f01ec06b350 msgr2=0x7f01ec06b760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:55.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.310+0000 7f01eb200640 1 -- 172.21.15.90:0/3412647741 shutdown_connections 2024-04-25T01:18:55.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.310+0000 7f01eb200640 1 -- 172.21.15.90:0/3412647741 wait complete. 2024-04-25T01:18:55.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.312+0000 7f01eb200640 1 Processor -- start 2024-04-25T01:18:55.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.312+0000 7f01eb200640 1 -- start start 2024-04-25T01:18:55.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.313+0000 7f01eb200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01ec0709d0 0x7f01ec161ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:55.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.313+0000 7f01eb200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01ec0d01f0 0x7f01ec0a94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:55.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.313+0000 7f01eb200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01ec14ba80 0x7f01ec1499e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:55.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.313+0000 7f01eb200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f01ec14be60 con 0x7f01ec0709d0 2024-04-25T01:18:55.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.313+0000 7f01eb200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f01ec0cf5f0 con 0x7f01ec14ba80 2024-04-25T01:18:55.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.313+0000 7f01eb200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f01ec14bfe0 con 0x7f01ec0d01f0 2024-04-25T01:18:55.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.313+0000 7f01e9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01ec0709d0 0x7f01ec161ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:55.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.314+0000 7f01e9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01ec0709d0 0x7f01ec161ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45186/0 (socket says 172.21.15.90:45186) 2024-04-25T01:18:55.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.314+0000 7f01e9e00640 1 -- 172.21.15.90:0/1309194810 learned_addr learned my addr 172.21.15.90:0/1309194810 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:55.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.314+0000 7f01e9e00640 1 -- 172.21.15.90:0/1309194810 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01ec0d01f0 msgr2=0x7f01ec0a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:55.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.314+0000 7f01e9e00640 1 --2- 172.21.15.90:0/1309194810 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01ec0d01f0 0x7f01ec0a94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:55.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.314+0000 7f01e9e00640 1 -- 172.21.15.90:0/1309194810 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01ec14ba80 msgr2=0x7f01ec1499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:55.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.314+0000 7f01ea800640 1 --2- 172.21.15.90:0/1309194810 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01ec14ba80 0x7f01ec1499e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:55.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.314+0000 7f01e9e00640 1 --2- 172.21.15.90:0/1309194810 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01ec14ba80 0x7f01ec1499e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:55.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.314+0000 7f01e9e00640 1 -- 172.21.15.90:0/1309194810 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01ec162400 con 0x7f01ec0709d0 2024-04-25T01:18:55.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.314+0000 7f01e9e00640 1 --2- 172.21.15.90:0/1309194810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01ec0709d0 0x7f01ec161ec0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f01dc00c820 tx=0x7f01dc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:55.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.315+0000 7f01e2a00640 1 -- 172.21.15.90:0/1309194810 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01dc015070 con 0x7f01ec0709d0 2024-04-25T01:18:55.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.315+0000 7f01e2a00640 1 -- 172.21.15.90:0/1309194810 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01dc0079d0 con 0x7f01ec0709d0 2024-04-25T01:18:55.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.315+0000 7f01e2a00640 1 -- 172.21.15.90:0/1309194810 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01dc007d80 con 0x7f01ec0709d0 2024-04-25T01:18:55.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.315+0000 7f01eb200640 1 -- 172.21.15.90:0/1309194810 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01ec148020 con 0x7f01ec0709d0 2024-04-25T01:18:55.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.315+0000 7f01eb200640 1 -- 172.21.15.90:0/1309194810 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01ec1485e0 con 0x7f01ec0709d0 2024-04-25T01:18:55.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.316+0000 7f01eb200640 1 -- 172.21.15.90:0/1309194810 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01ec04f500 con 0x7f01ec0709d0 2024-04-25T01:18:55.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.317+0000 7f01e2a00640 1 -- 172.21.15.90:0/1309194810 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f01dc018070 con 0x7f01ec0709d0 2024-04-25T01:18:55.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.318+0000 7f01e2a00640 1 --2- 172.21.15.90:0/1309194810 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f01c806be30 0x7f01c806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:55.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.318+0000 7f01e2a00640 1 -- 172.21.15.90:0/1309194810 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5421+0+0 (secure 0 0 0) 0x7f01dc08aae0 con 0x7f01ec0709d0 2024-04-25T01:18:55.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.318+0000 7f01e1400640 1 --2- 172.21.15.90:0/1309194810 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f01c806be30 0x7f01c806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:55.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.319+0000 7f01e1400640 1 --2- 172.21.15.90:0/1309194810 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f01c806be30 0x7f01c806e2f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f01ec14a6e0 tx=0x7f01d40079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:55.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.322+0000 7f01e2a00640 1 -- 172.21.15.90:0/1309194810 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f01dc059640 con 0x7f01ec0709d0 2024-04-25T01:18:55.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.550+0000 7f01eb200640 1 -- 172.21.15.90:0/1309194810 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f01ec0ce670 con 0x7f01ec0709d0 2024-04-25T01:18:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.557+0000 7f01e2a00640 1 -- 172.21.15.90:0/1309194810 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v12) ==== 104+0+0 (secure 0 0 0) 0x7f01dc058f40 con 0x7f01ec0709d0 2024-04-25T01:18:55.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.559+0000 7f01eb200640 1 -- 172.21.15.90:0/1309194810 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f01c806be30 msgr2=0x7f01c806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:55.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.559+0000 7f01eb200640 1 --2- 172.21.15.90:0/1309194810 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f01c806be30 0x7f01c806e2f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f01ec14a6e0 tx=0x7f01d40079b0 comp rx=0 tx=0).stop 2024-04-25T01:18:55.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.559+0000 7f01eb200640 1 -- 172.21.15.90:0/1309194810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01ec0709d0 msgr2=0x7f01ec161ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:55.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.559+0000 7f01eb200640 1 --2- 172.21.15.90:0/1309194810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01ec0709d0 0x7f01ec161ec0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f01dc00c820 tx=0x7f01dc00cce0 comp rx=0 tx=0).stop 2024-04-25T01:18:55.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.560+0000 7f01eb200640 1 -- 172.21.15.90:0/1309194810 shutdown_connections 2024-04-25T01:18:55.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.560+0000 7f01eb200640 1 --2- 172.21.15.90:0/1309194810 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f01c806be30 0x7f01c806e2f0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:55.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.560+0000 7f01eb200640 1 --2- 172.21.15.90:0/1309194810 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01ec14ba80 0x7f01ec1499e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:55.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.560+0000 7f01eb200640 1 --2- 172.21.15.90:0/1309194810 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f01ec0d01f0 0x7f01ec0a94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:55.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.560+0000 7f01eb200640 1 --2- 172.21.15.90:0/1309194810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01ec0709d0 0x7f01ec161ec0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:55.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.560+0000 7f01eb200640 1 -- 172.21.15.90:0/1309194810 >> 172.21.15.90:0/1309194810 conn(0x7f01ec06b350 msgr2=0x7f01ec0ccb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:55.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.560+0000 7f01eb200640 1 -- 172.21.15.90:0/1309194810 shutdown_connections 2024-04-25T01:18:55.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.560+0000 7f01eb200640 1 -- 172.21.15.90:0/1309194810 wait complete. 2024-04-25T01:18:55.561 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds debug_mds 2024-04-25T01:18:55.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.778+0000 7fa632e00640 1 Processor -- start 2024-04-25T01:18:55.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.779+0000 7fa632e00640 1 -- start start 2024-04-25T01:18:55.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.780+0000 7fa632e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa62c06ed30 0x7fa62c0b3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:55.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.780+0000 7fa632e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa62c15e3f0 con 0x7fa62c06dd80 2024-04-25T01:18:55.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.780+0000 7fa632e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa62c06e230 con 0x7fa62c15e010 2024-04-25T01:18:55.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.780+0000 7fa632e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa62c06cd20 con 0x7fa62c06ed30 2024-04-25T01:18:55.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.781+0000 7fa623400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa62c06dd80 0x7fa62c06e620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:42092/0 (socket says 172.21.15.90:42092) 2024-04-25T01:18:55.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.781+0000 7fa623400640 1 -- 172.21.15.90:0/2219882293 learned_addr learned my addr 172.21.15.90:0/2219882293 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:55.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.781+0000 7fa630e00640 1 --2- 172.21.15.90:0/2219882293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa62c06ed30 0x7fa62c0b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:55.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.781+0000 7fa62b400640 1 -- 172.21.15.90:0/2219882293 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 718073351 0 0) 0x7fa62c15e3f0 con 0x7fa62c06dd80 2024-04-25T01:18:55.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.781+0000 7fa62b400640 1 -- 172.21.15.90:0/2219882293 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa610003520 con 0x7fa62c06dd80 2024-04-25T01:18:55.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.781+0000 7fa62b400640 1 -- 172.21.15.90:0/2219882293 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2452870728 0 0) 0x7fa610003520 con 0x7fa62c06dd80 2024-04-25T01:18:55.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.781+0000 7fa62b400640 1 -- 172.21.15.90:0/2219882293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa62c06ed30 msgr2=0x7fa62c0b3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:55.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.781+0000 7fa62b400640 1 --2- 172.21.15.90:0/2219882293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa62c06ed30 0x7fa62c0b3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:55.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.781+0000 7fa62b400640 1 -- 172.21.15.90:0/2219882293 >> v1:172.21.15.103:6789/0 conn(0x7fa62c15e010 legacy=0x7fa62c071520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:55.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.781+0000 7fa62b400640 1 -- 172.21.15.90:0/2219882293 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa62c15f440 con 0x7fa62c06dd80 2024-04-25T01:18:55.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.781+0000 7fa62b400640 1 -- 172.21.15.90:0/2219882293 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7fa614001670 con 0x7fa62c06dd80 2024-04-25T01:18:55.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.781+0000 7fa62b400640 1 -- 172.21.15.90:0/2219882293 >> v1:172.21.15.90:6789/0 conn(0x7fa62c06dd80 legacy=0x7fa62c06e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:55.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.781+0000 7fa62b400640 1 --2- 172.21.15.90:0/2219882293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6100039e0 0x7fa610003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:55.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.782+0000 7fa62b400640 1 --2- 172.21.15.90:0/2219882293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa62c06ed30 0x7fa610004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:55.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.782+0000 7fa62b400640 1 --2- 172.21.15.90:0/2219882293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa610004d40 0x7fa6100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:55.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.782+0000 7fa62b400640 1 -- 172.21.15.90:0/2219882293 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c06cd20 con 0x7fa6100039e0 2024-04-25T01:18:55.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.782+0000 7fa62b400640 1 -- 172.21.15.90:0/2219882293 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa61000bad0 con 0x7fa610004d40 2024-04-25T01:18:55.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.782+0000 7fa62b400640 1 -- 172.21.15.90:0/2219882293 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa61000bcd0 con 0x7fa62c06ed30 2024-04-25T01:18:55.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.782+0000 7fa62be00640 1 --2- 172.21.15.90:0/2219882293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6100039e0 0x7fa610003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:55.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.782+0000 7fa62be00640 1 -- 172.21.15.90:0/2219882293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa62c06ed30 msgr2=0x7fa610004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:55.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.782+0000 7fa62be00640 1 --2- 172.21.15.90:0/2219882293 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa62c06ed30 0x7fa610004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:55.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.782+0000 7fa62be00640 1 -- 172.21.15.90:0/2219882293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa610004d40 msgr2=0x7fa6100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:55.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.782+0000 7fa630e00640 1 --2- 172.21.15.90:0/2219882293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa610004d40 0x7fa6100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:55.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.782+0000 7fa62be00640 1 --2- 172.21.15.90:0/2219882293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa610004d40 0x7fa6100051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:55.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.782+0000 7fa62be00640 1 -- 172.21.15.90:0/2219882293 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa610003520 con 0x7fa6100039e0 2024-04-25T01:18:55.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.783+0000 7fa62be00640 1 --2- 172.21.15.90:0/2219882293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6100039e0 0x7fa610003e00 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fa61c001e20 tx=0x7fa610005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ec213e5d44925bf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:55.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.783+0000 7fa62b400640 1 -- 172.21.15.90:0/2219882293 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa61c001a70 con 0x7fa6100039e0 2024-04-25T01:18:55.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.784+0000 7fa62b400640 1 -- 172.21.15.90:0/2219882293 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa61c02ae40 con 0x7fa6100039e0 2024-04-25T01:18:55.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.784+0000 7fa62b400640 1 -- 172.21.15.90:0/2219882293 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa61c003bb0 con 0x7fa6100039e0 2024-04-25T01:18:55.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.784+0000 7fa632e00640 1 -- 172.21.15.90:0/2219882293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6100039e0 msgr2=0x7fa610003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:55.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.784+0000 7fa632e00640 1 --2- 172.21.15.90:0/2219882293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6100039e0 0x7fa610003e00 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fa61c001e20 tx=0x7fa610005830 comp rx=0 tx=0).stop 2024-04-25T01:18:55.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.784+0000 7fa630e00640 1 -- 172.21.15.90:0/2219882293 reap_dead start 2024-04-25T01:18:55.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.784+0000 7fa632e00640 1 -- 172.21.15.90:0/2219882293 shutdown_connections 2024-04-25T01:18:55.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.784+0000 7fa632e00640 1 -- 172.21.15.90:0/2219882293 >> 172.21.15.90:0/2219882293 conn(0x7fa62c0b17b0 msgr2=0x7fa62c0b1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:55.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.784+0000 7fa632e00640 1 -- 172.21.15.90:0/2219882293 shutdown_connections 2024-04-25T01:18:55.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.784+0000 7fa632e00640 1 -- 172.21.15.90:0/2219882293 wait complete. 2024-04-25T01:18:55.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.786+0000 7fa632e00640 1 Processor -- start 2024-04-25T01:18:55.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.786+0000 7fa632e00640 1 -- start start 2024-04-25T01:18:55.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.787+0000 7fa632e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6100039e0 0x7fa62c158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:55.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.787+0000 7fa632e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa62c156e80 0x7fa62c15b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:55.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.787+0000 7fa632e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa62c06e7d0 0x7fa62c15b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:55.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.787+0000 7fa632e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c06d2a0 con 0x7fa62c156e80 2024-04-25T01:18:55.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.787+0000 7fa632e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c06cfc0 con 0x7fa6100039e0 2024-04-25T01:18:55.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.787+0000 7fa632e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa62c06d580 con 0x7fa62c06e7d0 2024-04-25T01:18:55.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.787+0000 7fa623400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa62c156e80 0x7fa62c15b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:55.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.787+0000 7fa623400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa62c156e80 0x7fa62c15b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45206/0 (socket says 172.21.15.90:45206) 2024-04-25T01:18:55.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.787+0000 7fa623400640 1 -- 172.21.15.90:0/1353543987 learned_addr learned my addr 172.21.15.90:0/1353543987 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:55.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.788+0000 7fa623400640 1 -- 172.21.15.90:0/1353543987 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa62c06e7d0 msgr2=0x7fa62c15b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:55.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.788+0000 7fa623400640 1 --2- 172.21.15.90:0/1353543987 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa62c06e7d0 0x7fa62c15b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:55.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.788+0000 7fa623400640 1 -- 172.21.15.90:0/1353543987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6100039e0 msgr2=0x7fa62c158ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:55.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.788+0000 7fa623400640 1 --2- 172.21.15.90:0/1353543987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6100039e0 0x7fa62c158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:55.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.788+0000 7fa623400640 1 -- 172.21.15.90:0/1353543987 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa62c0c4ac0 con 0x7fa62c156e80 2024-04-25T01:18:55.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.788+0000 7fa623400640 1 --2- 172.21.15.90:0/1353543987 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa62c156e80 0x7fa62c15b000 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fa61400c730 tx=0x7fa61400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:55.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.788+0000 7fa628c00640 1 -- 172.21.15.90:0/1353543987 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa614018070 con 0x7fa62c156e80 2024-04-25T01:18:55.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.788+0000 7fa628c00640 1 -- 172.21.15.90:0/1353543987 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6140072d0 con 0x7fa62c156e80 2024-04-25T01:18:55.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.788+0000 7fa628c00640 1 -- 172.21.15.90:0/1353543987 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6140047e0 con 0x7fa62c156e80 2024-04-25T01:18:55.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.788+0000 7fa632e00640 1 -- 172.21.15.90:0/1353543987 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa62c0c4d60 con 0x7fa62c156e80 2024-04-25T01:18:55.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.789+0000 7fa632e00640 1 -- 172.21.15.90:0/1353543987 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa62c1594e0 con 0x7fa62c156e80 2024-04-25T01:18:55.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.790+0000 7fa632e00640 1 -- 172.21.15.90:0/1353543987 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa62c1614d0 con 0x7fa62c156e80 2024-04-25T01:18:55.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.791+0000 7fa628c00640 1 -- 172.21.15.90:0/1353543987 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa614004b90 con 0x7fa62c156e80 2024-04-25T01:18:55.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.792+0000 7fa628c00640 1 --2- 172.21.15.90:0/1353543987 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa60806be30 0x7fa60806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:55.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.792+0000 7fa628c00640 1 -- 172.21.15.90:0/1353543987 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5421+0+0 (secure 0 0 0) 0x7fa61408a440 con 0x7fa62c156e80 2024-04-25T01:18:55.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.792+0000 7fa62be00640 1 --2- 172.21.15.90:0/1353543987 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa60806be30 0x7fa60806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:55.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.793+0000 7fa62be00640 1 --2- 172.21.15.90:0/1353543987 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa60806be30 0x7fa60806e2f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa61c032000 tx=0x7fa61c02c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:55.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.796+0000 7fa628c00640 1 -- 172.21.15.90:0/1353543987 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa614058de0 con 0x7fa62c156e80 2024-04-25T01:18:55.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:55.996+0000 7fa632e00640 1 -- 172.21.15.90:0/1353543987 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "debug_mds"} v 0) -- 0x7fa62c0b3190 con 0x7fa62c156e80 2024-04-25T01:18:55.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.001+0000 7fa628c00640 1 -- 172.21.15.90:0/1353543987 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "debug_mds"}]=0 v13) ==== 92+0+0 (secure 0 0 0) 0x7fa6140586e0 con 0x7fa62c156e80 2024-04-25T01:18:55.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.004+0000 7fa632e00640 1 -- 172.21.15.90:0/1353543987 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa60806be30 msgr2=0x7fa60806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:55.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.004+0000 7fa632e00640 1 --2- 172.21.15.90:0/1353543987 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa60806be30 0x7fa60806e2f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa61c032000 tx=0x7fa61c02c000 comp rx=0 tx=0).stop 2024-04-25T01:18:55.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.004+0000 7fa632e00640 1 -- 172.21.15.90:0/1353543987 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa62c156e80 msgr2=0x7fa62c15b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:55.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.004+0000 7fa632e00640 1 --2- 172.21.15.90:0/1353543987 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa62c156e80 0x7fa62c15b000 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fa61400c730 tx=0x7fa61400cbf0 comp rx=0 tx=0).stop 2024-04-25T01:18:55.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.004+0000 7fa632e00640 1 -- 172.21.15.90:0/1353543987 shutdown_connections 2024-04-25T01:18:55.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.004+0000 7fa632e00640 1 --2- 172.21.15.90:0/1353543987 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa60806be30 0x7fa60806e2f0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:55.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.004+0000 7fa632e00640 1 --2- 172.21.15.90:0/1353543987 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa62c06e7d0 0x7fa62c15b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:55.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.004+0000 7fa632e00640 1 --2- 172.21.15.90:0/1353543987 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa62c156e80 0x7fa62c15b000 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:55.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.004+0000 7fa632e00640 1 --2- 172.21.15.90:0/1353543987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6100039e0 0x7fa62c158ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:55.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.004+0000 7fa632e00640 1 -- 172.21.15.90:0/1353543987 >> 172.21.15.90:0/1353543987 conn(0x7fa62c0b17b0 msgr2=0x7fa62c0b33b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:55.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.005+0000 7fa632e00640 1 -- 172.21.15.90:0/1353543987 shutdown_connections 2024-04-25T01:18:55.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.005+0000 7fa632e00640 1 -- 172.21.15.90:0/1353543987 wait complete. 2024-04-25T01:18:56.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_recovery_pool.TestRecoveryPool.test_rebuild_simple' 2024-04-25T01:18:56.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.223+0000 7f2d52a00640 1 Processor -- start 2024-04-25T01:18:56.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.223+0000 7f2d52a00640 1 -- start start 2024-04-25T01:18:56.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.225+0000 7f2d52a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2d4c145640 0x7f2d4c15ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:56.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.225+0000 7f2d52a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d4c0cdf70 con 0x7f2d4c0b2a20 2024-04-25T01:18:56.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.225+0000 7f2d52a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d4c15d360 con 0x7f2d4c0d03a0 2024-04-25T01:18:56.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.225+0000 7f2d52a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f2d4c145a20 con 0x7f2d4c145640 2024-04-25T01:18:56.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.225+0000 7f2d4b400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f2d4c0b2a20 0x7f2d4c0cfc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:42094/0 (socket says 172.21.15.90:42094) 2024-04-25T01:18:56.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.225+0000 7f2d4b400640 1 -- 172.21.15.90:0/3534420151 learned_addr learned my addr 172.21.15.90:0/3534420151 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:56.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.225+0000 7f2d4be00640 1 --2- 172.21.15.90:0/3534420151 >> v2:172.21.15.90:3300/0 conn(0x7f2d4c145640 0x7f2d4c15ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:56.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.226+0000 7f2d4aa00640 1 -- 172.21.15.90:0/3534420151 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3540408062 0 0) 0x7f2d4c0cdf70 con 0x7f2d4c0b2a20 2024-04-25T01:18:56.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.226+0000 7f2d4aa00640 1 -- 172.21.15.90:0/3534420151 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d2c003520 con 0x7f2d4c0b2a20 2024-04-25T01:18:56.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.226+0000 7f2d4be00640 1 -- 172.21.15.90:0/3534420151 >> v1:172.21.15.103:6789/0 conn(0x7f2d4c0d03a0 legacy=0x7f2d4c0ce450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:56.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.226+0000 7f2d4be00640 1 -- 172.21.15.90:0/3534420151 >> v1:172.21.15.90:6789/0 conn(0x7f2d4c0b2a20 legacy=0x7f2d4c0cfc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:56.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.226+0000 7f2d4be00640 1 -- 172.21.15.90:0/3534420151 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d4c06d140 con 0x7f2d4c145640 2024-04-25T01:18:56.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.227+0000 7f2d4be00640 1 --2- 172.21.15.90:0/3534420151 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d4c145640 0x7f2d4c15ce20 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f2d44009d90 tx=0x7f2d4402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e50e81fb22b8c0a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:56.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.227+0000 7f2d4aa00640 1 -- 172.21.15.90:0/3534420151 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d44007740 con 0x7f2d4c145640 2024-04-25T01:18:56.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.227+0000 7f2d4aa00640 1 -- 172.21.15.90:0/3534420151 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d440078e0 con 0x7f2d4c145640 2024-04-25T01:18:56.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.227+0000 7f2d4aa00640 1 -- 172.21.15.90:0/3534420151 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d44007cb0 con 0x7f2d4c145640 2024-04-25T01:18:56.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.228+0000 7f2d52a00640 1 -- 172.21.15.90:0/3534420151 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d4c145640 msgr2=0x7f2d4c15ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:56.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.228+0000 7f2d52a00640 1 --2- 172.21.15.90:0/3534420151 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d4c145640 0x7f2d4c15ce20 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f2d44009d90 tx=0x7f2d4402a6a0 comp rx=0 tx=0).stop 2024-04-25T01:18:56.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.228+0000 7f2d52a00640 1 -- 172.21.15.90:0/3534420151 shutdown_connections 2024-04-25T01:18:56.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.228+0000 7f2d52a00640 1 --2- 172.21.15.90:0/3534420151 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d4c145640 0x7f2d4c15ce20 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:56.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.228+0000 7f2d52a00640 1 -- 172.21.15.90:0/3534420151 >> 172.21.15.90:0/3534420151 conn(0x7f2d4c06b320 msgr2=0x7f2d4c06b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:56.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.228+0000 7f2d52a00640 1 -- 172.21.15.90:0/3534420151 shutdown_connections 2024-04-25T01:18:56.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.228+0000 7f2d52a00640 1 -- 172.21.15.90:0/3534420151 wait complete. 2024-04-25T01:18:56.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.230+0000 7f2d52a00640 1 Processor -- start 2024-04-25T01:18:56.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.230+0000 7f2d52a00640 1 -- start start 2024-04-25T01:18:56.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.231+0000 7f2d52a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d4c0b2a20 0x7f2d4c0c4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:56.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.231+0000 7f2d52a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d4c0d03a0 0x7f2d4c158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:56.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.231+0000 7f2d52a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d4c145640 0x7f2d4c072560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:56.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.231+0000 7f2d52a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c06d9e0 con 0x7f2d4c0d03a0 2024-04-25T01:18:56.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.231+0000 7f2d52a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c06d700 con 0x7f2d4c145640 2024-04-25T01:18:56.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.231+0000 7f2d52a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f2d4c06cbc0 con 0x7f2d4c0b2a20 2024-04-25T01:18:56.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.231+0000 7f2d4b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d4c0d03a0 0x7f2d4c158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:56.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.231+0000 7f2d4b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d4c0d03a0 0x7f2d4c158770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45224/0 (socket says 172.21.15.90:45224) 2024-04-25T01:18:56.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.231+0000 7f2d4b400640 1 -- 172.21.15.90:0/1984554959 learned_addr learned my addr 172.21.15.90:0/1984554959 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:56.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.232+0000 7f2d50a00640 1 --2- 172.21.15.90:0/1984554959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d4c145640 0x7f2d4c072560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:56.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.232+0000 7f2d4b400640 1 -- 172.21.15.90:0/1984554959 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d4c0b2a20 msgr2=0x7f2d4c0c4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:56.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.232+0000 7f2d4be00640 1 --2- 172.21.15.90:0/1984554959 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d4c0b2a20 0x7f2d4c0c4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:56.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.232+0000 7f2d4b400640 1 --2- 172.21.15.90:0/1984554959 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d4c0b2a20 0x7f2d4c0c4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:56.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.232+0000 7f2d4b400640 1 -- 172.21.15.90:0/1984554959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d4c145640 msgr2=0x7f2d4c072560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:56.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.232+0000 7f2d4b400640 1 --2- 172.21.15.90:0/1984554959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d4c145640 0x7f2d4c072560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:56.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.232+0000 7f2d4b400640 1 -- 172.21.15.90:0/1984554959 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d4c0714c0 con 0x7f2d4c0d03a0 2024-04-25T01:18:56.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.232+0000 7f2d4b400640 1 --2- 172.21.15.90:0/1984554959 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d4c0d03a0 0x7f2d4c158770 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f2d340010e0 tx=0x7f2d3400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:56.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.232+0000 7f2d43e00640 1 -- 172.21.15.90:0/1984554959 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d34017070 con 0x7f2d4c0d03a0 2024-04-25T01:18:56.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.232+0000 7f2d43e00640 1 -- 172.21.15.90:0/1984554959 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d34005d00 con 0x7f2d4c0d03a0 2024-04-25T01:18:56.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.232+0000 7f2d52a00640 1 -- 172.21.15.90:0/1984554959 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d4c0717a0 con 0x7f2d4c0d03a0 2024-04-25T01:18:56.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.232+0000 7f2d43e00640 1 -- 172.21.15.90:0/1984554959 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d340072b0 con 0x7f2d4c0d03a0 2024-04-25T01:18:56.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.233+0000 7f2d52a00640 1 -- 172.21.15.90:0/1984554959 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d4c071d60 con 0x7f2d4c0d03a0 2024-04-25T01:18:56.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.233+0000 7f2d52a00640 1 -- 172.21.15.90:0/1984554959 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d4c068190 con 0x7f2d4c0d03a0 2024-04-25T01:18:56.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.235+0000 7f2d43e00640 1 -- 172.21.15.90:0/1984554959 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2d3401a070 con 0x7f2d4c0d03a0 2024-04-25T01:18:56.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.235+0000 7f2d43e00640 1 --2- 172.21.15.90:0/1984554959 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2d2806be60 0x7f2d2806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:56.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.236+0000 7f2d43e00640 1 -- 172.21.15.90:0/1984554959 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5421+0+0 (secure 0 0 0) 0x7f2d3408a0d0 con 0x7f2d4c0d03a0 2024-04-25T01:18:56.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.236+0000 7f2d4be00640 1 --2- 172.21.15.90:0/1984554959 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2d2806be60 0x7f2d2806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:56.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.236+0000 7f2d4be00640 1 --2- 172.21.15.90:0/1984554959 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2d2806be60 0x7f2d2806e320 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2d44034000 tx=0x7f2d44033040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:56.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.240+0000 7f2d43e00640 1 -- 172.21.15.90:0/1984554959 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2d34058c30 con 0x7f2d4c0d03a0 2024-04-25T01:18:56.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:56.507+0000 7f2d52a00640 1 -- 172.21.15.90:0/1984554959 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_recovery_pool.TestRecoveryPool.test_rebuild_simple"]} v 0) -- 0x7f2d4c0b4620 con 0x7f2d4c0d03a0 2024-04-25T01:18:57.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:57.091+0000 7f2d43e00640 1 -- 172.21.15.90:0/1984554959 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_recovery_pool.TestRecoveryPool.test_rebuild_simple"]}]=0 v168) ==== 147+0+0 (secure 0 0 0) 0x7f2d34058530 con 0x7f2d4c0d03a0 2024-04-25T01:18:57.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:57.095+0000 7f2d52a00640 1 -- 172.21.15.90:0/1984554959 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2d2806be60 msgr2=0x7f2d2806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:57.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:57.095+0000 7f2d52a00640 1 --2- 172.21.15.90:0/1984554959 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2d2806be60 0x7f2d2806e320 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2d44034000 tx=0x7f2d44033040 comp rx=0 tx=0).stop 2024-04-25T01:18:57.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:57.095+0000 7f2d52a00640 1 -- 172.21.15.90:0/1984554959 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d4c0d03a0 msgr2=0x7f2d4c158770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:57.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:57.095+0000 7f2d52a00640 1 --2- 172.21.15.90:0/1984554959 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d4c0d03a0 0x7f2d4c158770 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f2d340010e0 tx=0x7f2d3400c960 comp rx=0 tx=0).stop 2024-04-25T01:18:57.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:57.095+0000 7f2d52a00640 1 -- 172.21.15.90:0/1984554959 shutdown_connections 2024-04-25T01:18:57.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:57.095+0000 7f2d52a00640 1 --2- 172.21.15.90:0/1984554959 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f2d2806be60 0x7f2d2806e320 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:57.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:57.095+0000 7f2d52a00640 1 --2- 172.21.15.90:0/1984554959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d4c145640 0x7f2d4c072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:57.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:57.095+0000 7f2d52a00640 1 --2- 172.21.15.90:0/1984554959 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d4c0d03a0 0x7f2d4c158770 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:57.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:57.095+0000 7f2d52a00640 1 --2- 172.21.15.90:0/1984554959 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f2d4c0b2a20 0x7f2d4c0c4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:57.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:57.095+0000 7f2d52a00640 1 -- 172.21.15.90:0/1984554959 >> 172.21.15.90:0/1984554959 conn(0x7f2d4c06b320 msgr2=0x7f2d4c0aff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:57.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:57.096+0000 7f2d52a00640 1 -- 172.21.15.90:0/1984554959 shutdown_connections 2024-04-25T01:18:57.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:57.096+0000 7f2d52a00640 1 -- 172.21.15.90:0/1984554959 wait complete. 2024-04-25T01:18:57.099 INFO:tasks.cephfs_test_runner:test_rebuild_simple (tasks.cephfs.test_recovery_pool.TestRecoveryPool) ... ok 2024-04-25T01:18:57.099 INFO:tasks.cephfs_test_runner: 2024-04-25T01:18:57.099 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-25T01:18:57.099 INFO:tasks.cephfs_test_runner:Ran 1 test in 88.191s 2024-04-25T01:18:57.099 INFO:tasks.cephfs_test_runner: 2024-04-25T01:18:57.099 INFO:tasks.cephfs_test_runner:OK 2024-04-25T01:18:57.100 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-04-25T01:18:57.108 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-25T01:18:57.115 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-25T01:18:57.115 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:18:57.116 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:18:57.168 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052717,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T01:18:57.169 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T01:18:57.169 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:18:57.222 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052717,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T01:18:57.223 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:18:57.223 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:18:57.280 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052734,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T01:18:57.281 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-25T01:18:57.281 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:18:57.333 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052734,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T01:18:57.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:57.333 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-04-25T01:18:57.390 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T01:18:57.390 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-25T01:18:57.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:57.391 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T01:18:57.481 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:57.481 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T01:18:57.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:57.552 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link delete ceph-brx 2024-04-25T01:18:57.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:57.686 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-04-25T01:18:57.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:57.731 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link delete ceph-brx 2024-04-25T01:18:57.849 INFO:teuthology.orchestra.run.smithi090.stderr:Cannot find device "ceph-brx" 2024-04-25T01:18:57.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:18:57.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:57.853 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-04-25T01:18:57.909 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-25T01:18:57.910 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-25T01:18:57.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:57.911 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T01:18:58.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:58.002 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T01:18:58.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:58.079 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link delete ceph-brx 2024-04-25T01:18:58.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:58.228 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-04-25T01:18:58.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:18:58.270 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link delete ceph-brx 2024-04-25T01:18:58.389 INFO:teuthology.orchestra.run.smithi103.stderr:Cannot find device "ceph-brx" 2024-04-25T01:18:58.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:18:58.393 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-25T01:18:58.402 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-25T01:18:58.402 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:18:58.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.630+0000 7f7bf3200640 1 Processor -- start 2024-04-25T01:18:58.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.630+0000 7f7bf3200640 1 -- start start 2024-04-25T01:18:58.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.631+0000 7f7bf3200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7bf415a980 0x7f7bf406d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:58.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.631+0000 7f7bf3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bf4071dd0 0x7f7bf40cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:58.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.631+0000 7f7bf3200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bf40ccb10 con 0x7f7bf40d0070 2024-04-25T01:18:58.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.631+0000 7f7bf3200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f7bf4072230 con 0x7f7bf415a980 2024-04-25T01:18:58.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.631+0000 7f7bf3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7bf40aede0 con 0x7f7bf4071dd0 2024-04-25T01:18:58.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.632+0000 7f7bf1400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bf4071dd0 0x7f7bf40cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:58.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.632+0000 7f7bf1e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7bf415a980 0x7f7bf406d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:58.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.632+0000 7f7bf1400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bf4071dd0 0x7f7bf40cfb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.90:45782/0 (socket says 172.21.15.90:45782) 2024-04-25T01:18:58.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.632+0000 7f7bf1400640 1 -- 172.21.15.90:0/2187760496 learned_addr learned my addr 172.21.15.90:0/2187760496 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:58.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.633+0000 7f7bf1e00640 1 -- 172.21.15.90:0/2187760496 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bf4071dd0 msgr2=0x7f7bf40cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:58.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.633+0000 7f7bf1e00640 1 --2- 172.21.15.90:0/2187760496 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bf4071dd0 0x7f7bf40cfb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:58.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.633+0000 7f7bf1e00640 1 -- 172.21.15.90:0/2187760496 >> v1:172.21.15.103:6789/0 conn(0x7f7bf40d0070 legacy=0x7f7bf40b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:58.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.633+0000 7f7bf1e00640 1 -- 172.21.15.90:0/2187760496 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bf406ddc0 con 0x7f7bf415a980 2024-04-25T01:18:58.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.633+0000 7f7bf1400640 1 --2- 172.21.15.90:0/2187760496 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bf4071dd0 0x7f7bf40cfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:18:58.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.634+0000 7f7bf1e00640 1 --2- 172.21.15.90:0/2187760496 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bf415a980 0x7f7bf406d770 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7be40097a0 tx=0x7f7be402a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31549e260f63dee2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:58.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.635+0000 7f7bf0a00640 1 -- 172.21.15.90:0/2187760496 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be4037070 con 0x7f7bf415a980 2024-04-25T01:18:58.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.635+0000 7f7bf0a00640 1 -- 172.21.15.90:0/2187760496 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7be402ae50 con 0x7f7bf415a980 2024-04-25T01:18:58.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.635+0000 7f7bf0a00640 1 -- 172.21.15.90:0/2187760496 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be4032660 con 0x7f7bf415a980 2024-04-25T01:18:58.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.635+0000 7f7bf3200640 1 -- 172.21.15.90:0/2187760496 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bf415a980 msgr2=0x7f7bf406d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:58.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.635+0000 7f7bf3200640 1 --2- 172.21.15.90:0/2187760496 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bf415a980 0x7f7bf406d770 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7be40097a0 tx=0x7f7be402a100 comp rx=0 tx=0).stop 2024-04-25T01:18:58.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.635+0000 7f7bf3200640 1 -- 172.21.15.90:0/2187760496 shutdown_connections 2024-04-25T01:18:58.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.635+0000 7f7bf3200640 1 --2- 172.21.15.90:0/2187760496 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bf4071dd0 0x7f7bf40cfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:58.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.635+0000 7f7bf3200640 1 --2- 172.21.15.90:0/2187760496 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bf415a980 0x7f7bf406d770 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:58.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.635+0000 7f7bf3200640 1 -- 172.21.15.90:0/2187760496 >> 172.21.15.90:0/2187760496 conn(0x7f7bf406b320 msgr2=0x7f7bf406b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:58.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.635+0000 7f7bf3200640 1 -- 172.21.15.90:0/2187760496 shutdown_connections 2024-04-25T01:18:58.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.635+0000 7f7bf3200640 1 -- 172.21.15.90:0/2187760496 wait complete. 2024-04-25T01:18:58.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.637+0000 7f7bf3200640 1 Processor -- start 2024-04-25T01:18:58.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.637+0000 7f7bf3200640 1 -- start start 2024-04-25T01:18:58.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.638+0000 7f7bf3200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bf4071dd0 0x7f7bf4146960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:58.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.638+0000 7f7bf3200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7bf40d0070 0x7f7bf40aa3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:58.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.638+0000 7f7bf3200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bf415a980 0x7f7bf4148650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:58.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.638+0000 7f7bf3200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf406e660 con 0x7f7bf40d0070 2024-04-25T01:18:58.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.638+0000 7f7bf3200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf406e380 con 0x7f7bf415a980 2024-04-25T01:18:58.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.638+0000 7f7bf3200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7bf415ad60 con 0x7f7bf4071dd0 2024-04-25T01:18:58.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.639+0000 7f7bf1400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7bf40d0070 0x7f7bf40aa3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:58.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.639+0000 7f7bf1400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7bf40d0070 0x7f7bf40aa3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45234/0 (socket says 172.21.15.90:45234) 2024-04-25T01:18:58.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.639+0000 7f7bf1400640 1 -- 172.21.15.90:0/2393705037 learned_addr learned my addr 172.21.15.90:0/2393705037 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:58.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.639+0000 7f7bf1e00640 1 --2- 172.21.15.90:0/2393705037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bf4071dd0 0x7f7bf4146960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:58.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.639+0000 7f7bf2800640 1 --2- 172.21.15.90:0/2393705037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bf415a980 0x7f7bf4148650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:58.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.639+0000 7f7bf1400640 1 -- 172.21.15.90:0/2393705037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bf4071dd0 msgr2=0x7f7bf4146960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:58.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.639+0000 7f7bf1400640 1 --2- 172.21.15.90:0/2393705037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bf4071dd0 0x7f7bf4146960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:58.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.639+0000 7f7bf1400640 1 -- 172.21.15.90:0/2393705037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bf415a980 msgr2=0x7f7bf4148650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:58.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.639+0000 7f7bf1400640 1 --2- 172.21.15.90:0/2393705037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bf415a980 0x7f7bf4148650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:58.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.639+0000 7f7bf1400640 1 -- 172.21.15.90:0/2393705037 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bf41475b0 con 0x7f7bf40d0070 2024-04-25T01:18:58.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.639+0000 7f7bf2800640 1 --2- 172.21.15.90:0/2393705037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bf415a980 0x7f7bf4148650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:58.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.639+0000 7f7bf1400640 1 --2- 172.21.15.90:0/2393705037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7bf40d0070 0x7f7bf40aa3a0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc00bfd0 tx=0x7f7bdc00e7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:58.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.640+0000 7f7bea000640 1 -- 172.21.15.90:0/2393705037 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bdc003990 con 0x7f7bf40d0070 2024-04-25T01:18:58.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.640+0000 7f7bea000640 1 -- 172.21.15.90:0/2393705037 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bdc003b30 con 0x7f7bf40d0070 2024-04-25T01:18:58.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.640+0000 7f7bf3200640 1 -- 172.21.15.90:0/2393705037 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bf4147890 con 0x7f7bf40d0070 2024-04-25T01:18:58.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.640+0000 7f7bea000640 1 -- 172.21.15.90:0/2393705037 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bdc004030 con 0x7f7bf40d0070 2024-04-25T01:18:58.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.640+0000 7f7bf3200640 1 -- 172.21.15.90:0/2393705037 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bf4147e50 con 0x7f7bf40d0070 2024-04-25T01:18:58.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.641+0000 7f7bf3200640 1 -- 172.21.15.90:0/2393705037 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bf404f500 con 0x7f7bf40d0070 2024-04-25T01:18:58.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.642+0000 7f7bea000640 1 -- 172.21.15.90:0/2393705037 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7bdc005d10 con 0x7f7bf40d0070 2024-04-25T01:18:58.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.643+0000 7f7bea000640 1 --2- 172.21.15.90:0/2393705037 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7bd006be30 0x7f7bd006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:58.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.643+0000 7f7bea000640 1 -- 172.21.15.90:0/2393705037 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5421+0+0 (secure 0 0 0) 0x7f7bdc08b1e0 con 0x7f7bf40d0070 2024-04-25T01:18:58.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.643+0000 7f7bf1e00640 1 --2- 172.21.15.90:0/2393705037 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7bd006be30 0x7f7bd006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:58.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.644+0000 7f7bf1e00640 1 --2- 172.21.15.90:0/2393705037 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7bd006be30 0x7f7bd006e2f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7be4034000 tx=0x7f7be402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:58.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.648+0000 7f7bea000640 1 -- 172.21.15.90:0/2393705037 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7bdc022020 con 0x7f7bf40d0070 2024-04-25T01:18:58.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.843+0000 7f7bf3200640 1 -- 172.21.15.90:0/2393705037 --> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7bf415dce0 con 0x7f7bd006be30 2024-04-25T01:18:58.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.845+0000 7f7bea000640 1 -- 172.21.15.90:0/2393705037 <== mgr.4104 v2:172.21.15.90:6832/2198086733 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13508 (secure 0 0 0) 0x7f7be40454e0 con 0x7f7bd006be30 2024-04-25T01:18:58.828 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:58.828 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-25T01:18:58.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.847+0000 7f7bf3200640 1 -- 172.21.15.90:0/2393705037 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7bd006be30 msgr2=0x7f7bd006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:58.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.847+0000 7f7bf3200640 1 --2- 172.21.15.90:0/2393705037 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7bd006be30 0x7f7bd006e2f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7be4034000 tx=0x7f7be402e000 comp rx=0 tx=0).stop 2024-04-25T01:18:58.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.847+0000 7f7bf3200640 1 -- 172.21.15.90:0/2393705037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7bf40d0070 msgr2=0x7f7bf40aa3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:58.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.847+0000 7f7bf3200640 1 --2- 172.21.15.90:0/2393705037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7bf40d0070 0x7f7bf40aa3a0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc00bfd0 tx=0x7f7bdc00e7d0 comp rx=0 tx=0).stop 2024-04-25T01:18:58.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.847+0000 7f7bf3200640 1 -- 172.21.15.90:0/2393705037 shutdown_connections 2024-04-25T01:18:58.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.847+0000 7f7bf3200640 1 --2- 172.21.15.90:0/2393705037 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7bd006be30 0x7f7bd006e2f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:58.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.847+0000 7f7bf3200640 1 --2- 172.21.15.90:0/2393705037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bf415a980 0x7f7bf4148650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:58.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.847+0000 7f7bf3200640 1 --2- 172.21.15.90:0/2393705037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7bf40d0070 0x7f7bf40aa3a0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:58.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.847+0000 7f7bf3200640 1 --2- 172.21.15.90:0/2393705037 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7bf4071dd0 0x7f7bf4146960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:58.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.847+0000 7f7bf3200640 1 -- 172.21.15.90:0/2393705037 >> 172.21.15.90:0/2393705037 conn(0x7f7bf406b320 msgr2=0x7f7bf40cdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:58.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.848+0000 7f7bf3200640 1 -- 172.21.15.90:0/2393705037 shutdown_connections 2024-04-25T01:18:58.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:58.848+0000 7f7bf3200640 1 -- 172.21.15.90:0/2393705037 wait complete. 2024-04-25T01:18:58.850 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-04-25T01:18:57.873941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":242816,"kb_used_data":18328,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754731904,"statfs":{"total":773094113280,"available":772845469696,"internally_reserved":0,"allocated":18767872,"data_stored":16038532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":229756465},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6322156,"num_objects":-15,"num_object_clones":0,"num_object_copies":-30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-15,"num_whiteouts":0,"num_read":-53,"num_read_kb":-39,"num_write":-50,"num_write_kb":-6193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001549"},"pg_stats":[{"pgid":"1.0","version":"18'117","reported_seq":226,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-25T01:18:54.007929+0000","last_change":"2024-04-25T01:16:17.466569+0000","last_active":"2024-04-25T01:18:54.007929+0000","last_peered":"2024-04-25T01:18:54.007929+0000","last_clean":"2024-04-25T01:18:54.007929+0000","last_became_active":"2024-04-25T01:16:17.465687+0000","last_became_peered":"2024-04-25T01:16:17.465687+0000","last_unstale":"2024-04-25T01:18:54.007929+0000","last_undegraded":"2024-04-25T01:18:54.007929+0000","last_fullsized":"2024-04-25T01:18:54.007929+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:16.440559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:16.440559+0000","last_clean_scrub_stamp":"2024-04-25T01:16:16.440559+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:18:17.493555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542179,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":626688,"data_stored":284542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542178,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":614400,"data_stored":278570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542179,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36472,"kb_used_data":5288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335368,"statfs":{"total":96636764160,"available":96599416832,"internally_reserved":0,"allocated":5414912,"data_stored":5069214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542178,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":659456,"data_stored":307982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542178,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33764,"kb_used_data":6744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338076,"statfs":{"total":96636764160,"available":96602189824,"internally_reserved":0,"allocated":6905856,"data_stored":6569534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542180,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":626688,"data_stored":284542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32360,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339480,"statfs":{"total":96636764160,"available":96603627520,"internally_reserved":0,"allocated":1208320,"data_stored":868426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542179,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29668,"kb_used_data":2648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342172,"statfs":{"total":96636764160,"available":96606384128,"internally_reserved":0,"allocated":2711552,"data_stored":2375722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:18:58.851 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:18:59.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.069+0000 7f0280a00640 1 Processor -- start 2024-04-25T01:18:59.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.069+0000 7f0280a00640 1 -- start start 2024-04-25T01:18:59.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.071+0000 7f0280a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f027c0709d0 0x7f027c0cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:59.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.071+0000 7f0280a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f027c070f90 con 0x7f027c1561c0 2024-04-25T01:18:59.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.071+0000 7f0280a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f027c071270 con 0x7f027c0d0240 2024-04-25T01:18:59.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.071+0000 7f0280a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f027c0b4310 con 0x7f027c0709d0 2024-04-25T01:18:59.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.071+0000 7f027b400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f027c1561c0 0x7f027c0cfb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:42104/0 (socket says 172.21.15.90:42104) 2024-04-25T01:18:59.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.071+0000 7f027b400640 1 -- 172.21.15.90:0/3734133798 learned_addr learned my addr 172.21.15.90:0/3734133798 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:18:59.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.071+0000 7f027be00640 1 --2- 172.21.15.90:0/3734133798 >> v2:172.21.15.103:3300/0 conn(0x7f027c0709d0 0x7f027c0cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:59.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.072+0000 7f027aa00640 1 -- 172.21.15.90:0/3734133798 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3694303593 0 0) 0x7f027c070f90 con 0x7f027c1561c0 2024-04-25T01:18:59.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.072+0000 7f027aa00640 1 -- 172.21.15.90:0/3734133798 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0260003520 con 0x7f027c1561c0 2024-04-25T01:18:59.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.072+0000 7f027aa00640 1 -- 172.21.15.90:0/3734133798 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3757123669 0 0) 0x7f0260003520 con 0x7f027c1561c0 2024-04-25T01:18:59.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.072+0000 7f027aa00640 1 -- 172.21.15.90:0/3734133798 >> v2:172.21.15.103:3300/0 conn(0x7f027c0709d0 msgr2=0x7f027c0cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:59.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.072+0000 7f027aa00640 1 --2- 172.21.15.90:0/3734133798 >> v2:172.21.15.103:3300/0 conn(0x7f027c0709d0 0x7f027c0cdbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:59.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.072+0000 7f027aa00640 1 -- 172.21.15.90:0/3734133798 >> v1:172.21.15.103:6789/0 conn(0x7f027c0d0240 legacy=0x7f027c0b2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:59.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.072+0000 7f027aa00640 1 -- 172.21.15.90:0/3734133798 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f027c071550 con 0x7f027c1561c0 2024-04-25T01:18:59.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.072+0000 7f027aa00640 1 -- 172.21.15.90:0/3734133798 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f026c001670 con 0x7f027c1561c0 2024-04-25T01:18:59.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.072+0000 7f027aa00640 1 -- 172.21.15.90:0/3734133798 >> v1:172.21.15.90:6789/0 conn(0x7f027c1561c0 legacy=0x7f027c0cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:59.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.072+0000 7f027aa00640 1 --2- 172.21.15.90:0/3734133798 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f02600039b0 0x7f0260003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:59.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.072+0000 7f027aa00640 1 --2- 172.21.15.90:0/3734133798 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0260004800 0x7f0260004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:59.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.072+0000 7f027aa00640 1 --2- 172.21.15.90:0/3734133798 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02600052e0 0x7f026000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:59.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.072+0000 7f027aa00640 1 -- 172.21.15.90:0/3734133798 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f027c0b4310 con 0x7f0260004800 2024-04-25T01:18:59.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.072+0000 7f027aa00640 1 -- 172.21.15.90:0/3734133798 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f026000c1b0 con 0x7f02600052e0 2024-04-25T01:18:59.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.072+0000 7f027aa00640 1 -- 172.21.15.90:0/3734133798 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f026000c3b0 con 0x7f02600039b0 2024-04-25T01:18:59.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.072+0000 7f0272a00640 1 --2- 172.21.15.90:0/3734133798 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0260004800 0x7f0260004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:59.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.073+0000 7f027be00640 1 --2- 172.21.15.90:0/3734133798 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02600052e0 0x7f026000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:59.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.073+0000 7f027b400640 1 --2- 172.21.15.90:0/3734133798 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f02600039b0 0x7f0260003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:59.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.073+0000 7f0272a00640 1 -- 172.21.15.90:0/3734133798 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f02600039b0 msgr2=0x7f0260003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:59.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.073+0000 7f0272a00640 1 --2- 172.21.15.90:0/3734133798 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f02600039b0 0x7f0260003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:59.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.073+0000 7f0272a00640 1 -- 172.21.15.90:0/3734133798 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02600052e0 msgr2=0x7f026000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:59.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.073+0000 7f0272a00640 1 --2- 172.21.15.90:0/3734133798 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02600052e0 0x7f026000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:59.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.073+0000 7f0272a00640 1 -- 172.21.15.90:0/3734133798 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0260003520 con 0x7f0260004800 2024-04-25T01:18:59.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.073+0000 7f027b400640 1 --2- 172.21.15.90:0/3734133798 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f02600039b0 0x7f0260003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:59.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.073+0000 7f027be00640 1 --2- 172.21.15.90:0/3734133798 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02600052e0 0x7f026000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:59.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.073+0000 7f027be00640 1 -- 172.21.15.90:0/3734133798 reap_dead start 2024-04-25T01:18:59.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.074+0000 7f0272a00640 1 --2- 172.21.15.90:0/3734133798 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0260004800 0x7f0260004c80 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f0264001720 tx=0x7f0260005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b3b839d738f5a44 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:59.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.074+0000 7f027aa00640 1 -- 172.21.15.90:0/3734133798 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0264001370 con 0x7f0260004800 2024-04-25T01:18:59.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.074+0000 7f027aa00640 1 -- 172.21.15.90:0/3734133798 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02640052d0 con 0x7f0260004800 2024-04-25T01:18:59.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.074+0000 7f027aa00640 1 -- 172.21.15.90:0/3734133798 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026402ab90 con 0x7f0260004800 2024-04-25T01:18:59.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.075+0000 7f0280a00640 1 -- 172.21.15.90:0/3734133798 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0260004800 msgr2=0x7f0260004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:59.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.075+0000 7f0280a00640 1 --2- 172.21.15.90:0/3734133798 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0260004800 0x7f0260004c80 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f0264001720 tx=0x7f0260005830 comp rx=0 tx=0).stop 2024-04-25T01:18:59.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.075+0000 7f0280a00640 1 -- 172.21.15.90:0/3734133798 shutdown_connections 2024-04-25T01:18:59.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.075+0000 7f0280a00640 1 --2- 172.21.15.90:0/3734133798 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0260004800 0x7f0260004c80 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:59.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.075+0000 7f0280a00640 1 -- 172.21.15.90:0/3734133798 >> 172.21.15.90:0/3734133798 conn(0x7f027c06b320 msgr2=0x7f027c06b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:59.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.075+0000 7f0280a00640 1 -- 172.21.15.90:0/3734133798 shutdown_connections 2024-04-25T01:18:59.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.075+0000 7f0280a00640 1 -- 172.21.15.90:0/3734133798 wait complete. 2024-04-25T01:18:59.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.077+0000 7f0280a00640 1 Processor -- start 2024-04-25T01:18:59.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.077+0000 7f0280a00640 1 -- start start 2024-04-25T01:18:59.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.078+0000 7f0280a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0260004800 0x7f027c148ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:59.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.078+0000 7f0280a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f027c06e510 0x7f027c15c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:59.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.078+0000 7f0280a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f027c0ad8c0 0x7f027c0add40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:59.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.078+0000 7f0280a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f027c0cdbf0 con 0x7f027c06e510 2024-04-25T01:18:59.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.078+0000 7f0280a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f027c0cdd70 con 0x7f027c0ad8c0 2024-04-25T01:18:59.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.078+0000 7f0280a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f027c0cdef0 con 0x7f0260004800 2024-04-25T01:18:59.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.078+0000 7f0272a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f027c06e510 0x7f027c15c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:59.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.078+0000 7f0272a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f027c06e510 0x7f027c15c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45254/0 (socket says 172.21.15.90:45254) 2024-04-25T01:18:59.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.078+0000 7f0272a00640 1 -- 172.21.15.90:0/2674679239 learned_addr learned my addr 172.21.15.90:0/2674679239 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:59.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.079+0000 7f0272a00640 1 -- 172.21.15.90:0/2674679239 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0260004800 msgr2=0x7f027c148ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:59.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.079+0000 7f0272a00640 1 --2- 172.21.15.90:0/2674679239 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0260004800 0x7f027c148ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:59.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.079+0000 7f0272a00640 1 -- 172.21.15.90:0/2674679239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f027c0ad8c0 msgr2=0x7f027c0add40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:59.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.079+0000 7f0272a00640 1 --2- 172.21.15.90:0/2674679239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f027c0ad8c0 0x7f027c0add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:59.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.079+0000 7f0272a00640 1 -- 172.21.15.90:0/2674679239 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f027c06d470 con 0x7f027c06e510 2024-04-25T01:18:59.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.079+0000 7f0272a00640 1 --2- 172.21.15.90:0/2674679239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f027c06e510 0x7f027c15c750 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f0264000c00 tx=0x7f0264001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:59.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.079+0000 7f0273e00640 1 -- 172.21.15.90:0/2674679239 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0264040030 con 0x7f027c06e510 2024-04-25T01:18:59.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.079+0000 7f0273e00640 1 -- 172.21.15.90:0/2674679239 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0264005480 con 0x7f027c06e510 2024-04-25T01:18:59.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.079+0000 7f0280a00640 1 -- 172.21.15.90:0/2674679239 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f027c06d750 con 0x7f027c06e510 2024-04-25T01:18:59.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.079+0000 7f0273e00640 1 -- 172.21.15.90:0/2674679239 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0264001370 con 0x7f027c06e510 2024-04-25T01:18:59.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.079+0000 7f0280a00640 1 -- 172.21.15.90:0/2674679239 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f027c06dd10 con 0x7f027c06e510 2024-04-25T01:18:59.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.080+0000 7f0280a00640 1 -- 172.21.15.90:0/2674679239 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f027c04f500 con 0x7f027c06e510 2024-04-25T01:18:59.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.081+0000 7f0273e00640 1 -- 172.21.15.90:0/2674679239 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0264043070 con 0x7f027c06e510 2024-04-25T01:18:59.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.082+0000 7f0273e00640 1 --2- 172.21.15.90:0/2674679239 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f025806be30 0x7f025806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:59.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.082+0000 7f0273e00640 1 -- 172.21.15.90:0/2674679239 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5421+0+0 (secure 0 0 0) 0x7f02640aa540 con 0x7f027c06e510 2024-04-25T01:18:59.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.083+0000 7f027b400640 1 --2- 172.21.15.90:0/2674679239 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f025806be30 0x7f025806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:59.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.083+0000 7f027b400640 1 --2- 172.21.15.90:0/2674679239 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f025806be30 0x7f025806e2f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f026c0010e0 tx=0x7f026c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:59.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.087+0000 7f0273e00640 1 -- 172.21.15.90:0/2674679239 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f026404b020 con 0x7f027c06e510 2024-04-25T01:18:59.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.283+0000 7f0280a00640 1 -- 172.21.15.90:0/2674679239 --> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f027c0cfde0 con 0x7f025806be30 2024-04-25T01:18:59.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.285+0000 7f0273e00640 1 -- 172.21.15.90:0/2674679239 <== mgr.4104 v2:172.21.15.90:6832/2198086733 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13508 (secure 0 0 0) 0x7f027c0cfde0 con 0x7f025806be30 2024-04-25T01:18:59.268 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:59.268 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-25T01:18:59.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.288+0000 7f0280a00640 1 -- 172.21.15.90:0/2674679239 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f025806be30 msgr2=0x7f025806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:59.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.288+0000 7f0280a00640 1 --2- 172.21.15.90:0/2674679239 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f025806be30 0x7f025806e2f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f026c0010e0 tx=0x7f026c009040 comp rx=0 tx=0).stop 2024-04-25T01:18:59.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.288+0000 7f0280a00640 1 -- 172.21.15.90:0/2674679239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f027c06e510 msgr2=0x7f027c15c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:59.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.288+0000 7f0280a00640 1 --2- 172.21.15.90:0/2674679239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f027c06e510 0x7f027c15c750 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f0264000c00 tx=0x7f0264001120 comp rx=0 tx=0).stop 2024-04-25T01:18:59.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.288+0000 7f0280a00640 1 -- 172.21.15.90:0/2674679239 shutdown_connections 2024-04-25T01:18:59.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.288+0000 7f0280a00640 1 --2- 172.21.15.90:0/2674679239 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f025806be30 0x7f025806e2f0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:59.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.288+0000 7f0280a00640 1 --2- 172.21.15.90:0/2674679239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f027c0ad8c0 0x7f027c0add40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:59.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.288+0000 7f0280a00640 1 --2- 172.21.15.90:0/2674679239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f027c06e510 0x7f027c15c750 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:59.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.288+0000 7f0280a00640 1 --2- 172.21.15.90:0/2674679239 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0260004800 0x7f027c148ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:59.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.288+0000 7f0280a00640 1 -- 172.21.15.90:0/2674679239 >> 172.21.15.90:0/2674679239 conn(0x7f027c06b320 msgr2=0x7f027c0723f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:59.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.288+0000 7f0280a00640 1 -- 172.21.15.90:0/2674679239 shutdown_connections 2024-04-25T01:18:59.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.288+0000 7f0280a00640 1 -- 172.21.15.90:0/2674679239 wait complete. 2024-04-25T01:18:59.289 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-04-25T01:18:57.873941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":242816,"kb_used_data":18328,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754731904,"statfs":{"total":773094113280,"available":772845469696,"internally_reserved":0,"allocated":18767872,"data_stored":16038532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":229756465},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6322156,"num_objects":-15,"num_object_clones":0,"num_object_copies":-30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-15,"num_whiteouts":0,"num_read":-53,"num_read_kb":-39,"num_write":-50,"num_write_kb":-6193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001549"},"pg_stats":[{"pgid":"1.0","version":"18'117","reported_seq":226,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-25T01:18:54.007929+0000","last_change":"2024-04-25T01:16:17.466569+0000","last_active":"2024-04-25T01:18:54.007929+0000","last_peered":"2024-04-25T01:18:54.007929+0000","last_clean":"2024-04-25T01:18:54.007929+0000","last_became_active":"2024-04-25T01:16:17.465687+0000","last_became_peered":"2024-04-25T01:16:17.465687+0000","last_unstale":"2024-04-25T01:18:54.007929+0000","last_undegraded":"2024-04-25T01:18:54.007929+0000","last_fullsized":"2024-04-25T01:18:54.007929+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:16.440559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:16.440559+0000","last_clean_scrub_stamp":"2024-04-25T01:16:16.440559+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:18:17.493555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542179,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":626688,"data_stored":284542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542178,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":614400,"data_stored":278570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542179,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36472,"kb_used_data":5288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335368,"statfs":{"total":96636764160,"available":96599416832,"internally_reserved":0,"allocated":5414912,"data_stored":5069214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542178,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":659456,"data_stored":307982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542178,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33764,"kb_used_data":6744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338076,"statfs":{"total":96636764160,"available":96602189824,"internally_reserved":0,"allocated":6905856,"data_stored":6569534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542180,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":626688,"data_stored":284542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32360,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339480,"statfs":{"total":96636764160,"available":96603627520,"internally_reserved":0,"allocated":1208320,"data_stored":868426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542179,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29668,"kb_used_data":2648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342172,"statfs":{"total":96636764160,"available":96606384128,"internally_reserved":0,"allocated":2711552,"data_stored":2375722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:18:59.289 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-25T01:18:59.289 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:18:59.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.494+0000 7ff1b4a00640 1 Processor -- start 2024-04-25T01:18:59.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.494+0000 7ff1b4a00640 1 -- start start 2024-04-25T01:18:59.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.496+0000 7ff1b4a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff1b0145640 0x7ff1b015ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:59.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.496+0000 7ff1b4a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff1b00b2a20 0x7ff1b00cfc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:59.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.496+0000 7ff1b4a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff1b006cbc0 0x7ff1b00d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:59.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.496+0000 7ff1b4a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7ff1b00cdf70 con 0x7ff1b0145640 2024-04-25T01:18:59.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.496+0000 7ff1b4a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff1b015d360 con 0x7ff1b006cbc0 2024-04-25T01:18:59.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.496+0000 7ff1b4a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff1b0145a20 con 0x7ff1b00b2a20 2024-04-25T01:18:59.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.496+0000 7ff1ade00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff1b0145640 0x7ff1b015ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:59.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.496+0000 7ff1ade00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff1b0145640 0x7ff1b015ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45262/0 (socket says 172.21.15.90:45262) 2024-04-25T01:18:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.496+0000 7ff1ade00640 1 -- 172.21.15.90:0/3129081762 learned_addr learned my addr 172.21.15.90:0/3129081762 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.496+0000 7ff1ade00640 1 -- 172.21.15.90:0/3129081762 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff1b00b2a20 msgr2=0x7ff1b00cfc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.496+0000 7ff1ade00640 1 --2- 172.21.15.90:0/3129081762 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff1b00b2a20 0x7ff1b00cfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.496+0000 7ff1ade00640 1 -- 172.21.15.90:0/3129081762 >> v2:172.21.15.103:3300/0 conn(0x7ff1b006cbc0 msgr2=0x7ff1b00d01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.496+0000 7ff1ade00640 1 --2- 172.21.15.90:0/3129081762 >> v2:172.21.15.103:3300/0 conn(0x7ff1b006cbc0 0x7ff1b00d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.496+0000 7ff1ade00640 1 -- 172.21.15.90:0/3129081762 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1b006d140 con 0x7ff1b0145640 2024-04-25T01:18:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.498+0000 7ff1ade00640 1 --2- 172.21.15.90:0/3129081762 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1b0145640 0x7ff1b015ce20 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7ff1b00ccd80 tx=0x7ff1a002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85029532451a0a6e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.498+0000 7ff1aca00640 1 -- 172.21.15.90:0/3129081762 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a0037070 con 0x7ff1b0145640 2024-04-25T01:18:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.498+0000 7ff1aca00640 1 -- 172.21.15.90:0/3129081762 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1a0032440 con 0x7ff1b0145640 2024-04-25T01:18:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.498+0000 7ff1aca00640 1 -- 172.21.15.90:0/3129081762 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a0032810 con 0x7ff1b0145640 2024-04-25T01:18:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.498+0000 7ff1b4a00640 1 -- 172.21.15.90:0/3129081762 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1b0145640 msgr2=0x7ff1b015ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.498+0000 7ff1b4a00640 1 --2- 172.21.15.90:0/3129081762 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1b0145640 0x7ff1b015ce20 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7ff1b00ccd80 tx=0x7ff1a002a1e0 comp rx=0 tx=0).stop 2024-04-25T01:18:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.499+0000 7ff1b4a00640 1 -- 172.21.15.90:0/3129081762 shutdown_connections 2024-04-25T01:18:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.499+0000 7ff1b4a00640 1 --2- 172.21.15.90:0/3129081762 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff1b00b2a20 0x7ff1b00cfc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.499+0000 7ff1b4a00640 1 --2- 172.21.15.90:0/3129081762 >> v2:172.21.15.103:3300/0 conn(0x7ff1b006cbc0 0x7ff1b00d01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.499+0000 7ff1b4a00640 1 --2- 172.21.15.90:0/3129081762 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1b0145640 0x7ff1b015ce20 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.499+0000 7ff1b4a00640 1 -- 172.21.15.90:0/3129081762 >> 172.21.15.90:0/3129081762 conn(0x7ff1b006b320 msgr2=0x7ff1b006b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.499+0000 7ff1b4a00640 1 -- 172.21.15.90:0/3129081762 shutdown_connections 2024-04-25T01:18:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.499+0000 7ff1b4a00640 1 -- 172.21.15.90:0/3129081762 wait complete. 2024-04-25T01:18:59.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.500+0000 7ff1b4a00640 1 Processor -- start 2024-04-25T01:18:59.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.501+0000 7ff1b4a00640 1 -- start start 2024-04-25T01:18:59.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.501+0000 7ff1b4a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1b006cbc0 0x7ff1b01f7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:59.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.502+0000 7ff1b4a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1b00b2a20 0x7ff1b0158090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:59.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.502+0000 7ff1b4a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff1b0145640 0x7ff1b01585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:59.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.502+0000 7ff1b4a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b006d9e0 con 0x7ff1b006cbc0 2024-04-25T01:18:59.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.502+0000 7ff1b4a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b006d700 con 0x7ff1b00b2a20 2024-04-25T01:18:59.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.502+0000 7ff1b4a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff1b015f9c0 con 0x7ff1b0145640 2024-04-25T01:18:59.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.502+0000 7ff1ade00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1b006cbc0 0x7ff1b01f7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:59.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.502+0000 7ff1ade00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1b006cbc0 0x7ff1b01f7e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45270/0 (socket says 172.21.15.90:45270) 2024-04-25T01:18:59.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.502+0000 7ff1ade00640 1 -- 172.21.15.90:0/170826403 learned_addr learned my addr 172.21.15.90:0/170826403 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:59.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.502+0000 7ff1ade00640 1 -- 172.21.15.90:0/170826403 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff1b0145640 msgr2=0x7ff1b01585d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:59.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.502+0000 7ff1ade00640 1 --2- 172.21.15.90:0/170826403 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff1b0145640 0x7ff1b01585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:59.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.502+0000 7ff1ade00640 1 -- 172.21.15.90:0/170826403 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1b00b2a20 msgr2=0x7ff1b0158090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:59.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.502+0000 7ff1ade00640 1 --2- 172.21.15.90:0/170826403 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1b00b2a20 0x7ff1b0158090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:59.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.502+0000 7ff1ade00640 1 -- 172.21.15.90:0/170826403 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1b0071b40 con 0x7ff1b006cbc0 2024-04-25T01:18:59.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.503+0000 7ff1ade00640 1 --2- 172.21.15.90:0/170826403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1b006cbc0 0x7ff1b01f7e50 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0000c00 tx=0x7ff1a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:59.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.503+0000 7ff1a6000640 1 -- 172.21.15.90:0/170826403 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a0041070 con 0x7ff1b006cbc0 2024-04-25T01:18:59.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.503+0000 7ff1a6000640 1 -- 172.21.15.90:0/170826403 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1a003c870 con 0x7ff1b006cbc0 2024-04-25T01:18:59.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.503+0000 7ff1b4a00640 1 -- 172.21.15.90:0/170826403 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1b0071860 con 0x7ff1b006cbc0 2024-04-25T01:18:59.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.503+0000 7ff1a6000640 1 -- 172.21.15.90:0/170826403 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a0037040 con 0x7ff1b006cbc0 2024-04-25T01:18:59.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.503+0000 7ff1b4a00640 1 -- 172.21.15.90:0/170826403 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1b0131f30 con 0x7ff1b006cbc0 2024-04-25T01:18:59.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.504+0000 7ff1b4a00640 1 -- 172.21.15.90:0/170826403 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1b0068190 con 0x7ff1b006cbc0 2024-04-25T01:18:59.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.505+0000 7ff1a6000640 1 -- 172.21.15.90:0/170826403 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff1a0044070 con 0x7ff1b006cbc0 2024-04-25T01:18:59.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.506+0000 7ff1a6000640 1 --2- 172.21.15.90:0/170826403 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff18c06be30 0x7ff18c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:59.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.506+0000 7ff1a6000640 1 -- 172.21.15.90:0/170826403 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5421+0+0 (secure 0 0 0) 0x7ff1a00ab900 con 0x7ff1b006cbc0 2024-04-25T01:18:59.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.507+0000 7ff1ad400640 1 --2- 172.21.15.90:0/170826403 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff18c06be30 0x7ff18c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:59.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.507+0000 7ff1ad400640 1 --2- 172.21.15.90:0/170826403 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff18c06be30 0x7ff18c06e2f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0072620 tx=0x7ff198006c90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:59.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.511+0000 7ff1a6000640 1 -- 172.21.15.90:0/170826403 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff1a007a3d0 con 0x7ff1b006cbc0 2024-04-25T01:18:59.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.704+0000 7ff1b4a00640 1 -- 172.21.15.90:0/170826403 --> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff1b015bfe0 con 0x7ff18c06be30 2024-04-25T01:18:59.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.705+0000 7ff1a6000640 1 -- 172.21.15.90:0/170826403 <== mgr.4104 v2:172.21.15.90:6832/2198086733 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13508 (secure 0 0 0) 0x7ff1b015bfe0 con 0x7ff18c06be30 2024-04-25T01:18:59.688 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:18:59.688 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-25T01:18:59.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.708+0000 7ff1b4a00640 1 -- 172.21.15.90:0/170826403 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff18c06be30 msgr2=0x7ff18c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:59.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.708+0000 7ff1b4a00640 1 --2- 172.21.15.90:0/170826403 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff18c06be30 0x7ff18c06e2f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0072620 tx=0x7ff198006c90 comp rx=0 tx=0).stop 2024-04-25T01:18:59.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.708+0000 7ff1b4a00640 1 -- 172.21.15.90:0/170826403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1b006cbc0 msgr2=0x7ff1b01f7e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:59.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.708+0000 7ff1b4a00640 1 --2- 172.21.15.90:0/170826403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1b006cbc0 0x7ff1b01f7e50 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0000c00 tx=0x7ff1a00046c0 comp rx=0 tx=0).stop 2024-04-25T01:18:59.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.708+0000 7ff1b4a00640 1 -- 172.21.15.90:0/170826403 shutdown_connections 2024-04-25T01:18:59.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.708+0000 7ff1b4a00640 1 --2- 172.21.15.90:0/170826403 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff18c06be30 0x7ff18c06e2f0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:59.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.708+0000 7ff1b4a00640 1 --2- 172.21.15.90:0/170826403 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff1b0145640 0x7ff1b01585d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:59.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.708+0000 7ff1b4a00640 1 --2- 172.21.15.90:0/170826403 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1b00b2a20 0x7ff1b0158090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:59.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.708+0000 7ff1b4a00640 1 --2- 172.21.15.90:0/170826403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1b006cbc0 0x7ff1b01f7e50 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:59.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.708+0000 7ff1b4a00640 1 -- 172.21.15.90:0/170826403 >> 172.21.15.90:0/170826403 conn(0x7ff1b006b320 msgr2=0x7ff1b00cedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:59.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.708+0000 7ff1b4a00640 1 -- 172.21.15.90:0/170826403 shutdown_connections 2024-04-25T01:18:59.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.708+0000 7ff1b4a00640 1 -- 172.21.15.90:0/170826403 wait complete. 2024-04-25T01:18:59.709 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-04-25T01:18:57.873941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":242816,"kb_used_data":18328,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754731904,"statfs":{"total":773094113280,"available":772845469696,"internally_reserved":0,"allocated":18767872,"data_stored":16038532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":229756465},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6322156,"num_objects":-15,"num_object_clones":0,"num_object_copies":-30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-15,"num_whiteouts":0,"num_read":-53,"num_read_kb":-39,"num_write":-50,"num_write_kb":-6193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001549"},"pg_stats":[{"pgid":"1.0","version":"18'117","reported_seq":226,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-25T01:18:54.007929+0000","last_change":"2024-04-25T01:16:17.466569+0000","last_active":"2024-04-25T01:18:54.007929+0000","last_peered":"2024-04-25T01:18:54.007929+0000","last_clean":"2024-04-25T01:18:54.007929+0000","last_became_active":"2024-04-25T01:16:17.465687+0000","last_became_peered":"2024-04-25T01:16:17.465687+0000","last_unstale":"2024-04-25T01:18:54.007929+0000","last_undegraded":"2024-04-25T01:18:54.007929+0000","last_fullsized":"2024-04-25T01:18:54.007929+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:16:16.440559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:16:16.440559+0000","last_clean_scrub_stamp":"2024-04-25T01:16:16.440559+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:18:17.493555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542179,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":626688,"data_stored":284542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542178,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":614400,"data_stored":278570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542179,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36472,"kb_used_data":5288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335368,"statfs":{"total":96636764160,"available":96599416832,"internally_reserved":0,"allocated":5414912,"data_stored":5069214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542178,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":659456,"data_stored":307982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542178,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33764,"kb_used_data":6744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338076,"statfs":{"total":96636764160,"available":96602189824,"internally_reserved":0,"allocated":6905856,"data_stored":6569534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542180,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":626688,"data_stored":284542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32360,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339480,"statfs":{"total":96636764160,"available":96603627520,"internally_reserved":0,"allocated":1208320,"data_stored":868426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542179,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29668,"kb_used_data":2648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342172,"statfs":{"total":96636764160,"available":96606384128,"internally_reserved":0,"allocated":2711552,"data_stored":2375722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:18:59.710 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:18:59.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.928+0000 7fb6ec200640 1 Processor -- start 2024-04-25T01:18:59.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.928+0000 7fb6ec200640 1 -- start start 2024-04-25T01:18:59.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.929+0000 7fb6ec200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb6e41445e0 0x7fb6e40cfc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:59.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.930+0000 7fb6ec200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb6e4071320 0x7fb6e40d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:59.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.930+0000 7fb6ec200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6e41449c0 con 0x7fb6e40728c0 2024-04-25T01:18:59.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.930+0000 7fb6ec200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fb6e4072430 con 0x7fb6e4071320 2024-04-25T01:18:59.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.930+0000 7fb6ec200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb6e40b2d70 con 0x7fb6e41445e0 2024-04-25T01:18:59.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.930+0000 7fb6ea200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb6e4071320 0x7fb6e40d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:59.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.930+0000 7fb6ea200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb6e4071320 0x7fb6e40d01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45284/0 (socket says 172.21.15.90:45284) 2024-04-25T01:18:59.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.930+0000 7fb6ea200640 1 -- 172.21.15.90:0/4183106210 learned_addr learned my addr 172.21.15.90:0/4183106210 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:59.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.930+0000 7fb6e9800640 1 --2- 172.21.15.90:0/4183106210 >> v2:172.21.15.103:3300/0 conn(0x7fb6e41445e0 0x7fb6e40cfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:59.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.930+0000 7fb6ea200640 1 -- 172.21.15.90:0/4183106210 >> v2:172.21.15.103:3300/0 conn(0x7fb6e41445e0 msgr2=0x7fb6e40cfc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:59.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.930+0000 7fb6ea200640 1 --2- 172.21.15.90:0/4183106210 >> v2:172.21.15.103:3300/0 conn(0x7fb6e41445e0 0x7fb6e40cfc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:59.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.930+0000 7fb6ea200640 1 -- 172.21.15.90:0/4183106210 >> v1:172.21.15.103:6789/0 conn(0x7fb6e40728c0 legacy=0x7fb6e4070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:18:59.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.931+0000 7fb6ea200640 1 -- 172.21.15.90:0/4183106210 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6e4071820 con 0x7fb6e4071320 2024-04-25T01:18:59.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.931+0000 7fb6e9800640 1 --2- 172.21.15.90:0/4183106210 >> v2:172.21.15.103:3300/0 conn(0x7fb6e41445e0 0x7fb6e40cfc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:18:59.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.932+0000 7fb6ea200640 1 --2- 172.21.15.90:0/4183106210 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6e4071320 0x7fb6e40d01d0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb6e40d08a0 tx=0x7fb6dc02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1777867ef29cfe2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:59.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.932+0000 7fb6e3e00640 1 -- 172.21.15.90:0/4183106210 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc036070 con 0x7fb6e4071320 2024-04-25T01:18:59.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.932+0000 7fb6e3e00640 1 -- 172.21.15.90:0/4183106210 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6dc031600 con 0x7fb6e4071320 2024-04-25T01:18:59.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.932+0000 7fb6e3e00640 1 -- 172.21.15.90:0/4183106210 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc0319d0 con 0x7fb6e4071320 2024-04-25T01:18:59.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.933+0000 7fb6ec200640 1 -- 172.21.15.90:0/4183106210 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6e4071320 msgr2=0x7fb6e40d01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:59.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.933+0000 7fb6ec200640 1 --2- 172.21.15.90:0/4183106210 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6e4071320 0x7fb6e40d01d0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb6e40d08a0 tx=0x7fb6dc02a6f0 comp rx=0 tx=0).stop 2024-04-25T01:18:59.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.933+0000 7fb6ec200640 1 -- 172.21.15.90:0/4183106210 shutdown_connections 2024-04-25T01:18:59.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.933+0000 7fb6ec200640 1 --2- 172.21.15.90:0/4183106210 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6e4071320 0x7fb6e40d01d0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:59.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.933+0000 7fb6ec200640 1 --2- 172.21.15.90:0/4183106210 >> v2:172.21.15.103:3300/0 conn(0x7fb6e41445e0 0x7fb6e40cfc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:59.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.933+0000 7fb6ec200640 1 -- 172.21.15.90:0/4183106210 >> 172.21.15.90:0/4183106210 conn(0x7fb6e406b320 msgr2=0x7fb6e406b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:18:59.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.933+0000 7fb6ec200640 1 -- 172.21.15.90:0/4183106210 shutdown_connections 2024-04-25T01:18:59.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.933+0000 7fb6ec200640 1 -- 172.21.15.90:0/4183106210 wait complete. 2024-04-25T01:18:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.934+0000 7fb6ec200640 1 Processor -- start 2024-04-25T01:18:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.935+0000 7fb6ec200640 1 -- start start 2024-04-25T01:18:59.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.936+0000 7fb6ec200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6e4071320 0x7fb6e41397a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:59.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.936+0000 7fb6ec200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6e40728c0 0x7fb6e413acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:59.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.936+0000 7fb6ec200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6e41445e0 0x7fb6e406cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:59.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.936+0000 7fb6ec200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e40720c0 con 0x7fb6e41445e0 2024-04-25T01:18:59.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.936+0000 7fb6ec200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e4071de0 con 0x7fb6e40728c0 2024-04-25T01:18:59.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.936+0000 7fb6ec200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb6e40cced0 con 0x7fb6e4071320 2024-04-25T01:18:59.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.936+0000 7fb6ea200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6e41445e0 0x7fb6e406cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:59.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.936+0000 7fb6ea200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6e41445e0 0x7fb6e406cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45298/0 (socket says 172.21.15.90:45298) 2024-04-25T01:18:59.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.936+0000 7fb6ea200640 1 -- 172.21.15.90:0/679202614 learned_addr learned my addr 172.21.15.90:0/679202614 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:18:59.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.936+0000 7fb6e9800640 1 --2- 172.21.15.90:0/679202614 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6e4071320 0x7fb6e41397a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:59.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.936+0000 7fb6ea200640 1 -- 172.21.15.90:0/679202614 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6e4071320 msgr2=0x7fb6e41397a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:59.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.936+0000 7fb6ea200640 1 --2- 172.21.15.90:0/679202614 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6e4071320 0x7fb6e41397a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:59.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.936+0000 7fb6ea200640 1 -- 172.21.15.90:0/679202614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6e40728c0 msgr2=0x7fb6e413acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:18:59.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.936+0000 7fb6ea200640 1 --2- 172.21.15.90:0/679202614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6e40728c0 0x7fb6e413acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:18:59.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.936+0000 7fb6ea200640 1 -- 172.21.15.90:0/679202614 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6e415bc00 con 0x7fb6e41445e0 2024-04-25T01:18:59.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.937+0000 7fb6ea200640 1 --2- 172.21.15.90:0/679202614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6e41445e0 0x7fb6e406cbc0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc009f40 tx=0x7fb6dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:59.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.937+0000 7fb6e1600640 1 -- 172.21.15.90:0/679202614 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc036050 con 0x7fb6e41445e0 2024-04-25T01:18:59.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.937+0000 7fb6e1600640 1 -- 172.21.15.90:0/679202614 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6dc042070 con 0x7fb6e41445e0 2024-04-25T01:18:59.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.937+0000 7fb6e1600640 1 -- 172.21.15.90:0/679202614 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc009510 con 0x7fb6e41445e0 2024-04-25T01:18:59.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.937+0000 7fb6ec200640 1 -- 172.21.15.90:0/679202614 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6e415bea0 con 0x7fb6e41445e0 2024-04-25T01:18:59.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.937+0000 7fb6ec200640 1 -- 172.21.15.90:0/679202614 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6e415c460 con 0x7fb6e41445e0 2024-04-25T01:18:59.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.939+0000 7fb6ec200640 1 -- 172.21.15.90:0/679202614 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6e4068190 con 0x7fb6e41445e0 2024-04-25T01:18:59.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.940+0000 7fb6e1600640 1 -- 172.21.15.90:0/679202614 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb6dc007bd0 con 0x7fb6e41445e0 2024-04-25T01:18:59.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.940+0000 7fb6e1600640 1 --2- 172.21.15.90:0/679202614 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb6c406be30 0x7fb6c406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:18:59.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.941+0000 7fb6e1600640 1 -- 172.21.15.90:0/679202614 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5421+0+0 (secure 0 0 0) 0x7fb6dc0abdd0 con 0x7fb6e41445e0 2024-04-25T01:18:59.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.941+0000 7fb6e9800640 1 --2- 172.21.15.90:0/679202614 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb6c406be30 0x7fb6c406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:18:59.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.942+0000 7fb6e9800640 1 --2- 172.21.15.90:0/679202614 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb6c406be30 0x7fb6c406e2f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb6d800a950 tx=0x7fb6d8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:18:59.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:18:59.945+0000 7fb6e1600640 1 -- 172.21.15.90:0/679202614 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb6dc07a810 con 0x7fb6e41445e0 2024-04-25T01:19:00.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:00.148+0000 7fb6ec200640 1 -- 172.21.15.90:0/679202614 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb6e406eab0 con 0x7fb6e41445e0 2024-04-25T01:19:00.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:00.150+0000 7fb6e1600640 1 -- 172.21.15.90:0/679202614 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) ==== 74+0+13387 (secure 0 0 0) 0x7fb6dc07a110 con 0x7fb6e41445e0 2024-04-25T01:19:00.133 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:19:00.133 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":54,"fsid":"e13fd5b3-cb9d-468a-93cf-dd1a7e0b6e21","created":"2024-04-25T01:16:02.799578+0000","modified":"2024-04-25T01:18:55.070309+0000","last_up_change":"2024-04-25T01:16:14.433294+0000","last_in_change":"2024-04-25T01:16:06.250204+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:16:15.894126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ad7c9e8-f014-464d-9354-a71afeaf587c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6818","nonce":2625481430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6824","nonce":2625481430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6829","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6831","nonce":2625481430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2625481430},{"type":"v1","addr":"172.21.15.90:6828","nonce":2625481430}]},"public_addr":"172.21.15.90:6818/2625481430","cluster_addr":"172.21.15.90:6824/2625481430","heartbeat_back_addr":"172.21.15.90:6831/2625481430","heartbeat_front_addr":"172.21.15.90:6828/2625481430","state":["exists","up"]},{"osd":1,"uuid":"eae07c14-363b-4f7f-8682-d1bd39111e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6801","nonce":2685473653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6803","nonce":2685473653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6807","nonce":2685473653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2685473653},{"type":"v1","addr":"172.21.15.90:6805","nonce":2685473653}]},"public_addr":"172.21.15.90:6801/2685473653","cluster_addr":"172.21.15.90:6803/2685473653","heartbeat_back_addr":"172.21.15.90:6807/2685473653","heartbeat_front_addr":"172.21.15.90:6805/2685473653","state":["exists","up"]},{"osd":2,"uuid":"84b4e1c1-f54d-4e3a-aa55-b92747e5af15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6809","nonce":906792737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6811","nonce":906792737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6821","nonce":906792737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":906792737},{"type":"v1","addr":"172.21.15.90:6815","nonce":906792737}]},"public_addr":"172.21.15.90:6809/906792737","cluster_addr":"172.21.15.90:6811/906792737","heartbeat_back_addr":"172.21.15.90:6821/906792737","heartbeat_front_addr":"172.21.15.90:6815/906792737","state":["exists","up"]},{"osd":3,"uuid":"ed9a6b3e-a131-4d9b-bb1a-704a83612ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6814","nonce":8683104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6820","nonce":8683104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6830","nonce":8683104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":8683104},{"type":"v1","addr":"172.21.15.90:6825","nonce":8683104}]},"public_addr":"172.21.15.90:6814/8683104","cluster_addr":"172.21.15.90:6820/8683104","heartbeat_back_addr":"172.21.15.90:6830/8683104","heartbeat_front_addr":"172.21.15.90:6825/8683104","state":["exists","up"]},{"osd":4,"uuid":"6e51d4c4-c19d-443f-b453-0aa2487b2b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6821","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6822","nonce":1971061680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6825","nonce":1971061680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6829","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6831","nonce":1971061680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6827","nonce":1971061680},{"type":"v1","addr":"172.21.15.103:6828","nonce":1971061680}]},"public_addr":"172.21.15.103:6822/1971061680","cluster_addr":"172.21.15.103:6825/1971061680","heartbeat_back_addr":"172.21.15.103:6831/1971061680","heartbeat_front_addr":"172.21.15.103:6828/1971061680","state":["exists","up"]},{"osd":5,"uuid":"a948695d-848b-413e-91d3-3f4efa56e573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6801","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6803","nonce":1348846168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6807","nonce":1348846168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6815","nonce":1348846168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":1348846168},{"type":"v1","addr":"172.21.15.103:6811","nonce":1348846168}]},"public_addr":"172.21.15.103:6803/1348846168","cluster_addr":"172.21.15.103:6807/1348846168","heartbeat_back_addr":"172.21.15.103:6815/1348846168","heartbeat_front_addr":"172.21.15.103:6811/1348846168","state":["exists","up"]},{"osd":6,"uuid":"19f8c647-a668-42a2-ac95-c4020fd212a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6817","nonce":2008388906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6819","nonce":2008388906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6830","nonce":2008388906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2008388906},{"type":"v1","addr":"172.21.15.103:6823","nonce":2008388906}]},"public_addr":"172.21.15.103:6817/2008388906","cluster_addr":"172.21.15.103:6819/2008388906","heartbeat_back_addr":"172.21.15.103:6830/2008388906","heartbeat_front_addr":"172.21.15.103:6823/2008388906","state":["exists","up"]},{"osd":7,"uuid":"847a35bd-7317-46c1-a9df-2a0832bd1056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6802","nonce":1145679154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6806","nonce":1145679154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6814","nonce":1145679154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1145679154},{"type":"v1","addr":"172.21.15.103:6810","nonce":1145679154}]},"public_addr":"172.21.15.103:6802/1145679154","cluster_addr":"172.21.15.103:6806/1145679154","heartbeat_back_addr":"172.21.15.103:6814/1145679154","heartbeat_front_addr":"172.21.15.103:6810/1145679154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.649223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.541733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.665139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.638577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.710972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.864110+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.894637+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:16:11.566218+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:6834/2850373997":"2024-04-26T01:18:51.996323+0000","172.21.15.103:6832/2850373997":"2024-04-26T01:18:51.996323+0000","172.21.15.90:6837/431390375":"2024-04-26T01:18:15.602554+0000","172.21.15.90:6836/431390375":"2024-04-26T01:18:15.602554+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:19:00.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:00.152+0000 7fb6ec200640 1 -- 172.21.15.90:0/679202614 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb6c406be30 msgr2=0x7fb6c406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:00.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:00.152+0000 7fb6ec200640 1 --2- 172.21.15.90:0/679202614 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb6c406be30 0x7fb6c406e2f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb6d800a950 tx=0x7fb6d8008040 comp rx=0 tx=0).stop 2024-04-25T01:19:00.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:00.152+0000 7fb6ec200640 1 -- 172.21.15.90:0/679202614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6e41445e0 msgr2=0x7fb6e406cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:00.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:00.152+0000 7fb6ec200640 1 --2- 172.21.15.90:0/679202614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6e41445e0 0x7fb6e406cbc0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc009f40 tx=0x7fb6dc004a40 comp rx=0 tx=0).stop 2024-04-25T01:19:00.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:00.153+0000 7fb6ec200640 1 -- 172.21.15.90:0/679202614 shutdown_connections 2024-04-25T01:19:00.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:00.153+0000 7fb6ec200640 1 --2- 172.21.15.90:0/679202614 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb6c406be30 0x7fb6c406e2f0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:00.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:00.153+0000 7fb6ec200640 1 --2- 172.21.15.90:0/679202614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6e41445e0 0x7fb6e406cbc0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:00.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:00.153+0000 7fb6ec200640 1 --2- 172.21.15.90:0/679202614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6e40728c0 0x7fb6e413acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:00.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:00.153+0000 7fb6ec200640 1 --2- 172.21.15.90:0/679202614 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb6e4071320 0x7fb6e41397a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:00.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:00.153+0000 7fb6ec200640 1 -- 172.21.15.90:0/679202614 >> 172.21.15.90:0/679202614 conn(0x7fb6e406b320 msgr2=0x7fb6e40cfc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:00.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:00.153+0000 7fb6ec200640 1 -- 172.21.15.90:0/679202614 shutdown_connections 2024-04-25T01:19:00.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:00.153+0000 7fb6ec200640 1 -- 172.21.15.90:0/679202614 wait complete. 2024-04-25T01:19:01.155 INFO:tasks.ceph:Scrubbing osd.0 2024-04-25T01:19:01.155 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-25T01:19:01.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.385+0000 7ff2eb200640 1 Processor -- start 2024-04-25T01:19:01.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.386+0000 7ff2eb200640 1 -- start start 2024-04-25T01:19:01.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.387+0000 7ff2eb200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2ec0aee80 0x7ff2ec06e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:01.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.387+0000 7ff2eb200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff2ec166560 0x7ff2ec0b2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:01.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.387+0000 7ff2eb200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2ec071150 con 0x7ff2ec070c00 2024-04-25T01:19:01.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.387+0000 7ff2eb200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff2ec166cb0 con 0x7ff2ec166560 2024-04-25T01:19:01.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.387+0000 7ff2eb200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff2ec071d60 con 0x7ff2ec0aee80 2024-04-25T01:19:01.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.388+0000 7ff2ea800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff2ec166560 0x7ff2ec0b2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:01.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.388+0000 7ff2e9e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2ec0aee80 0x7ff2ec06e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:01.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.388+0000 7ff2ea800640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff2ec166560 0x7ff2ec0b2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.90:42598/0 (socket says 172.21.15.90:42598) 2024-04-25T01:19:01.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.388+0000 7ff2ea800640 1 -- 172.21.15.90:0/918821063 learned_addr learned my addr 172.21.15.90:0/918821063 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:19:01.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.389+0000 7ff2ea800640 1 -- 172.21.15.90:0/918821063 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2ec0aee80 msgr2=0x7ff2ec06e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:01.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.389+0000 7ff2ea800640 1 --2- 172.21.15.90:0/918821063 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2ec0aee80 0x7ff2ec06e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:01.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.389+0000 7ff2ea800640 1 -- 172.21.15.90:0/918821063 >> v1:172.21.15.103:6789/0 conn(0x7ff2ec070c00 legacy=0x7ff2ec165e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:01.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.389+0000 7ff2ea800640 1 -- 172.21.15.90:0/918821063 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2ec071430 con 0x7ff2ec166560 2024-04-25T01:19:01.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.390+0000 7ff2ea800640 1 --2- 172.21.15.90:0/918821063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2ec166560 0x7ff2ec0b2840 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4009e80 tx=0x7ff2e402a740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=846c7b26ca62a001 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:01.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.391+0000 7ff2e9400640 1 -- 172.21.15.90:0/918821063 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e4036070 con 0x7ff2ec166560 2024-04-25T01:19:01.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.391+0000 7ff2e9400640 1 -- 172.21.15.90:0/918821063 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2e4031600 con 0x7ff2ec166560 2024-04-25T01:19:01.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.391+0000 7ff2e9400640 1 -- 172.21.15.90:0/918821063 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e40319d0 con 0x7ff2ec166560 2024-04-25T01:19:01.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.391+0000 7ff2eb200640 1 -- 172.21.15.90:0/918821063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2ec166560 msgr2=0x7ff2ec0b2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:01.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.391+0000 7ff2eb200640 1 --2- 172.21.15.90:0/918821063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2ec166560 0x7ff2ec0b2840 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4009e80 tx=0x7ff2e402a740 comp rx=0 tx=0).stop 2024-04-25T01:19:01.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.392+0000 7ff2eb200640 1 -- 172.21.15.90:0/918821063 shutdown_connections 2024-04-25T01:19:01.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.392+0000 7ff2eb200640 1 --2- 172.21.15.90:0/918821063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2ec166560 0x7ff2ec0b2840 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:01.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.392+0000 7ff2eb200640 1 --2- 172.21.15.90:0/918821063 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2ec0aee80 0x7ff2ec06e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:01.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.392+0000 7ff2eb200640 1 -- 172.21.15.90:0/918821063 >> 172.21.15.90:0/918821063 conn(0x7ff2ec0b1460 msgr2=0x7ff2ec0b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:01.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.392+0000 7ff2eb200640 1 -- 172.21.15.90:0/918821063 shutdown_connections 2024-04-25T01:19:01.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.392+0000 7ff2eb200640 1 -- 172.21.15.90:0/918821063 wait complete. 2024-04-25T01:19:01.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.393+0000 7ff2eb200640 1 Processor -- start 2024-04-25T01:19:01.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.394+0000 7ff2eb200640 1 -- start start 2024-04-25T01:19:01.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.394+0000 7ff2eb200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2ec070c00 0x7ff2ec15da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:01.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.394+0000 7ff2eb200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2ec0aee80 0x7ff2ec0cf350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:01.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.394+0000 7ff2eb200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2ec166560 0x7ff2ec1fbf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:01.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.394+0000 7ff2eb200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff2ec06ebe0 con 0x7ff2ec070c00 2024-04-25T01:19:01.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.394+0000 7ff2eb200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff2ec0719f0 con 0x7ff2ec166560 2024-04-25T01:19:01.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.394+0000 7ff2eb200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff2ec0b2d80 con 0x7ff2ec0aee80 2024-04-25T01:19:01.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.395+0000 7ff2e9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2ec070c00 0x7ff2ec15da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:01.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.395+0000 7ff2e9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2ec070c00 0x7ff2ec15da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45304/0 (socket says 172.21.15.90:45304) 2024-04-25T01:19:01.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.395+0000 7ff2e9e00640 1 -- 172.21.15.90:0/3895677052 learned_addr learned my addr 172.21.15.90:0/3895677052 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:19:01.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.395+0000 7ff2e3e00640 1 --2- 172.21.15.90:0/3895677052 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2ec0aee80 0x7ff2ec0cf350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:01.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.395+0000 7ff2e9e00640 1 -- 172.21.15.90:0/3895677052 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2ec0aee80 msgr2=0x7ff2ec0cf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:01.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.395+0000 7ff2ea800640 1 --2- 172.21.15.90:0/3895677052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2ec166560 0x7ff2ec1fbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:01.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.395+0000 7ff2e9e00640 1 --2- 172.21.15.90:0/3895677052 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff2ec0aee80 0x7ff2ec0cf350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:01.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.395+0000 7ff2e9e00640 1 -- 172.21.15.90:0/3895677052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2ec166560 msgr2=0x7ff2ec1fbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:01.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.395+0000 7ff2e9e00640 1 --2- 172.21.15.90:0/3895677052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2ec166560 0x7ff2ec1fbf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:01.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.395+0000 7ff2e9e00640 1 -- 172.21.15.90:0/3895677052 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2ec1f8150 con 0x7ff2ec070c00 2024-04-25T01:19:01.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.396+0000 7ff2e9e00640 1 --2- 172.21.15.90:0/3895677052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2ec070c00 0x7ff2ec15da60 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc00e9b0 tx=0x7ff2dc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:01.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.396+0000 7ff2e2000640 1 -- 172.21.15.90:0/3895677052 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc017070 con 0x7ff2ec070c00 2024-04-25T01:19:01.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.396+0000 7ff2e2000640 1 -- 172.21.15.90:0/3895677052 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2dc00c9a0 con 0x7ff2ec070c00 2024-04-25T01:19:01.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.396+0000 7ff2e2000640 1 -- 172.21.15.90:0/3895677052 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc00cd90 con 0x7ff2ec070c00 2024-04-25T01:19:01.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.396+0000 7ff2eb200640 1 -- 172.21.15.90:0/3895677052 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2ec1f8430 con 0x7ff2ec070c00 2024-04-25T01:19:01.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.396+0000 7ff2eb200640 1 -- 172.21.15.90:0/3895677052 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2ec1f89f0 con 0x7ff2ec070c00 2024-04-25T01:19:01.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.397+0000 7ff2eb200640 1 -- 172.21.15.90:0/3895677052 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff2ec073050 con 0x7ff2ec070c00 2024-04-25T01:19:01.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.398+0000 7ff2e2000640 1 -- 172.21.15.90:0/3895677052 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff2dc01a070 con 0x7ff2ec070c00 2024-04-25T01:19:01.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.399+0000 7ff2e2000640 1 --2- 172.21.15.90:0/3895677052 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff2c806be30 0x7ff2c806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:01.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.399+0000 7ff2e2000640 1 -- 172.21.15.90:0/3895677052 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5421+0+0 (secure 0 0 0) 0x7ff2dc08b150 con 0x7ff2ec070c00 2024-04-25T01:19:01.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.399+0000 7ff2e3e00640 1 --2- 172.21.15.90:0/3895677052 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff2c806be30 0x7ff2c806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:01.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.399+0000 7ff2e2000640 1 --2- 172.21.15.90:0/3895677052 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x7ff2c8072370 0x7ff2c80747b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:01.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.399+0000 7ff2e2000640 1 -- 172.21.15.90:0/3895677052 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff2dc01fe50 con 0x7ff2c8072370 2024-04-25T01:19:01.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.399+0000 7ff2e2000640 1 -- 172.21.15.90:0/3895677052 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7ff2dc092050 con 0x7ff2ec070c00 2024-04-25T01:19:01.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.400+0000 7ff2ea800640 1 --2- 172.21.15.90:0/3895677052 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x7ff2c8072370 0x7ff2c80747b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:01.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.400+0000 7ff2e3e00640 1 --2- 172.21.15.90:0/3895677052 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff2c806be30 0x7ff2c806e2f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec161b80 tx=0x7ff2d4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:01.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.400+0000 7ff2ea800640 1 --2- 172.21.15.90:0/3895677052 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x7ff2c8072370 0x7ff2c80747b0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:01.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.401+0000 7ff2e2000640 1 -- 172.21.15.90:0/3895677052 <== osd.0 v2:172.21.15.90:6816/2625481430 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff2dc01fe50 con 0x7ff2c8072370 2024-04-25T01:19:01.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.415+0000 7ff2eb200640 1 -- 172.21.15.90:0/3895677052 --> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff2ec1f8e40 con 0x7ff2c8072370 2024-04-25T01:19:01.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.416+0000 7ff2e2000640 1 -- 172.21.15.90:0/3895677052 <== osd.0 v2:172.21.15.90:6816/2625481430 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff2ec1f8e40 con 0x7ff2c8072370 2024-04-25T01:19:01.399 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T01:19:01.399 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T01:19:01.399 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T01:19:01.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.416+0000 7ff2eb200640 1 -- 172.21.15.90:0/3895677052 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x7ff2c8072370 msgr2=0x7ff2c80747b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:01.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.416+0000 7ff2eb200640 1 --2- 172.21.15.90:0/3895677052 >> [v2:172.21.15.90:6816/2625481430,v1:172.21.15.90:6818/2625481430] conn(0x7ff2c8072370 0x7ff2c80747b0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:01.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.417+0000 7ff2eb200640 1 -- 172.21.15.90:0/3895677052 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff2c806be30 msgr2=0x7ff2c806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:01.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.417+0000 7ff2eb200640 1 --2- 172.21.15.90:0/3895677052 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff2c806be30 0x7ff2c806e2f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec161b80 tx=0x7ff2d4009040 comp rx=0 tx=0).stop 2024-04-25T01:19:01.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.417+0000 7ff2eb200640 1 -- 172.21.15.90:0/3895677052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2ec070c00 msgr2=0x7ff2ec15da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:01.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.417+0000 7ff2eb200640 1 --2- 172.21.15.90:0/3895677052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2ec070c00 0x7ff2ec15da60 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc00e9b0 tx=0x7ff2dc00ee70 comp rx=0 tx=0).stop 2024-04-25T01:19:01.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.417+0000 7ff2ea800640 1 -- 172.21.15.90:0/3895677052 reap_dead start 2024-04-25T01:19:01.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.417+0000 7ff2eb200640 1 -- 172.21.15.90:0/3895677052 shutdown_connections 2024-04-25T01:19:01.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.417+0000 7ff2eb200640 1 -- 172.21.15.90:0/3895677052 >> 172.21.15.90:0/3895677052 conn(0x7ff2ec0b1460 msgr2=0x7ff2ec161e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:01.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.417+0000 7ff2eb200640 1 -- 172.21.15.90:0/3895677052 shutdown_connections 2024-04-25T01:19:01.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.417+0000 7ff2eb200640 1 -- 172.21.15.90:0/3895677052 wait complete. 2024-04-25T01:19:01.417 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-25T01:19:01.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.636+0000 7f600ec00640 1 Processor -- start 2024-04-25T01:19:01.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.636+0000 7f600ec00640 1 -- start start 2024-04-25T01:19:01.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.638+0000 7f600ec00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6008142570 0x7f600806cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:01.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.638+0000 7f600ec00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f6008070ba0 0x7f600806d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:01.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.638+0000 7f600ec00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6008142950 con 0x7f600806d640 2024-04-25T01:19:01.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.638+0000 7f600ec00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f6008071c40 con 0x7f6008142570 2024-04-25T01:19:01.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.638+0000 7f600ec00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f6008071030 con 0x7f6008070ba0 2024-04-25T01:19:01.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.638+0000 7f6007e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6008142570 0x7f600806cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:01.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.638+0000 7f6007e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6008142570 0x7f600806cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45308/0 (socket says 172.21.15.90:45308) 2024-04-25T01:19:01.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.638+0000 7f6007e00640 1 -- 172.21.15.90:0/1221413269 learned_addr learned my addr 172.21.15.90:0/1221413269 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:19:01.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.638+0000 7f6007400640 1 --2- 172.21.15.90:0/1221413269 >> v2:172.21.15.103:3300/0 conn(0x7f6008070ba0 0x7f600806d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:01.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.639+0000 7f6007e00640 1 -- 172.21.15.90:0/1221413269 >> v2:172.21.15.103:3300/0 conn(0x7f6008070ba0 msgr2=0x7f600806d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:01.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.639+0000 7f6007e00640 1 --2- 172.21.15.90:0/1221413269 >> v2:172.21.15.103:3300/0 conn(0x7f6008070ba0 0x7f600806d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:01.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.639+0000 7f6007e00640 1 -- 172.21.15.90:0/1221413269 >> v1:172.21.15.103:6789/0 conn(0x7f600806d640 legacy=0x7f60080ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:01.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.639+0000 7f6007e00640 1 -- 172.21.15.90:0/1221413269 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6008071310 con 0x7f6008142570 2024-04-25T01:19:01.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.640+0000 7f6007e00640 1 --2- 172.21.15.90:0/1221413269 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6008142570 0x7f600806cbc0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f600806e970 tx=0x7f5ff802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44e25e6bbfc19ad4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:01.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.640+0000 7f6006a00640 1 -- 172.21.15.90:0/1221413269 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff8037070 con 0x7f6008142570 2024-04-25T01:19:01.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.641+0000 7f6006a00640 1 -- 172.21.15.90:0/1221413269 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ff8032440 con 0x7f6008142570 2024-04-25T01:19:01.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.641+0000 7f6006a00640 1 -- 172.21.15.90:0/1221413269 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff8032810 con 0x7f6008142570 2024-04-25T01:19:01.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.641+0000 7f600ec00640 1 -- 172.21.15.90:0/1221413269 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6008142570 msgr2=0x7f600806cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:01.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.641+0000 7f600ec00640 1 --2- 172.21.15.90:0/1221413269 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6008142570 0x7f600806cbc0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f600806e970 tx=0x7f5ff802a230 comp rx=0 tx=0).stop 2024-04-25T01:19:01.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.641+0000 7f600ec00640 1 -- 172.21.15.90:0/1221413269 shutdown_connections 2024-04-25T01:19:01.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.641+0000 7f600ec00640 1 --2- 172.21.15.90:0/1221413269 >> v2:172.21.15.103:3300/0 conn(0x7f6008070ba0 0x7f600806d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:01.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.641+0000 7f600ec00640 1 --2- 172.21.15.90:0/1221413269 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6008142570 0x7f600806cbc0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:01.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.641+0000 7f600ec00640 1 -- 172.21.15.90:0/1221413269 >> 172.21.15.90:0/1221413269 conn(0x7f600806b140 msgr2=0x7f600806b550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:01.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.641+0000 7f600ec00640 1 -- 172.21.15.90:0/1221413269 shutdown_connections 2024-04-25T01:19:01.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.641+0000 7f600ec00640 1 -- 172.21.15.90:0/1221413269 wait complete. 2024-04-25T01:19:01.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.643+0000 7f600ec00640 1 Processor -- start 2024-04-25T01:19:01.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.643+0000 7f600ec00640 1 -- start start 2024-04-25T01:19:01.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.644+0000 7f600ec00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f600806d640 0x7f6008153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:01.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.644+0000 7f600ec00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6008142570 0x7f60080aaaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:01.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.644+0000 7f600ec00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60080ab030 0x7f60080a94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:01.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.644+0000 7f600ec00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6008072c60 con 0x7f6008142570 2024-04-25T01:19:01.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.644+0000 7f600ec00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f60080718d0 con 0x7f600806d640 2024-04-25T01:19:01.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.644+0000 7f600ec00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f6008072de0 con 0x7f60080ab030 2024-04-25T01:19:01.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.645+0000 7f6007400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6008142570 0x7f60080aaaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:01.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.645+0000 7f6007400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6008142570 0x7f60080aaaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45320/0 (socket says 172.21.15.90:45320) 2024-04-25T01:19:01.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.645+0000 7f6007400640 1 -- 172.21.15.90:0/3146911793 learned_addr learned my addr 172.21.15.90:0/3146911793 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:19:01.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.645+0000 7f6007e00640 1 --2- 172.21.15.90:0/3146911793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f600806d640 0x7f6008153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:01.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.645+0000 7f600cc00640 1 --2- 172.21.15.90:0/3146911793 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60080ab030 0x7f60080a94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:01.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.645+0000 7f6007400640 1 -- 172.21.15.90:0/3146911793 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60080ab030 msgr2=0x7f60080a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:01.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.645+0000 7f6007400640 1 --2- 172.21.15.90:0/3146911793 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60080ab030 0x7f60080a94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:01.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.645+0000 7f6007400640 1 -- 172.21.15.90:0/3146911793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f600806d640 msgr2=0x7f6008153700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:01.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.645+0000 7f6007400640 1 --2- 172.21.15.90:0/3146911793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f600806d640 0x7f6008153700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:01.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.645+0000 7f6007400640 1 -- 172.21.15.90:0/3146911793 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6008150f60 con 0x7f6008142570 2024-04-25T01:19:01.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.645+0000 7f6007e00640 1 --2- 172.21.15.90:0/3146911793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f600806d640 0x7f6008153700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:19:01.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.645+0000 7f600cc00640 1 --2- 172.21.15.90:0/3146911793 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60080ab030 0x7f60080a94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:19:01.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.645+0000 7f6007400640 1 --2- 172.21.15.90:0/3146911793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6008142570 0x7f60080aaaf0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f5ff0007f20 tx=0x7f5ff000e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:01.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.646+0000 7f5fffe00640 1 -- 172.21.15.90:0/3146911793 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff0014620 con 0x7f6008142570 2024-04-25T01:19:01.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.646+0000 7f5fffe00640 1 -- 172.21.15.90:0/3146911793 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ff0019070 con 0x7f6008142570 2024-04-25T01:19:01.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.646+0000 7f5fffe00640 1 -- 172.21.15.90:0/3146911793 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff0014a00 con 0x7f6008142570 2024-04-25T01:19:01.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.646+0000 7f600ec00640 1 -- 172.21.15.90:0/3146911793 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60080aa130 con 0x7f6008142570 2024-04-25T01:19:01.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.646+0000 7f600ec00640 1 -- 172.21.15.90:0/3146911793 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60080aa6f0 con 0x7f6008142570 2024-04-25T01:19:01.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.647+0000 7f600ec00640 1 -- 172.21.15.90:0/3146911793 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60080ab440 con 0x7f6008142570 2024-04-25T01:19:01.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.648+0000 7f5fffe00640 1 -- 172.21.15.90:0/3146911793 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5ff0014ba0 con 0x7f6008142570 2024-04-25T01:19:01.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.649+0000 7f5fffe00640 1 --2- 172.21.15.90:0/3146911793 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f5fe406be30 0x7f5fe406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:01.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.649+0000 7f5fffe00640 1 -- 172.21.15.90:0/3146911793 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5421+0+0 (secure 0 0 0) 0x7f5ff008b690 con 0x7f6008142570 2024-04-25T01:19:01.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.649+0000 7f6007e00640 1 --2- 172.21.15.90:0/3146911793 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f5fe406be30 0x7f5fe406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:01.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.650+0000 7f6007e00640 1 --2- 172.21.15.90:0/3146911793 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f5fe406be30 0x7f5fe406e2f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5ff8034000 tx=0x7f5ff802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:01.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.653+0000 7f5fffe00640 1 -- 172.21.15.90:0/3146911793 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f60080ab440 con 0x7f6008142570 2024-04-25T01:19:01.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.850+0000 7f600ec00640 1 -- 172.21.15.90:0/3146911793 --> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f600815b3c0 con 0x7f5fe406be30 2024-04-25T01:19:01.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.850+0000 7f5fffe00640 1 -- 172.21.15.90:0/3146911793 <== mgr.4104 v2:172.21.15.90:6832/2198086733 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f600815b3c0 con 0x7f5fe406be30 2024-04-25T01:19:01.833 INFO:teuthology.orchestra.run.smithi090.stderr:instructed osd(s) 0 to deep-scrub 2024-04-25T01:19:01.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.853+0000 7f600ec00640 1 -- 172.21.15.90:0/3146911793 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f5fe406be30 msgr2=0x7f5fe406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:01.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.853+0000 7f600ec00640 1 --2- 172.21.15.90:0/3146911793 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f5fe406be30 0x7f5fe406e2f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5ff8034000 tx=0x7f5ff802e000 comp rx=0 tx=0).stop 2024-04-25T01:19:01.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.853+0000 7f600ec00640 1 -- 172.21.15.90:0/3146911793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6008142570 msgr2=0x7f60080aaaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:01.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.853+0000 7f600ec00640 1 --2- 172.21.15.90:0/3146911793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6008142570 0x7f60080aaaf0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f5ff0007f20 tx=0x7f5ff000e570 comp rx=0 tx=0).stop 2024-04-25T01:19:01.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.853+0000 7f600ec00640 1 -- 172.21.15.90:0/3146911793 shutdown_connections 2024-04-25T01:19:01.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.853+0000 7f600ec00640 1 --2- 172.21.15.90:0/3146911793 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f5fe406be30 0x7f5fe406e2f0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:01.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.853+0000 7f600ec00640 1 --2- 172.21.15.90:0/3146911793 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f60080ab030 0x7f60080a94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:01.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.853+0000 7f600ec00640 1 --2- 172.21.15.90:0/3146911793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6008142570 0x7f60080aaaf0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:01.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.853+0000 7f600ec00640 1 --2- 172.21.15.90:0/3146911793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f600806d640 0x7f6008153700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:01.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.853+0000 7f600ec00640 1 -- 172.21.15.90:0/3146911793 >> 172.21.15.90:0/3146911793 conn(0x7f600806b140 msgr2=0x7f600815eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:01.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.853+0000 7f600ec00640 1 -- 172.21.15.90:0/3146911793 shutdown_connections 2024-04-25T01:19:01.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:01.854+0000 7f600ec00640 1 -- 172.21.15.90:0/3146911793 wait complete. 2024-04-25T01:19:01.856 INFO:tasks.ceph:Scrubbing osd.1 2024-04-25T01:19:01.856 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-25T01:19:02.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.072+0000 7f7ef8200640 1 Processor -- start 2024-04-25T01:19:02.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.072+0000 7f7ef8200640 1 -- start start 2024-04-25T01:19:02.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.074+0000 7f7ef8200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ef01616f0 0x7f7ef0161b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.074+0000 7f7ef8200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f7ef00ae160 0x7f7ef00ae5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.074+0000 7f7ef8200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ef0071c10 con 0x7f7ef0072620 2024-04-25T01:19:02.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.074+0000 7f7ef8200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f7ef0071ef0 con 0x7f7ef00ae160 2024-04-25T01:19:02.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.074+0000 7f7ef8200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7ef00721d0 con 0x7f7ef01616f0 2024-04-25T01:19:02.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.074+0000 7f7ef4e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f7ef0072620 0x7f7ef01620d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:42110/0 (socket says 172.21.15.90:42110) 2024-04-25T01:19:02.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.074+0000 7f7ef4e00640 1 -- 172.21.15.90:0/3303860514 learned_addr learned my addr 172.21.15.90:0/3303860514 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:19:02.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.075+0000 7f7ef6200640 1 --2- 172.21.15.90:0/3303860514 >> v2:172.21.15.103:3300/0 conn(0x7f7ef00ae160 0x7f7ef00ae5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:02.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.075+0000 7f7ef5800640 1 --2- 172.21.15.90:0/3303860514 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ef01616f0 0x7f7ef0161b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:02.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.075+0000 7f7eefe00640 1 -- 172.21.15.90:0/3303860514 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2028115871 0 0) 0x7f7ef0071c10 con 0x7f7ef0072620 2024-04-25T01:19:02.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.075+0000 7f7eefe00640 1 -- 172.21.15.90:0/3303860514 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ed4003520 con 0x7f7ef0072620 2024-04-25T01:19:02.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.075+0000 7f7eefe00640 1 -- 172.21.15.90:0/3303860514 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2253856761 0 0) 0x7f7ed4003520 con 0x7f7ef0072620 2024-04-25T01:19:02.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.075+0000 7f7eefe00640 1 -- 172.21.15.90:0/3303860514 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ef01616f0 msgr2=0x7f7ef0161b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:02.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.075+0000 7f7eefe00640 1 --2- 172.21.15.90:0/3303860514 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ef01616f0 0x7f7ef0161b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:02.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.075+0000 7f7eefe00640 1 -- 172.21.15.90:0/3303860514 >> v2:172.21.15.103:3300/0 conn(0x7f7ef00ae160 msgr2=0x7f7ef00ae5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:02.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.075+0000 7f7eefe00640 1 --2- 172.21.15.90:0/3303860514 >> v2:172.21.15.103:3300/0 conn(0x7f7ef00ae160 0x7f7ef00ae5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:02.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.075+0000 7f7eefe00640 1 -- 172.21.15.90:0/3303860514 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef00ad960 con 0x7f7ef0072620 2024-04-25T01:19:02.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.075+0000 7f7eefe00640 1 -- 172.21.15.90:0/3303860514 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f7edc001670 con 0x7f7ef0072620 2024-04-25T01:19:02.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.075+0000 7f7eefe00640 1 -- 172.21.15.90:0/3303860514 >> v1:172.21.15.90:6789/0 conn(0x7f7ef0072620 legacy=0x7f7ef01620d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:02.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.075+0000 7f7eefe00640 1 --2- 172.21.15.90:0/3303860514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ed40039b0 0x7f7ed4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.075+0000 7f7eefe00640 1 --2- 172.21.15.90:0/3303860514 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ef01616f0 0x7f7ed4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.075+0000 7f7eefe00640 1 --2- 172.21.15.90:0/3303860514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ed4004d40 0x7f7ed40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.075+0000 7f7eefe00640 1 -- 172.21.15.90:0/3303860514 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef0071ef0 con 0x7f7ed40039b0 2024-04-25T01:19:02.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.075+0000 7f7eefe00640 1 -- 172.21.15.90:0/3303860514 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef00721d0 con 0x7f7ed4004d40 2024-04-25T01:19:02.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.075+0000 7f7eefe00640 1 -- 172.21.15.90:0/3303860514 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7ed400b9c0 con 0x7f7ef01616f0 2024-04-25T01:19:02.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.076+0000 7f7ef5800640 1 --2- 172.21.15.90:0/3303860514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ed40039b0 0x7f7ed4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:02.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.076+0000 7f7ef6200640 1 --2- 172.21.15.90:0/3303860514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ed4004d40 0x7f7ed40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:02.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.076+0000 7f7ef5800640 1 -- 172.21.15.90:0/3303860514 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ef01616f0 msgr2=0x7f7ed4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:02.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.076+0000 7f7ef5800640 1 --2- 172.21.15.90:0/3303860514 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ef01616f0 0x7f7ed4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:02.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.076+0000 7f7ef5800640 1 -- 172.21.15.90:0/3303860514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ed4004d40 msgr2=0x7f7ed40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:02.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.076+0000 7f7ef5800640 1 --2- 172.21.15.90:0/3303860514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ed4004d40 0x7f7ed40051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:02.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.076+0000 7f7ef5800640 1 -- 172.21.15.90:0/3303860514 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ed4003520 con 0x7f7ed40039b0 2024-04-25T01:19:02.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.077+0000 7f7ef5800640 1 --2- 172.21.15.90:0/3303860514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ed40039b0 0x7f7ed4003dd0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0009ac0 tx=0x7f7ed4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2154b8c9fd0d6de1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:02.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.077+0000 7f7eefe00640 1 -- 172.21.15.90:0/3303860514 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0036070 con 0x7f7ed40039b0 2024-04-25T01:19:02.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.077+0000 7f7eefe00640 1 -- 172.21.15.90:0/3303860514 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ee0031400 con 0x7f7ed40039b0 2024-04-25T01:19:02.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.077+0000 7f7eefe00640 1 -- 172.21.15.90:0/3303860514 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0009690 con 0x7f7ed40039b0 2024-04-25T01:19:02.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.078+0000 7f7ef8200640 1 -- 172.21.15.90:0/3303860514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ed40039b0 msgr2=0x7f7ed4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:02.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.078+0000 7f7ef8200640 1 --2- 172.21.15.90:0/3303860514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ed40039b0 0x7f7ed4003dd0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0009ac0 tx=0x7f7ed4005830 comp rx=0 tx=0).stop 2024-04-25T01:19:02.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.078+0000 7f7ef6200640 1 -- 172.21.15.90:0/3303860514 reap_dead start 2024-04-25T01:19:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.078+0000 7f7ef8200640 1 -- 172.21.15.90:0/3303860514 shutdown_connections 2024-04-25T01:19:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.078+0000 7f7ef8200640 1 -- 172.21.15.90:0/3303860514 >> 172.21.15.90:0/3303860514 conn(0x7f7ef00b16b0 msgr2=0x7f7ef00b1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.078+0000 7f7ef8200640 1 -- 172.21.15.90:0/3303860514 shutdown_connections 2024-04-25T01:19:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.078+0000 7f7ef8200640 1 -- 172.21.15.90:0/3303860514 wait complete. 2024-04-25T01:19:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.080+0000 7f7ef8200640 1 Processor -- start 2024-04-25T01:19:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.080+0000 7f7ef8200640 1 -- start start 2024-04-25T01:19:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.081+0000 7f7ef8200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ed40039b0 0x7f7ef0153170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.081+0000 7f7ef8200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ef0145430 0x7f7ef0159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.081+0000 7f7ef8200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ef0159ff0 0x7f7ef015a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.081+0000 7f7ef8200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef006cdf0 con 0x7f7ef0145430 2024-04-25T01:19:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.081+0000 7f7ef8200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef00718c0 con 0x7f7ef0159ff0 2024-04-25T01:19:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.081+0000 7f7ef8200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7ef01627e0 con 0x7f7ed40039b0 2024-04-25T01:19:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.081+0000 7f7ef4e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ef0145430 0x7f7ef0159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.081+0000 7f7ef4e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ef0145430 0x7f7ef0159ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45346/0 (socket says 172.21.15.90:45346) 2024-04-25T01:19:02.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.081+0000 7f7ef4e00640 1 -- 172.21.15.90:0/3510676546 learned_addr learned my addr 172.21.15.90:0/3510676546 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:19:02.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.082+0000 7f7ef6200640 1 --2- 172.21.15.90:0/3510676546 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ef0159ff0 0x7f7ef015a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:02.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.082+0000 7f7ef5800640 1 --2- 172.21.15.90:0/3510676546 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ed40039b0 0x7f7ef0153170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:02.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.082+0000 7f7ef4e00640 1 -- 172.21.15.90:0/3510676546 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ed40039b0 msgr2=0x7f7ef0153170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:02.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.082+0000 7f7ef4e00640 1 --2- 172.21.15.90:0/3510676546 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7ed40039b0 0x7f7ef0153170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:02.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.082+0000 7f7ef4e00640 1 -- 172.21.15.90:0/3510676546 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ef0159ff0 msgr2=0x7f7ef015a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:02.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.082+0000 7f7ef4e00640 1 --2- 172.21.15.90:0/3510676546 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ef0159ff0 0x7f7ef015a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:02.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.082+0000 7f7ef4e00640 1 -- 172.21.15.90:0/3510676546 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef00cc230 con 0x7f7ef0145430 2024-04-25T01:19:02.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.082+0000 7f7ef6200640 1 --2- 172.21.15.90:0/3510676546 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ef0159ff0 0x7f7ef015a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:19:02.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.082+0000 7f7ef4e00640 1 --2- 172.21.15.90:0/3510676546 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ef0145430 0x7f7ef0159ab0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f7edc00c790 tx=0x7f7edc00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:02.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.082+0000 7f7eed600640 1 -- 172.21.15.90:0/3510676546 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7edc018070 con 0x7f7ef0145430 2024-04-25T01:19:02.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.082+0000 7f7eed600640 1 -- 172.21.15.90:0/3510676546 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7edc0074e0 con 0x7f7ef0145430 2024-04-25T01:19:02.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.082+0000 7f7ef8200640 1 -- 172.21.15.90:0/3510676546 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ef00cc510 con 0x7f7ef0145430 2024-04-25T01:19:02.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.082+0000 7f7eed600640 1 -- 172.21.15.90:0/3510676546 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7edc004a20 con 0x7f7ef0145430 2024-04-25T01:19:02.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.083+0000 7f7ef8200640 1 -- 172.21.15.90:0/3510676546 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ef01458b0 con 0x7f7ef0145430 2024-04-25T01:19:02.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.084+0000 7f7ef8200640 1 -- 172.21.15.90:0/3510676546 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7ef0155e30 con 0x7f7ef0145430 2024-04-25T01:19:02.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.085+0000 7f7eed600640 1 -- 172.21.15.90:0/3510676546 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7edc01b070 con 0x7f7ef0145430 2024-04-25T01:19:02.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.086+0000 7f7eed600640 1 --2- 172.21.15.90:0/3510676546 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7ed006be30 0x7f7ed006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.086+0000 7f7eed600640 1 -- 172.21.15.90:0/3510676546 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5421+0+0 (secure 0 0 0) 0x7f7edc08a320 con 0x7f7ef0145430 2024-04-25T01:19:02.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.086+0000 7f7ef5800640 1 --2- 172.21.15.90:0/3510676546 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7ed006be30 0x7f7ed006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:02.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.086+0000 7f7eed600640 1 --2- 172.21.15.90:0/3510676546 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x7f7ed0072370 0x7f7ed00747b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.086+0000 7f7eed600640 1 -- 172.21.15.90:0/3510676546 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7ed0074e80 con 0x7f7ed0072370 2024-04-25T01:19:02.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.086+0000 7f7eed600640 1 -- 172.21.15.90:0/3510676546 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f7edc091050 con 0x7f7ef0145430 2024-04-25T01:19:02.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.086+0000 7f7ef6200640 1 --2- 172.21.15.90:0/3510676546 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x7f7ed0072370 0x7f7ed00747b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:02.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.087+0000 7f7ef5800640 1 --2- 172.21.15.90:0/3510676546 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7ed006be30 0x7f7ed006e2f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0033000 tx=0x7f7ee0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:02.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.087+0000 7f7ef6200640 1 --2- 172.21.15.90:0/3510676546 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x7f7ed0072370 0x7f7ed00747b0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:02.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.088+0000 7f7eed600640 1 -- 172.21.15.90:0/3510676546 <== osd.1 v2:172.21.15.90:6800/2685473653 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7ed0074e80 con 0x7f7ed0072370 2024-04-25T01:19:02.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.102+0000 7f7ef8200640 1 -- 172.21.15.90:0/3510676546 --> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f7ef0068190 con 0x7f7ed0072370 2024-04-25T01:19:02.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.103+0000 7f7eed600640 1 -- 172.21.15.90:0/3510676546 <== osd.1 v2:172.21.15.90:6800/2685473653 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f7ef0068190 con 0x7f7ed0072370 2024-04-25T01:19:02.086 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T01:19:02.086 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T01:19:02.086 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T01:19:02.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.103+0000 7f7ef8200640 1 -- 172.21.15.90:0/3510676546 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x7f7ed0072370 msgr2=0x7f7ed00747b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:02.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.103+0000 7f7ef8200640 1 --2- 172.21.15.90:0/3510676546 >> [v2:172.21.15.90:6800/2685473653,v1:172.21.15.90:6801/2685473653] conn(0x7f7ed0072370 0x7f7ed00747b0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:02.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.104+0000 7f7ef8200640 1 -- 172.21.15.90:0/3510676546 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7ed006be30 msgr2=0x7f7ed006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:02.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.104+0000 7f7ef8200640 1 --2- 172.21.15.90:0/3510676546 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7ed006be30 0x7f7ed006e2f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0033000 tx=0x7f7ee0008030 comp rx=0 tx=0).stop 2024-04-25T01:19:02.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.104+0000 7f7ef8200640 1 -- 172.21.15.90:0/3510676546 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ef0145430 msgr2=0x7f7ef0159ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:02.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.104+0000 7f7ef8200640 1 --2- 172.21.15.90:0/3510676546 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ef0145430 0x7f7ef0159ab0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f7edc00c790 tx=0x7f7edc00cc50 comp rx=0 tx=0).stop 2024-04-25T01:19:02.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.104+0000 7f7ef6200640 1 -- 172.21.15.90:0/3510676546 reap_dead start 2024-04-25T01:19:02.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.104+0000 7f7ef8200640 1 -- 172.21.15.90:0/3510676546 shutdown_connections 2024-04-25T01:19:02.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.104+0000 7f7ef8200640 1 -- 172.21.15.90:0/3510676546 >> 172.21.15.90:0/3510676546 conn(0x7f7ef00b16b0 msgr2=0x7f7ef00d02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:02.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.104+0000 7f7ef8200640 1 -- 172.21.15.90:0/3510676546 shutdown_connections 2024-04-25T01:19:02.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.104+0000 7f7ef8200640 1 -- 172.21.15.90:0/3510676546 wait complete. 2024-04-25T01:19:02.104 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-25T01:19:02.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.321+0000 7f8447200640 1 Processor -- start 2024-04-25T01:19:02.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.322+0000 7f8447200640 1 -- start start 2024-04-25T01:19:02.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.323+0000 7f8447200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f844815df20 0x7f84480d1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.323+0000 7f8447200640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f844815e300 0x7f84480d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.323+0000 7f8447200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84480d01b0 con 0x7f84480d29d0 2024-04-25T01:19:02.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.323+0000 7f8447200640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f84480d22f0 con 0x7f844815e300 2024-04-25T01:19:02.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.323+0000 7f8447200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f84480d0690 con 0x7f844815df20 2024-04-25T01:19:02.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.323+0000 7f8446800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f84480d29d0 0x7f84480b3660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:42126/0 (socket says 172.21.15.90:42126) 2024-04-25T01:19:02.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.323+0000 7f8446800640 1 -- 172.21.15.90:0/984572937 learned_addr learned my addr 172.21.15.90:0/984572937 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:19:02.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.324+0000 7f8445e00640 1 --2- 172.21.15.90:0/984572937 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f844815df20 0x7f84480d1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:02.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.324+0000 7f8444a00640 1 -- 172.21.15.90:0/984572937 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2840171180 0 0) 0x7f84480d01b0 con 0x7f84480d29d0 2024-04-25T01:19:02.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.324+0000 7f8444a00640 1 -- 172.21.15.90:0/984572937 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f842c003520 con 0x7f84480d29d0 2024-04-25T01:19:02.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.324+0000 7f8445400640 1 --2- 172.21.15.90:0/984572937 >> v2:172.21.15.103:3300/0 conn(0x7f844815e300 0x7f84480d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:02.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.324+0000 7f8444a00640 1 -- 172.21.15.90:0/984572937 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1112259555 0 0) 0x7f842c003520 con 0x7f84480d29d0 2024-04-25T01:19:02.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.324+0000 7f8444a00640 1 -- 172.21.15.90:0/984572937 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f844815df20 msgr2=0x7f84480d1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:02.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.324+0000 7f8444a00640 1 --2- 172.21.15.90:0/984572937 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f844815df20 0x7f84480d1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:02.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.324+0000 7f8444a00640 1 -- 172.21.15.90:0/984572937 >> v2:172.21.15.103:3300/0 conn(0x7f844815e300 msgr2=0x7f84480d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:02.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.324+0000 7f8444a00640 1 --2- 172.21.15.90:0/984572937 >> v2:172.21.15.103:3300/0 conn(0x7f844815e300 0x7f84480d2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:02.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.324+0000 7f8444a00640 1 -- 172.21.15.90:0/984572937 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84480d08e0 con 0x7f84480d29d0 2024-04-25T01:19:02.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.324+0000 7f8444a00640 1 -- 172.21.15.90:0/984572937 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f84400019f0 con 0x7f84480d29d0 2024-04-25T01:19:02.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.324+0000 7f8444a00640 1 -- 172.21.15.90:0/984572937 >> v1:172.21.15.90:6789/0 conn(0x7f84480d29d0 legacy=0x7f84480b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:02.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.325+0000 7f8444a00640 1 --2- 172.21.15.90:0/984572937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f842c0039b0 0x7f842c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.325+0000 7f8444a00640 1 --2- 172.21.15.90:0/984572937 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f844815df20 0x7f842c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.325+0000 7f8444a00640 1 --2- 172.21.15.90:0/984572937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f842c004d40 0x7f842c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.325+0000 7f8444a00640 1 -- 172.21.15.90:0/984572937 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f84480d22f0 con 0x7f842c004d40 2024-04-25T01:19:02.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.325+0000 7f8444a00640 1 -- 172.21.15.90:0/984572937 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f84480d0690 con 0x7f842c0039b0 2024-04-25T01:19:02.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.325+0000 7f8444a00640 1 -- 172.21.15.90:0/984572937 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f842c00baa0 con 0x7f844815df20 2024-04-25T01:19:02.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.325+0000 7f8446800640 1 --2- 172.21.15.90:0/984572937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f842c004d40 0x7f842c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:02.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.325+0000 7f8445e00640 1 --2- 172.21.15.90:0/984572937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f842c0039b0 0x7f842c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:02.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.325+0000 7f8445400640 1 --2- 172.21.15.90:0/984572937 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f844815df20 0x7f842c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:02.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.325+0000 7f8446800640 1 -- 172.21.15.90:0/984572937 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f844815df20 msgr2=0x7f842c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:02.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.325+0000 7f8446800640 1 --2- 172.21.15.90:0/984572937 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f844815df20 0x7f842c004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:02.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.325+0000 7f8446800640 1 -- 172.21.15.90:0/984572937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f842c0039b0 msgr2=0x7f842c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:02.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.325+0000 7f8446800640 1 --2- 172.21.15.90:0/984572937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f842c0039b0 0x7f842c003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:02.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.325+0000 7f8446800640 1 -- 172.21.15.90:0/984572937 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f842c003520 con 0x7f842c004d40 2024-04-25T01:19:02.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.326+0000 7f8446800640 1 --2- 172.21.15.90:0/984572937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f842c004d40 0x7f842c0051c0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f844000a7a0 tx=0x7f842c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10ae90aa267d21e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:02.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.327+0000 7f8444a00640 1 -- 172.21.15.90:0/984572937 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f844000a3f0 con 0x7f842c004d40 2024-04-25T01:19:02.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.327+0000 7f8444a00640 1 -- 172.21.15.90:0/984572937 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f844002f070 con 0x7f842c004d40 2024-04-25T01:19:02.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.327+0000 7f8444a00640 1 -- 172.21.15.90:0/984572937 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f844002bc20 con 0x7f842c004d40 2024-04-25T01:19:02.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.327+0000 7f8447200640 1 -- 172.21.15.90:0/984572937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f842c004d40 msgr2=0x7f842c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:02.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.327+0000 7f8447200640 1 --2- 172.21.15.90:0/984572937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f842c004d40 0x7f842c0051c0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f844000a7a0 tx=0x7f842c005830 comp rx=0 tx=0).stop 2024-04-25T01:19:02.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.327+0000 7f8446800640 1 -- 172.21.15.90:0/984572937 reap_dead start 2024-04-25T01:19:02.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.327+0000 7f8447200640 1 -- 172.21.15.90:0/984572937 shutdown_connections 2024-04-25T01:19:02.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.327+0000 7f8447200640 1 -- 172.21.15.90:0/984572937 >> 172.21.15.90:0/984572937 conn(0x7f844809cff0 msgr2=0x7f84480d3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:02.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.327+0000 7f8447200640 1 -- 172.21.15.90:0/984572937 shutdown_connections 2024-04-25T01:19:02.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.327+0000 7f8447200640 1 -- 172.21.15.90:0/984572937 wait complete. 2024-04-25T01:19:02.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.329+0000 7f8447200640 1 Processor -- start 2024-04-25T01:19:02.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.329+0000 7f8447200640 1 -- start start 2024-04-25T01:19:02.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.330+0000 7f8447200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f842c004d40 0x7f84480b9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.330+0000 7f8447200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84480b6880 0x7f84481f6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.330+0000 7f8447200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84481f7240 0x7f844815c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.330+0000 7f8447200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f84480d1db0 con 0x7f842c004d40 2024-04-25T01:19:02.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.330+0000 7f8447200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f84480d0ea0 con 0x7f84480b6880 2024-04-25T01:19:02.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.330+0000 7f8447200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f84480d1f30 con 0x7f84481f7240 2024-04-25T01:19:02.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.331+0000 7f8445e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f842c004d40 0x7f84480b9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:02.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.331+0000 7f8445e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f842c004d40 0x7f84480b9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45368/0 (socket says 172.21.15.90:45368) 2024-04-25T01:19:02.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.331+0000 7f8445e00640 1 -- 172.21.15.90:0/3239196789 learned_addr learned my addr 172.21.15.90:0/3239196789 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:19:02.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.331+0000 7f8445e00640 1 -- 172.21.15.90:0/3239196789 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84481f7240 msgr2=0x7f844815c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:02.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.331+0000 7f8445e00640 1 --2- 172.21.15.90:0/3239196789 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84481f7240 0x7f844815c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:02.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.331+0000 7f8445e00640 1 -- 172.21.15.90:0/3239196789 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84480b6880 msgr2=0x7f84481f6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:02.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.331+0000 7f8445e00640 1 --2- 172.21.15.90:0/3239196789 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84480b6880 0x7f84481f6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:02.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.331+0000 7f8445e00640 1 -- 172.21.15.90:0/3239196789 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84480b6d10 con 0x7f842c004d40 2024-04-25T01:19:02.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.331+0000 7f8445e00640 1 --2- 172.21.15.90:0/3239196789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f842c004d40 0x7f84480b9310 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f8438010890 tx=0x7f8438010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:02.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.332+0000 7f843e000640 1 -- 172.21.15.90:0/3239196789 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843800f040 con 0x7f842c004d40 2024-04-25T01:19:02.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.332+0000 7f843e000640 1 -- 172.21.15.90:0/3239196789 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f843800eae0 con 0x7f842c004d40 2024-04-25T01:19:02.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.332+0000 7f843e000640 1 -- 172.21.15.90:0/3239196789 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843801c400 con 0x7f842c004d40 2024-04-25T01:19:02.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.332+0000 7f8447200640 1 -- 172.21.15.90:0/3239196789 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84480b6ff0 con 0x7f842c004d40 2024-04-25T01:19:02.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.332+0000 7f8447200640 1 -- 172.21.15.90:0/3239196789 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84480b75b0 con 0x7f842c004d40 2024-04-25T01:19:02.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.333+0000 7f8447200640 1 -- 172.21.15.90:0/3239196789 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f844804f500 con 0x7f842c004d40 2024-04-25T01:19:02.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.334+0000 7f843e000640 1 -- 172.21.15.90:0/3239196789 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f843801c5a0 con 0x7f842c004d40 2024-04-25T01:19:02.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.335+0000 7f843e000640 1 --2- 172.21.15.90:0/3239196789 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f842806be30 0x7f842806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.335+0000 7f843e000640 1 -- 172.21.15.90:0/3239196789 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5421+0+0 (secure 0 0 0) 0x7f843808b380 con 0x7f842c004d40 2024-04-25T01:19:02.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.336+0000 7f8445400640 1 --2- 172.21.15.90:0/3239196789 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f842806be30 0x7f842806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:02.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.336+0000 7f8445400640 1 --2- 172.21.15.90:0/3239196789 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f842806be30 0x7f842806e2f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8434006e80 tx=0x7f8434006e10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:02.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.340+0000 7f843e000640 1 -- 172.21.15.90:0/3239196789 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8438018030 con 0x7f842c004d40 2024-04-25T01:19:02.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.536+0000 7f8447200640 1 -- 172.21.15.90:0/3239196789 --> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f84480b7db0 con 0x7f842806be30 2024-04-25T01:19:02.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.537+0000 7f843e000640 1 -- 172.21.15.90:0/3239196789 <== mgr.4104 v2:172.21.15.90:6832/2198086733 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f843400e510 con 0x7f842806be30 2024-04-25T01:19:02.520 INFO:teuthology.orchestra.run.smithi090.stderr:instructed osd(s) 1 to deep-scrub 2024-04-25T01:19:02.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.540+0000 7f8447200640 1 -- 172.21.15.90:0/3239196789 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f842806be30 msgr2=0x7f842806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:02.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.540+0000 7f8447200640 1 --2- 172.21.15.90:0/3239196789 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f842806be30 0x7f842806e2f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8434006e80 tx=0x7f8434006e10 comp rx=0 tx=0).stop 2024-04-25T01:19:02.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.540+0000 7f8447200640 1 -- 172.21.15.90:0/3239196789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f842c004d40 msgr2=0x7f84480b9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:02.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.540+0000 7f8447200640 1 --2- 172.21.15.90:0/3239196789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f842c004d40 0x7f84480b9310 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f8438010890 tx=0x7f8438010d50 comp rx=0 tx=0).stop 2024-04-25T01:19:02.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.540+0000 7f8447200640 1 -- 172.21.15.90:0/3239196789 shutdown_connections 2024-04-25T01:19:02.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.540+0000 7f8447200640 1 --2- 172.21.15.90:0/3239196789 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f842806be30 0x7f842806e2f0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:02.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.540+0000 7f8447200640 1 --2- 172.21.15.90:0/3239196789 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f84481f7240 0x7f844815c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:02.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.540+0000 7f8447200640 1 --2- 172.21.15.90:0/3239196789 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84480b6880 0x7f84481f6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:02.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.540+0000 7f8447200640 1 --2- 172.21.15.90:0/3239196789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f842c004d40 0x7f84480b9310 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:02.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.540+0000 7f8447200640 1 -- 172.21.15.90:0/3239196789 >> 172.21.15.90:0/3239196789 conn(0x7f844809cff0 msgr2=0x7f844806c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:02.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.540+0000 7f8447200640 1 -- 172.21.15.90:0/3239196789 shutdown_connections 2024-04-25T01:19:02.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.540+0000 7f8447200640 1 -- 172.21.15.90:0/3239196789 wait complete. 2024-04-25T01:19:02.542 INFO:tasks.ceph:Scrubbing osd.2 2024-04-25T01:19:02.542 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-25T01:19:02.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.767+0000 7fcb3d000640 1 Processor -- start 2024-04-25T01:19:02.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.768+0000 7fcb3d000640 1 -- start start 2024-04-25T01:19:02.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.769+0000 7fcb3d000640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fcb3815eed0 0x7fcb380cd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.769+0000 7fcb3d000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb3806f5e0 con 0x7fcb3806c260 2024-04-25T01:19:02.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.769+0000 7fcb3d000640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb3806f020 con 0x7fcb3806e540 2024-04-25T01:19:02.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.769+0000 7fcb3d000640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fcb3806f300 con 0x7fcb3815eed0 2024-04-25T01:19:02.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.769+0000 7fcb36800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fcb3806c260 0x7fcb380b4fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:42140/0 (socket says 172.21.15.90:42140) 2024-04-25T01:19:02.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.769+0000 7fcb36800640 1 -- 172.21.15.90:0/3992923260 learned_addr learned my addr 172.21.15.90:0/3992923260 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:19:02.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.770+0000 7fcb35e00640 1 --2- 172.21.15.90:0/3992923260 >> v2:172.21.15.103:3300/0 conn(0x7fcb3815eed0 0x7fcb380cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:02.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.770+0000 7fcb34a00640 1 -- 172.21.15.90:0/3992923260 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1048931821 0 0) 0x7fcb3806f5e0 con 0x7fcb3806c260 2024-04-25T01:19:02.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.770+0000 7fcb34a00640 1 -- 172.21.15.90:0/3992923260 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb1c003520 con 0x7fcb3806c260 2024-04-25T01:19:02.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.770+0000 7fcb34a00640 1 -- 172.21.15.90:0/3992923260 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2628007577 0 0) 0x7fcb1c003520 con 0x7fcb3806c260 2024-04-25T01:19:02.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.770+0000 7fcb34a00640 1 -- 172.21.15.90:0/3992923260 >> v2:172.21.15.103:3300/0 conn(0x7fcb3815eed0 msgr2=0x7fcb380cd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:02.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.770+0000 7fcb34a00640 1 --2- 172.21.15.90:0/3992923260 >> v2:172.21.15.103:3300/0 conn(0x7fcb3815eed0 0x7fcb380cd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:02.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.770+0000 7fcb34a00640 1 -- 172.21.15.90:0/3992923260 >> v1:172.21.15.103:6789/0 conn(0x7fcb3806e540 legacy=0x7fcb3806bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:02.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.770+0000 7fcb34a00640 1 -- 172.21.15.90:0/3992923260 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb380ccc70 con 0x7fcb3806c260 2024-04-25T01:19:02.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.770+0000 7fcb34a00640 1 -- 172.21.15.90:0/3992923260 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7fcb30001670 con 0x7fcb3806c260 2024-04-25T01:19:02.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.770+0000 7fcb34a00640 1 -- 172.21.15.90:0/3992923260 >> v1:172.21.15.90:6789/0 conn(0x7fcb3806c260 legacy=0x7fcb380b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:02.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.770+0000 7fcb34a00640 1 --2- 172.21.15.90:0/3992923260 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb1c0039b0 0x7fcb1c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.770+0000 7fcb34a00640 1 --2- 172.21.15.90:0/3992923260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb1c004800 0x7fcb1c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.771+0000 7fcb34a00640 1 --2- 172.21.15.90:0/3992923260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb1c0051c0 0x7fcb1c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.771+0000 7fcb34a00640 1 -- 172.21.15.90:0/3992923260 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3806f300 con 0x7fcb1c004800 2024-04-25T01:19:02.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.771+0000 7fcb34a00640 1 -- 172.21.15.90:0/3992923260 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1c00c010 con 0x7fcb1c0051c0 2024-04-25T01:19:02.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.771+0000 7fcb34a00640 1 -- 172.21.15.90:0/3992923260 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcb1c00c210 con 0x7fcb1c0039b0 2024-04-25T01:19:02.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.771+0000 7fcb35400640 1 --2- 172.21.15.90:0/3992923260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb1c004800 0x7fcb1c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:02.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.771+0000 7fcb35e00640 1 --2- 172.21.15.90:0/3992923260 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb1c0039b0 0x7fcb1c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:02.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.771+0000 7fcb35400640 1 -- 172.21.15.90:0/3992923260 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb1c0039b0 msgr2=0x7fcb1c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:02.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.771+0000 7fcb35400640 1 --2- 172.21.15.90:0/3992923260 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb1c0039b0 0x7fcb1c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:02.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.771+0000 7fcb35400640 1 -- 172.21.15.90:0/3992923260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb1c0051c0 msgr2=0x7fcb1c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:02.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.771+0000 7fcb35400640 1 --2- 172.21.15.90:0/3992923260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb1c0051c0 0x7fcb1c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:02.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.771+0000 7fcb35400640 1 -- 172.21.15.90:0/3992923260 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb380ccc70 con 0x7fcb1c004800 2024-04-25T01:19:02.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.771+0000 7fcb35e00640 1 --2- 172.21.15.90:0/3992923260 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb1c0039b0 0x7fcb1c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:19:02.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.771+0000 7fcb36800640 1 -- 172.21.15.90:0/3992923260 reap_dead start 2024-04-25T01:19:02.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.772+0000 7fcb35400640 1 --2- 172.21.15.90:0/3992923260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb1c004800 0x7fcb1c004c80 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fcb28001a30 tx=0x7fcb1c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d4406b1a5192041 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:02.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.773+0000 7fcb34a00640 1 -- 172.21.15.90:0/3992923260 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb28001a70 con 0x7fcb1c004800 2024-04-25T01:19:02.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.773+0000 7fcb34a00640 1 -- 172.21.15.90:0/3992923260 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb28003b70 con 0x7fcb1c004800 2024-04-25T01:19:02.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.773+0000 7fcb34a00640 1 -- 172.21.15.90:0/3992923260 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb280050d0 con 0x7fcb1c004800 2024-04-25T01:19:02.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.773+0000 7fcb3d000640 1 -- 172.21.15.90:0/3992923260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb1c004800 msgr2=0x7fcb1c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:02.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.773+0000 7fcb3d000640 1 --2- 172.21.15.90:0/3992923260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb1c004800 0x7fcb1c004c80 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fcb28001a30 tx=0x7fcb1c005830 comp rx=0 tx=0).stop 2024-04-25T01:19:02.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.774+0000 7fcb3d000640 1 -- 172.21.15.90:0/3992923260 shutdown_connections 2024-04-25T01:19:02.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.774+0000 7fcb3d000640 1 --2- 172.21.15.90:0/3992923260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb1c004800 0x7fcb1c004c80 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:02.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.774+0000 7fcb3d000640 1 -- 172.21.15.90:0/3992923260 >> 172.21.15.90:0/3992923260 conn(0x7fcb38072880 msgr2=0x7fcb38072c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:02.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.774+0000 7fcb3d000640 1 -- 172.21.15.90:0/3992923260 shutdown_connections 2024-04-25T01:19:02.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.774+0000 7fcb3d000640 1 -- 172.21.15.90:0/3992923260 wait complete. 2024-04-25T01:19:02.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.776+0000 7fcb3d000640 1 Processor -- start 2024-04-25T01:19:02.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.776+0000 7fcb3d000640 1 -- start start 2024-04-25T01:19:02.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.777+0000 7fcb3d000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb1c004800 0x7fcb3809d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.777+0000 7fcb3d000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb381575d0 0x7fcb38158910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.777+0000 7fcb3d000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb38158330 0x7fcb38158e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.777+0000 7fcb3d000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3815e550 con 0x7fcb38158330 2024-04-25T01:19:02.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.777+0000 7fcb3d000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3815e3d0 con 0x7fcb381575d0 2024-04-25T01:19:02.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.777+0000 7fcb3d000640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fcb3806e9d0 con 0x7fcb1c004800 2024-04-25T01:19:02.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.777+0000 7fcb36800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb38158330 0x7fcb38158e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:02.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.777+0000 7fcb36800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb38158330 0x7fcb38158e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45386/0 (socket says 172.21.15.90:45386) 2024-04-25T01:19:02.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.777+0000 7fcb36800640 1 -- 172.21.15.90:0/751449934 learned_addr learned my addr 172.21.15.90:0/751449934 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:19:02.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.778+0000 7fcb35400640 1 --2- 172.21.15.90:0/751449934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb381575d0 0x7fcb38158910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:02.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.778+0000 7fcb36800640 1 -- 172.21.15.90:0/751449934 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb1c004800 msgr2=0x7fcb3809d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:02.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.778+0000 7fcb36800640 1 --2- 172.21.15.90:0/751449934 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fcb1c004800 0x7fcb3809d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:02.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.778+0000 7fcb36800640 1 -- 172.21.15.90:0/751449934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb381575d0 msgr2=0x7fcb38158910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:02.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.778+0000 7fcb36800640 1 --2- 172.21.15.90:0/751449934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb381575d0 0x7fcb38158910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:02.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.778+0000 7fcb36800640 1 -- 172.21.15.90:0/751449934 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb380cd9f0 con 0x7fcb38158330 2024-04-25T01:19:02.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.778+0000 7fcb36800640 1 --2- 172.21.15.90:0/751449934 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb38158330 0x7fcb38158e50 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fcb300011e0 tx=0x7fcb3000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:02.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.778+0000 7fcb2e000640 1 -- 172.21.15.90:0/751449934 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb30018070 con 0x7fcb38158330 2024-04-25T01:19:02.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.778+0000 7fcb2e000640 1 -- 172.21.15.90:0/751449934 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb300072f0 con 0x7fcb38158330 2024-04-25T01:19:02.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.778+0000 7fcb2e000640 1 -- 172.21.15.90:0/751449934 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb300047e0 con 0x7fcb38158330 2024-04-25T01:19:02.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.778+0000 7fcb3d000640 1 -- 172.21.15.90:0/751449934 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb380cf010 con 0x7fcb38158330 2024-04-25T01:19:02.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.779+0000 7fcb3d000640 1 -- 172.21.15.90:0/751449934 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb380cf5d0 con 0x7fcb38158330 2024-04-25T01:19:02.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.779+0000 7fcb3d000640 1 -- 172.21.15.90:0/751449934 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcb38068190 con 0x7fcb38158330 2024-04-25T01:19:02.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.781+0000 7fcb2e000640 1 -- 172.21.15.90:0/751449934 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fcb3000a5b0 con 0x7fcb38158330 2024-04-25T01:19:02.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.782+0000 7fcb2e000640 1 --2- 172.21.15.90:0/751449934 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fcb1406be60 0x7fcb1406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.782+0000 7fcb2e000640 1 -- 172.21.15.90:0/751449934 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5421+0+0 (secure 0 0 0) 0x7fcb3008a0c0 con 0x7fcb38158330 2024-04-25T01:19:02.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.782+0000 7fcb35e00640 1 --2- 172.21.15.90:0/751449934 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fcb1406be60 0x7fcb1406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:02.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.782+0000 7fcb2e000640 1 --2- 172.21.15.90:0/751449934 >> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] conn(0x7fcb140723a0 0x7fcb140747e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.782+0000 7fcb2e000640 1 -- 172.21.15.90:0/751449934 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcb14074eb0 con 0x7fcb140723a0 2024-04-25T01:19:02.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.782+0000 7fcb2e000640 1 -- 172.21.15.90:0/751449934 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7fcb3008a470 con 0x7fcb38158330 2024-04-25T01:19:02.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.782+0000 7fcb35400640 1 --2- 172.21.15.90:0/751449934 >> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] conn(0x7fcb140723a0 0x7fcb140747e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:02.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.783+0000 7fcb35e00640 1 --2- 172.21.15.90:0/751449934 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fcb1406be60 0x7fcb1406e320 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fcb38050770 tx=0x7fcb20008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:02.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.783+0000 7fcb35400640 1 --2- 172.21.15.90:0/751449934 >> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] conn(0x7fcb140723a0 0x7fcb140747e0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:02.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.784+0000 7fcb2e000640 1 -- 172.21.15.90:0/751449934 <== osd.2 v2:172.21.15.90:6808/906792737 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcb14074eb0 con 0x7fcb140723a0 2024-04-25T01:19:02.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.798+0000 7fcb3d000640 1 -- 172.21.15.90:0/751449934 --> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fcb3809e2b0 con 0x7fcb140723a0 2024-04-25T01:19:02.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.799+0000 7fcb2e000640 1 -- 172.21.15.90:0/751449934 <== osd.2 v2:172.21.15.90:6808/906792737 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fcb3809e2b0 con 0x7fcb140723a0 2024-04-25T01:19:02.782 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T01:19:02.782 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T01:19:02.782 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T01:19:02.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.799+0000 7fcb3d000640 1 -- 172.21.15.90:0/751449934 >> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] conn(0x7fcb140723a0 msgr2=0x7fcb140747e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:02.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.799+0000 7fcb3d000640 1 --2- 172.21.15.90:0/751449934 >> [v2:172.21.15.90:6808/906792737,v1:172.21.15.90:6809/906792737] conn(0x7fcb140723a0 0x7fcb140747e0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:02.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.799+0000 7fcb3d000640 1 -- 172.21.15.90:0/751449934 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fcb1406be60 msgr2=0x7fcb1406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:02.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.799+0000 7fcb3d000640 1 --2- 172.21.15.90:0/751449934 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fcb1406be60 0x7fcb1406e320 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fcb38050770 tx=0x7fcb20008040 comp rx=0 tx=0).stop 2024-04-25T01:19:02.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.799+0000 7fcb3d000640 1 -- 172.21.15.90:0/751449934 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb38158330 msgr2=0x7fcb38158e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:02.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.799+0000 7fcb3d000640 1 --2- 172.21.15.90:0/751449934 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb38158330 0x7fcb38158e50 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fcb300011e0 tx=0x7fcb3000cb90 comp rx=0 tx=0).stop 2024-04-25T01:19:02.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.799+0000 7fcb36800640 1 -- 172.21.15.90:0/751449934 reap_dead start 2024-04-25T01:19:02.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.799+0000 7fcb3d000640 1 -- 172.21.15.90:0/751449934 shutdown_connections 2024-04-25T01:19:02.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.799+0000 7fcb3d000640 1 -- 172.21.15.90:0/751449934 >> 172.21.15.90:0/751449934 conn(0x7fcb38072880 msgr2=0x7fcb380aea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:02.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.799+0000 7fcb3d000640 1 -- 172.21.15.90:0/751449934 shutdown_connections 2024-04-25T01:19:02.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:02.800+0000 7fcb3d000640 1 -- 172.21.15.90:0/751449934 wait complete. 2024-04-25T01:19:02.800 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-25T01:19:02.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.007+0000 7fbf10c00640 1 Processor -- start 2024-04-25T01:19:02.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.008+0000 7fbf10c00640 1 -- start start 2024-04-25T01:19:02.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.009+0000 7fbf10c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbf0c142570 0x7fbf0c06cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.009+0000 7fbf10c00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fbf0c0720d0 0x7fbf0c15fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.009+0000 7fbf10c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf0c142950 con 0x7fbf0c070ba0 2024-04-25T01:19:02.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.009+0000 7fbf10c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fbf0c071c40 con 0x7fbf0c142570 2024-04-25T01:19:02.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.009+0000 7fbf10c00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fbf0c071030 con 0x7fbf0c0720d0 2024-04-25T01:19:02.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.010+0000 7fbf03e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fbf0c070ba0 0x7fbf0c06d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:42154/0 (socket says 172.21.15.90:42154) 2024-04-25T01:19:02.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.010+0000 7fbf03e00640 1 -- 172.21.15.90:0/1195859273 learned_addr learned my addr 172.21.15.90:0/1195859273 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:19:02.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.010+0000 7fbf09e00640 1 --2- 172.21.15.90:0/1195859273 >> v2:172.21.15.90:3300/0 conn(0x7fbf0c142570 0x7fbf0c06cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:02.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.010+0000 7fbf0a800640 1 --2- 172.21.15.90:0/1195859273 >> v2:172.21.15.103:3300/0 conn(0x7fbf0c0720d0 0x7fbf0c15fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:02.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.010+0000 7fbf09e00640 1 -- 172.21.15.90:0/1195859273 >> v2:172.21.15.103:3300/0 conn(0x7fbf0c0720d0 msgr2=0x7fbf0c15fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:02.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.010+0000 7fbf09e00640 1 --2- 172.21.15.90:0/1195859273 >> v2:172.21.15.103:3300/0 conn(0x7fbf0c0720d0 0x7fbf0c15fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:02.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.010+0000 7fbf09e00640 1 -- 172.21.15.90:0/1195859273 >> v1:172.21.15.90:6789/0 conn(0x7fbf0c070ba0 legacy=0x7fbf0c06d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:02.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.010+0000 7fbf09e00640 1 -- 172.21.15.90:0/1195859273 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf0c071310 con 0x7fbf0c142570 2024-04-25T01:19:02.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.010+0000 7fbf09400640 1 -- 172.21.15.90:0/1195859273 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 599914092 0 0) 0x7fbf0c142950 con 0x7fbf0c070ba0 2024-04-25T01:19:02.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.011+0000 7fbf09e00640 1 --2- 172.21.15.90:0/1195859273 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf0c142570 0x7fbf0c06cbc0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c0d0b00 tx=0x7fbefc02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6473969a2fb63c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:02.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.012+0000 7fbf09400640 1 -- 172.21.15.90:0/1195859273 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbefc037070 con 0x7fbf0c142570 2024-04-25T01:19:02.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.012+0000 7fbf09400640 1 -- 172.21.15.90:0/1195859273 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbefc032440 con 0x7fbf0c142570 2024-04-25T01:19:02.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.012+0000 7fbf09400640 1 -- 172.21.15.90:0/1195859273 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbefc032810 con 0x7fbf0c142570 2024-04-25T01:19:02.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.012+0000 7fbf10c00640 1 -- 172.21.15.90:0/1195859273 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf0c142570 msgr2=0x7fbf0c06cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:02.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.012+0000 7fbf10c00640 1 --2- 172.21.15.90:0/1195859273 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf0c142570 0x7fbf0c06cbc0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c0d0b00 tx=0x7fbefc02a1a0 comp rx=0 tx=0).stop 2024-04-25T01:19:02.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.013+0000 7fbf10c00640 1 -- 172.21.15.90:0/1195859273 shutdown_connections 2024-04-25T01:19:02.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.013+0000 7fbf10c00640 1 --2- 172.21.15.90:0/1195859273 >> v2:172.21.15.103:3300/0 conn(0x7fbf0c0720d0 0x7fbf0c15fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:02.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.013+0000 7fbf10c00640 1 --2- 172.21.15.90:0/1195859273 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf0c142570 0x7fbf0c06cbc0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:02.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.013+0000 7fbf10c00640 1 -- 172.21.15.90:0/1195859273 >> 172.21.15.90:0/1195859273 conn(0x7fbf0c06b140 msgr2=0x7fbf0c06b550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:02.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.013+0000 7fbf10c00640 1 -- 172.21.15.90:0/1195859273 shutdown_connections 2024-04-25T01:19:02.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.013+0000 7fbf10c00640 1 -- 172.21.15.90:0/1195859273 wait complete. 2024-04-25T01:19:02.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.014+0000 7fbf10c00640 1 Processor -- start 2024-04-25T01:19:02.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.015+0000 7fbf10c00640 1 -- start start 2024-04-25T01:19:02.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.015+0000 7fbf10c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf0c0720d0 0x7fbf0c155ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.015+0000 7fbf10c00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf0c142570 0x7fbf0c0cb040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.016+0000 7fbf10c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf0c13e4b0 0x7fbf0c154ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:02.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.016+0000 7fbf10c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c072c60 con 0x7fbf0c0720d0 2024-04-25T01:19:02.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.016+0000 7fbf10c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c0718d0 con 0x7fbf0c13e4b0 2024-04-25T01:19:02.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.016+0000 7fbf10c00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fbf0c072de0 con 0x7fbf0c142570 2024-04-25T01:19:02.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.016+0000 7fbf09e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf0c0720d0 0x7fbf0c155ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:02.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.016+0000 7fbf09e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf0c0720d0 0x7fbf0c155ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45402/0 (socket says 172.21.15.90:45402) 2024-04-25T01:19:02.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.016+0000 7fbf09e00640 1 -- 172.21.15.90:0/3739738118 learned_addr learned my addr 172.21.15.90:0/3739738118 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:19:03.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.016+0000 7fbf09e00640 1 -- 172.21.15.90:0/3739738118 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf0c142570 msgr2=0x7fbf0c0cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:03.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.016+0000 7fbf09e00640 1 --2- 172.21.15.90:0/3739738118 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf0c142570 0x7fbf0c0cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.016+0000 7fbf09e00640 1 -- 172.21.15.90:0/3739738118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf0c13e4b0 msgr2=0x7fbf0c154ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:03.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.016+0000 7fbf09e00640 1 --2- 172.21.15.90:0/3739738118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf0c13e4b0 0x7fbf0c154ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.016+0000 7fbf09e00640 1 -- 172.21.15.90:0/3739738118 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf0c13e990 con 0x7fbf0c0720d0 2024-04-25T01:19:03.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.017+0000 7fbf09e00640 1 --2- 172.21.15.90:0/3739738118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf0c0720d0 0x7fbf0c155ac0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fbefc009890 tx=0x7fbefc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:03.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.017+0000 7fbf02000640 1 -- 172.21.15.90:0/3739738118 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbefc037040 con 0x7fbf0c0720d0 2024-04-25T01:19:03.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.017+0000 7fbf02000640 1 -- 172.21.15.90:0/3739738118 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbefc042070 con 0x7fbf0c0720d0 2024-04-25T01:19:03.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.017+0000 7fbf02000640 1 -- 172.21.15.90:0/3739738118 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbefc030bb0 con 0x7fbf0c0720d0 2024-04-25T01:19:03.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.017+0000 7fbf10c00640 1 -- 172.21.15.90:0/3739738118 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf0c13ec70 con 0x7fbf0c0720d0 2024-04-25T01:19:03.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.017+0000 7fbf10c00640 1 -- 172.21.15.90:0/3739738118 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf0c13f230 con 0x7fbf0c0720d0 2024-04-25T01:19:03.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.018+0000 7fbf10c00640 1 -- 172.21.15.90:0/3739738118 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf0c04f5a0 con 0x7fbf0c0720d0 2024-04-25T01:19:03.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.019+0000 7fbf02000640 1 -- 172.21.15.90:0/3739738118 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbefc047460 con 0x7fbf0c0720d0 2024-04-25T01:19:03.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.020+0000 7fbf02000640 1 --2- 172.21.15.90:0/3739738118 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbee806be60 0x7fbee806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:03.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.020+0000 7fbf02000640 1 -- 172.21.15.90:0/3739738118 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5421+0+0 (secure 0 0 0) 0x7fbefc0ab960 con 0x7fbf0c0720d0 2024-04-25T01:19:03.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.020+0000 7fbf03e00640 1 --2- 172.21.15.90:0/3739738118 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbee806be60 0x7fbee806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:03.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.021+0000 7fbf03e00640 1 --2- 172.21.15.90:0/3739738118 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbee806be60 0x7fbee806e320 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fbef4007fd0 tx=0x7fbef4007920 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:03.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.025+0000 7fbf02000640 1 -- 172.21.15.90:0/3739738118 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fbefc032e20 con 0x7fbf0c0720d0 2024-04-25T01:19:03.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.226+0000 7fbf10c00640 1 -- 172.21.15.90:0/3739738118 --> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7fbf0c1418d0 con 0x7fbee806be60 2024-04-25T01:19:03.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.226+0000 7fbf02000640 1 -- 172.21.15.90:0/3739738118 <== mgr.4104 v2:172.21.15.90:6832/2198086733 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fbf0c1418d0 con 0x7fbee806be60 2024-04-25T01:19:03.209 INFO:teuthology.orchestra.run.smithi090.stderr:instructed osd(s) 2 to deep-scrub 2024-04-25T01:19:03.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.229+0000 7fbf10c00640 1 -- 172.21.15.90:0/3739738118 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbee806be60 msgr2=0x7fbee806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:03.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.229+0000 7fbf10c00640 1 --2- 172.21.15.90:0/3739738118 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbee806be60 0x7fbee806e320 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fbef4007fd0 tx=0x7fbef4007920 comp rx=0 tx=0).stop 2024-04-25T01:19:03.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.229+0000 7fbf10c00640 1 -- 172.21.15.90:0/3739738118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf0c0720d0 msgr2=0x7fbf0c155ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:03.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.229+0000 7fbf10c00640 1 --2- 172.21.15.90:0/3739738118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf0c0720d0 0x7fbf0c155ac0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fbefc009890 tx=0x7fbefc0046c0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.229+0000 7fbf10c00640 1 -- 172.21.15.90:0/3739738118 shutdown_connections 2024-04-25T01:19:03.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.229+0000 7fbf10c00640 1 --2- 172.21.15.90:0/3739738118 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fbee806be60 0x7fbee806e320 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.229+0000 7fbf10c00640 1 --2- 172.21.15.90:0/3739738118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf0c13e4b0 0x7fbf0c154ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.229+0000 7fbf10c00640 1 --2- 172.21.15.90:0/3739738118 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fbf0c142570 0x7fbf0c0cb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.229+0000 7fbf10c00640 1 --2- 172.21.15.90:0/3739738118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf0c0720d0 0x7fbf0c155ac0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.229+0000 7fbf10c00640 1 -- 172.21.15.90:0/3739738118 >> 172.21.15.90:0/3739738118 conn(0x7fbf0c06b140 msgr2=0x7fbf0c15bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:03.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.229+0000 7fbf10c00640 1 -- 172.21.15.90:0/3739738118 shutdown_connections 2024-04-25T01:19:03.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.229+0000 7fbf10c00640 1 -- 172.21.15.90:0/3739738118 wait complete. 2024-04-25T01:19:03.231 INFO:tasks.ceph:Scrubbing osd.3 2024-04-25T01:19:03.231 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-25T01:19:03.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.450+0000 7f8932600640 1 Processor -- start 2024-04-25T01:19:03.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.451+0000 7f8932600640 1 -- start start 2024-04-25T01:19:03.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.452+0000 7f8932600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f892c1616f0 0x7f892c161b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:03.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.452+0000 7f8932600640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f892c072620 0x7f892c1620d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:03.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.452+0000 7f8932600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f892c162610 0x7f892c0b29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:03.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.452+0000 7f8932600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f892c071c10 con 0x7f892c162610 2024-04-25T01:19:03.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.452+0000 7f8932600640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f892c071ef0 con 0x7f892c072620 2024-04-25T01:19:03.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.452+0000 7f8932600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f892c0721d0 con 0x7f892c1616f0 2024-04-25T01:19:03.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.453+0000 7f892be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f892c162610 0x7f892c0b29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:03.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.453+0000 7f892be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f892c162610 0x7f892c0b29b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45410/0 (socket says 172.21.15.90:45410) 2024-04-25T01:19:03.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.453+0000 7f892be00640 1 -- 172.21.15.90:0/552026906 learned_addr learned my addr 172.21.15.90:0/552026906 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:19:03.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.453+0000 7f8922a00640 1 --2- 172.21.15.90:0/552026906 >> v2:172.21.15.103:3300/0 conn(0x7f892c072620 0x7f892c1620d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:03.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.453+0000 7f892b400640 1 --2- 172.21.15.90:0/552026906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f892c1616f0 0x7f892c161b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:03.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.453+0000 7f892be00640 1 -- 172.21.15.90:0/552026906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f892c1616f0 msgr2=0x7f892c161b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:03.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.453+0000 7f892be00640 1 --2- 172.21.15.90:0/552026906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f892c1616f0 0x7f892c161b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.453+0000 7f892be00640 1 -- 172.21.15.90:0/552026906 >> v2:172.21.15.103:3300/0 conn(0x7f892c072620 msgr2=0x7f892c1620d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:03.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.453+0000 7f892be00640 1 --2- 172.21.15.90:0/552026906 >> v2:172.21.15.103:3300/0 conn(0x7f892c072620 0x7f892c1620d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.453+0000 7f892be00640 1 -- 172.21.15.90:0/552026906 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f892c0ad960 con 0x7f892c162610 2024-04-25T01:19:03.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.454+0000 7f892be00640 1 --2- 172.21.15.90:0/552026906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f892c162610 0x7f892c0b29b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f8924009e50 tx=0x7f892402a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de08637e7959f24e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:03.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.455+0000 7f892aa00640 1 -- 172.21.15.90:0/552026906 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8924036070 con 0x7f892c162610 2024-04-25T01:19:03.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.455+0000 7f892aa00640 1 -- 172.21.15.90:0/552026906 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8924031600 con 0x7f892c162610 2024-04-25T01:19:03.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.455+0000 7f892aa00640 1 -- 172.21.15.90:0/552026906 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89240319d0 con 0x7f892c162610 2024-04-25T01:19:03.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.455+0000 7f8932600640 1 -- 172.21.15.90:0/552026906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f892c162610 msgr2=0x7f892c0b29b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:03.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.455+0000 7f8932600640 1 --2- 172.21.15.90:0/552026906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f892c162610 0x7f892c0b29b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f8924009e50 tx=0x7f892402a6c0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.455+0000 7f8932600640 1 -- 172.21.15.90:0/552026906 shutdown_connections 2024-04-25T01:19:03.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.455+0000 7f8932600640 1 --2- 172.21.15.90:0/552026906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f892c162610 0x7f892c0b29b0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.455+0000 7f8932600640 1 --2- 172.21.15.90:0/552026906 >> v2:172.21.15.103:3300/0 conn(0x7f892c072620 0x7f892c1620d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.455+0000 7f8932600640 1 --2- 172.21.15.90:0/552026906 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f892c1616f0 0x7f892c161b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.456+0000 7f8932600640 1 -- 172.21.15.90:0/552026906 >> 172.21.15.90:0/552026906 conn(0x7f892c0b16b0 msgr2=0x7f892c0b1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:03.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.456+0000 7f8932600640 1 -- 172.21.15.90:0/552026906 shutdown_connections 2024-04-25T01:19:03.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.456+0000 7f8932600640 1 -- 172.21.15.90:0/552026906 wait complete. 2024-04-25T01:19:03.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.458+0000 7f8932600640 1 Processor -- start 2024-04-25T01:19:03.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.458+0000 7f8932600640 1 -- start start 2024-04-25T01:19:03.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.459+0000 7f8932600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f892c072620 0x7f892c149590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:03.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.459+0000 7f8932600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f892c1616f0 0x7f892c0cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:03.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.459+0000 7f8932600640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f892c162610 0x7f892c15cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:03.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.459+0000 7f8932600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f892c06cdf0 con 0x7f892c1616f0 2024-04-25T01:19:03.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.459+0000 7f8932600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f892c0718c0 con 0x7f892c072620 2024-04-25T01:19:03.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.459+0000 7f8932600640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f892c0b2ef0 con 0x7f892c162610 2024-04-25T01:19:03.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.459+0000 7f8922a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f892c1616f0 0x7f892c0cd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:03.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.459+0000 7f8922a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f892c1616f0 0x7f892c0cd0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45422/0 (socket says 172.21.15.90:45422) 2024-04-25T01:19:03.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.459+0000 7f8922a00640 1 -- 172.21.15.90:0/2889628516 learned_addr learned my addr 172.21.15.90:0/2889628516 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:19:03.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.460+0000 7f892b400640 1 --2- 172.21.15.90:0/2889628516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f892c072620 0x7f892c149590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:03.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.460+0000 7f892be00640 1 --2- 172.21.15.90:0/2889628516 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f892c162610 0x7f892c15cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:03.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.460+0000 7f8922a00640 1 -- 172.21.15.90:0/2889628516 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f892c162610 msgr2=0x7f892c15cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:03.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.460+0000 7f8922a00640 1 --2- 172.21.15.90:0/2889628516 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f892c162610 0x7f892c15cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.460+0000 7f8922a00640 1 -- 172.21.15.90:0/2889628516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f892c072620 msgr2=0x7f892c149590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:03.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.460+0000 7f8922a00640 1 --2- 172.21.15.90:0/2889628516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f892c072620 0x7f892c149590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.460+0000 7f8922a00640 1 -- 172.21.15.90:0/2889628516 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f892c0cdb60 con 0x7f892c1616f0 2024-04-25T01:19:03.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.460+0000 7f892b400640 1 --2- 172.21.15.90:0/2889628516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f892c072620 0x7f892c149590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:19:03.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.460+0000 7f8922a00640 1 --2- 172.21.15.90:0/2889628516 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f892c1616f0 0x7f892c0cd0a0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f891400d800 tx=0x7f891400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:03.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.460+0000 7f8923e00640 1 -- 172.21.15.90:0/2889628516 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8914018020 con 0x7f892c1616f0 2024-04-25T01:19:03.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.460+0000 7f8923e00640 1 -- 172.21.15.90:0/2889628516 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89140096a0 con 0x7f892c1616f0 2024-04-25T01:19:03.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.460+0000 7f8923e00640 1 -- 172.21.15.90:0/2889628516 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8914009a70 con 0x7f892c1616f0 2024-04-25T01:19:03.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.461+0000 7f8932600640 1 -- 172.21.15.90:0/2889628516 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f892c0cde40 con 0x7f892c1616f0 2024-04-25T01:19:03.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.461+0000 7f8932600640 1 -- 172.21.15.90:0/2889628516 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f892c0ce400 con 0x7f892c1616f0 2024-04-25T01:19:03.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.462+0000 7f8932600640 1 -- 172.21.15.90:0/2889628516 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f892c068190 con 0x7f892c1616f0 2024-04-25T01:19:03.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.463+0000 7f8923e00640 1 -- 172.21.15.90:0/2889628516 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f891401b020 con 0x7f892c1616f0 2024-04-25T01:19:03.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.464+0000 7f8923e00640 1 --2- 172.21.15.90:0/2889628516 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f890806be60 0x7f890806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:03.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.464+0000 7f8923e00640 1 -- 172.21.15.90:0/2889628516 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5421+0+0 (secure 0 0 0) 0x7f891408afe0 con 0x7f892c1616f0 2024-04-25T01:19:03.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.464+0000 7f8923e00640 1 --2- 172.21.15.90:0/2889628516 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x7f89080723a0 0x7f89080747e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:03.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.464+0000 7f892b400640 1 --2- 172.21.15.90:0/2889628516 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f890806be60 0x7f890806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:03.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.464+0000 7f8923e00640 1 -- 172.21.15.90:0/2889628516 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8908074eb0 con 0x7f89080723a0 2024-04-25T01:19:03.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.464+0000 7f8923e00640 1 -- 172.21.15.90:0/2889628516 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f891408d5e0 con 0x7f892c1616f0 2024-04-25T01:19:03.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.465+0000 7f892be00640 1 --2- 172.21.15.90:0/2889628516 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x7f89080723a0 0x7f89080747e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:03.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.465+0000 7f892b400640 1 --2- 172.21.15.90:0/2889628516 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f890806be60 0x7f890806e320 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f891c006fd0 tx=0x7f891c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:03.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.465+0000 7f892be00640 1 --2- 172.21.15.90:0/2889628516 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x7f89080723a0 0x7f89080747e0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:03.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.466+0000 7f8923e00640 1 -- 172.21.15.90:0/2889628516 <== osd.3 v2:172.21.15.90:6812/8683104 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8908074eb0 con 0x7f89080723a0 2024-04-25T01:19:03.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.481+0000 7f8932600640 1 -- 172.21.15.90:0/2889628516 --> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f892c06e330 con 0x7f89080723a0 2024-04-25T01:19:03.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.481+0000 7f8923e00640 1 -- 172.21.15.90:0/2889628516 <== osd.3 v2:172.21.15.90:6812/8683104 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f892c06e330 con 0x7f89080723a0 2024-04-25T01:19:03.464 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T01:19:03.464 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T01:19:03.464 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T01:19:03.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.482+0000 7f8932600640 1 -- 172.21.15.90:0/2889628516 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x7f89080723a0 msgr2=0x7f89080747e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:03.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.482+0000 7f8932600640 1 --2- 172.21.15.90:0/2889628516 >> [v2:172.21.15.90:6812/8683104,v1:172.21.15.90:6814/8683104] conn(0x7f89080723a0 0x7f89080747e0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.482+0000 7f8932600640 1 -- 172.21.15.90:0/2889628516 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f890806be60 msgr2=0x7f890806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:03.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.482+0000 7f8932600640 1 --2- 172.21.15.90:0/2889628516 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f890806be60 0x7f890806e320 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f891c006fd0 tx=0x7f891c008040 comp rx=0 tx=0).stop 2024-04-25T01:19:03.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.482+0000 7f8932600640 1 -- 172.21.15.90:0/2889628516 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f892c1616f0 msgr2=0x7f892c0cd0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:03.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.482+0000 7f8932600640 1 --2- 172.21.15.90:0/2889628516 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f892c1616f0 0x7f892c0cd0a0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f891400d800 tx=0x7f891400dcc0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.482+0000 7f892be00640 1 -- 172.21.15.90:0/2889628516 reap_dead start 2024-04-25T01:19:03.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.483+0000 7f8932600640 1 -- 172.21.15.90:0/2889628516 shutdown_connections 2024-04-25T01:19:03.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.483+0000 7f8932600640 1 -- 172.21.15.90:0/2889628516 >> 172.21.15.90:0/2889628516 conn(0x7f892c0b16b0 msgr2=0x7f892c06d980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:03.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.483+0000 7f8932600640 1 -- 172.21.15.90:0/2889628516 shutdown_connections 2024-04-25T01:19:03.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.483+0000 7f8932600640 1 -- 172.21.15.90:0/2889628516 wait complete. 2024-04-25T01:19:03.483 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-25T01:19:03.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.697+0000 7f897be00640 1 Processor -- start 2024-04-25T01:19:03.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.698+0000 7f897be00640 1 -- start start 2024-04-25T01:19:03.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.699+0000 7f897be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f897c0ae710 0x7f897c0cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:03.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.699+0000 7f897be00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f897c163300 0x7f897c0aeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:03.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.699+0000 7f897be00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f897c162700 con 0x7f897c165e70 2024-04-25T01:19:03.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.699+0000 7f897be00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f897c072d30 con 0x7f897c163300 2024-04-25T01:19:03.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.699+0000 7f897be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f897c070b70 con 0x7f897c0ae710 2024-04-25T01:19:03.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.699+0000 7f897b400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f897c165e70 0x7f897c166300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:42156/0 (socket says 172.21.15.90:42156) 2024-04-25T01:19:03.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.699+0000 7f897b400640 1 -- 172.21.15.90:0/934541163 learned_addr learned my addr 172.21.15.90:0/934541163 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:19:03.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.700+0000 7f897aa00640 1 --2- 172.21.15.90:0/934541163 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f897c0ae710 0x7f897c0cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:03.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.700+0000 7f8972000640 1 --2- 172.21.15.90:0/934541163 >> v2:172.21.15.103:3300/0 conn(0x7f897c163300 0x7f897c0aeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:03.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.700+0000 7f897a000640 1 -- 172.21.15.90:0/934541163 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1277373168 0 0) 0x7f897c162700 con 0x7f897c165e70 2024-04-25T01:19:03.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.700+0000 7f897a000640 1 -- 172.21.15.90:0/934541163 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8960003520 con 0x7f897c165e70 2024-04-25T01:19:03.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.700+0000 7f897a000640 1 -- 172.21.15.90:0/934541163 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3013904612 0 0) 0x7f8960003520 con 0x7f897c165e70 2024-04-25T01:19:03.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.700+0000 7f897a000640 1 -- 172.21.15.90:0/934541163 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f897c0ae710 msgr2=0x7f897c0cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:03.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.700+0000 7f897a000640 1 --2- 172.21.15.90:0/934541163 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f897c0ae710 0x7f897c0cf1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.700+0000 7f897a000640 1 -- 172.21.15.90:0/934541163 >> v2:172.21.15.103:3300/0 conn(0x7f897c163300 msgr2=0x7f897c0aeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:03.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.700+0000 7f897a000640 1 --2- 172.21.15.90:0/934541163 >> v2:172.21.15.103:3300/0 conn(0x7f897c163300 0x7f897c0aeaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.700+0000 7f897a000640 1 -- 172.21.15.90:0/934541163 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f897c0af030 con 0x7f897c165e70 2024-04-25T01:19:03.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.700+0000 7f897a000640 1 -- 172.21.15.90:0/934541163 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f89740019f0 con 0x7f897c165e70 2024-04-25T01:19:03.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.700+0000 7f897a000640 1 -- 172.21.15.90:0/934541163 >> v1:172.21.15.90:6789/0 conn(0x7f897c165e70 legacy=0x7f897c166300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:03.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.701+0000 7f897a000640 1 --2- 172.21.15.90:0/934541163 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f897c0ae710 0x7f89600039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:03.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.701+0000 7f897a000640 1 --2- 172.21.15.90:0/934541163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8960003f30 0x7f8960004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:03.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.701+0000 7f897a000640 1 --2- 172.21.15.90:0/934541163 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8960004d40 0x7f89600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:03.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.701+0000 7f897a000640 1 -- 172.21.15.90:0/934541163 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f897c072d30 con 0x7f8960003f30 2024-04-25T01:19:03.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.701+0000 7f897a000640 1 -- 172.21.15.90:0/934541163 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f897c070b70 con 0x7f8960004d40 2024-04-25T01:19:03.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.701+0000 7f897a000640 1 -- 172.21.15.90:0/934541163 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f896000bac0 con 0x7f897c0ae710 2024-04-25T01:19:03.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.701+0000 7f8972000640 1 --2- 172.21.15.90:0/934541163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8960003f30 0x7f8960004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:03.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.701+0000 7f8972000640 1 -- 172.21.15.90:0/934541163 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f897c0ae710 msgr2=0x7f89600039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:03.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.701+0000 7f897aa00640 1 --2- 172.21.15.90:0/934541163 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f897c0ae710 0x7f89600039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:03.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.701+0000 7f8972000640 1 --2- 172.21.15.90:0/934541163 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f897c0ae710 0x7f89600039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.701+0000 7f8972000640 1 -- 172.21.15.90:0/934541163 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8960004d40 msgr2=0x7f89600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:03.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.701+0000 7f897b400640 1 --2- 172.21.15.90:0/934541163 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8960004d40 0x7f89600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:03.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.701+0000 7f8972000640 1 --2- 172.21.15.90:0/934541163 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8960004d40 0x7f89600051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.701+0000 7f8972000640 1 -- 172.21.15.90:0/934541163 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8960003520 con 0x7f8960003f30 2024-04-25T01:19:03.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.702+0000 7f8972000640 1 --2- 172.21.15.90:0/934541163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8960003f30 0x7f8960004800 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f8964009af0 tx=0x7f8960005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f3a6b1d2bb3d32f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:03.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.703+0000 7f897a000640 1 -- 172.21.15.90:0/934541163 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8964036070 con 0x7f8960003f30 2024-04-25T01:19:03.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.703+0000 7f897a000640 1 -- 172.21.15.90:0/934541163 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8964031400 con 0x7f8960003f30 2024-04-25T01:19:03.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.703+0000 7f897a000640 1 -- 172.21.15.90:0/934541163 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8964009690 con 0x7f8960003f30 2024-04-25T01:19:03.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.703+0000 7f897be00640 1 -- 172.21.15.90:0/934541163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8960003f30 msgr2=0x7f8960004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:03.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.703+0000 7f897be00640 1 --2- 172.21.15.90:0/934541163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8960003f30 0x7f8960004800 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f8964009af0 tx=0x7f8960005830 comp rx=0 tx=0).stop 2024-04-25T01:19:03.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.703+0000 7f897b400640 1 -- 172.21.15.90:0/934541163 reap_dead start 2024-04-25T01:19:03.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.703+0000 7f897be00640 1 -- 172.21.15.90:0/934541163 shutdown_connections 2024-04-25T01:19:03.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.703+0000 7f897be00640 1 -- 172.21.15.90:0/934541163 >> 172.21.15.90:0/934541163 conn(0x7f897c0b1560 msgr2=0x7f897c0b1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:03.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.703+0000 7f897be00640 1 -- 172.21.15.90:0/934541163 shutdown_connections 2024-04-25T01:19:03.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.703+0000 7f897be00640 1 -- 172.21.15.90:0/934541163 wait complete. 2024-04-25T01:19:03.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.705+0000 7f897be00640 1 Processor -- start 2024-04-25T01:19:03.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.705+0000 7f897be00640 1 -- start start 2024-04-25T01:19:03.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.706+0000 7f897be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8960003f30 0x7f897c1f9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:03.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.706+0000 7f897be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897c1579c0 0x7f897c1f8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:03.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.706+0000 7f897be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f897c0b3010 0x7f897c0b3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:03.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.706+0000 7f897be00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f897c163760 con 0x7f897c1579c0 2024-04-25T01:19:03.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.706+0000 7f897be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f897c072a50 con 0x7f8960003f30 2024-04-25T01:19:03.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.706+0000 7f897be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f897c0cec70 con 0x7f897c0b3010 2024-04-25T01:19:03.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.706+0000 7f8972000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897c1579c0 0x7f897c1f8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:03.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.706+0000 7f8972000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897c1579c0 0x7f897c1f8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45438/0 (socket says 172.21.15.90:45438) 2024-04-25T01:19:03.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.706+0000 7f8972000640 1 -- 172.21.15.90:0/1707484551 learned_addr learned my addr 172.21.15.90:0/1707484551 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:19:03.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.707+0000 7f8972000640 1 -- 172.21.15.90:0/1707484551 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f897c0b3010 msgr2=0x7f897c0b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:03.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.707+0000 7f8972000640 1 --2- 172.21.15.90:0/1707484551 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f897c0b3010 0x7f897c0b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.707+0000 7f8972000640 1 -- 172.21.15.90:0/1707484551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8960003f30 msgr2=0x7f897c1f9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:03.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.707+0000 7f8972000640 1 --2- 172.21.15.90:0/1707484551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8960003f30 0x7f897c1f9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.707+0000 7f8972000640 1 -- 172.21.15.90:0/1707484551 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f897c15ce60 con 0x7f897c1579c0 2024-04-25T01:19:03.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.707+0000 7f8972000640 1 --2- 172.21.15.90:0/1707484551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897c1579c0 0x7f897c1f8d30 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f8964034040 tx=0x7f8964004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:03.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.707+0000 7f8973400640 1 -- 172.21.15.90:0/1707484551 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8964040020 con 0x7f897c1579c0 2024-04-25T01:19:03.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.707+0000 7f8973400640 1 -- 172.21.15.90:0/1707484551 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8964003810 con 0x7f897c1579c0 2024-04-25T01:19:03.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.707+0000 7f8973400640 1 -- 172.21.15.90:0/1707484551 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8964036040 con 0x7f897c1579c0 2024-04-25T01:19:03.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.707+0000 7f897be00640 1 -- 172.21.15.90:0/1707484551 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f897c15d100 con 0x7f897c1579c0 2024-04-25T01:19:03.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.708+0000 7f897be00640 1 -- 172.21.15.90:0/1707484551 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f897c15d6c0 con 0x7f897c1579c0 2024-04-25T01:19:03.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.709+0000 7f897be00640 1 -- 172.21.15.90:0/1707484551 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f897c04f500 con 0x7f897c1579c0 2024-04-25T01:19:03.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.710+0000 7f8973400640 1 -- 172.21.15.90:0/1707484551 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8964043020 con 0x7f897c1579c0 2024-04-25T01:19:03.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.710+0000 7f8973400640 1 --2- 172.21.15.90:0/1707484551 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f895806be60 0x7f895806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:03.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.711+0000 7f8973400640 1 -- 172.21.15.90:0/1707484551 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5421+0+0 (secure 0 0 0) 0x7f89640aacb0 con 0x7f897c1579c0 2024-04-25T01:19:03.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.711+0000 7f897aa00640 1 --2- 172.21.15.90:0/1707484551 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f895806be60 0x7f895806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:03.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.711+0000 7f897aa00640 1 --2- 172.21.15.90:0/1707484551 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f895806be60 0x7f895806e320 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f896c006f00 tx=0x7f896c006e10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:03.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.715+0000 7f8973400640 1 -- 172.21.15.90:0/1707484551 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f896404a070 con 0x7f897c1579c0 2024-04-25T01:19:03.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.916+0000 7f897be00640 1 -- 172.21.15.90:0/1707484551 --> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f897c15d900 con 0x7f895806be60 2024-04-25T01:19:03.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.917+0000 7f8973400640 1 -- 172.21.15.90:0/1707484551 <== mgr.4104 v2:172.21.15.90:6832/2198086733 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f896c004020 con 0x7f895806be60 2024-04-25T01:19:03.899 INFO:teuthology.orchestra.run.smithi090.stderr:instructed osd(s) 3 to deep-scrub 2024-04-25T01:19:03.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.919+0000 7f897be00640 1 -- 172.21.15.90:0/1707484551 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f895806be60 msgr2=0x7f895806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:03.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.919+0000 7f897be00640 1 --2- 172.21.15.90:0/1707484551 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f895806be60 0x7f895806e320 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f896c006f00 tx=0x7f896c006e10 comp rx=0 tx=0).stop 2024-04-25T01:19:03.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.919+0000 7f897be00640 1 -- 172.21.15.90:0/1707484551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897c1579c0 msgr2=0x7f897c1f8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:03.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.919+0000 7f897be00640 1 --2- 172.21.15.90:0/1707484551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897c1579c0 0x7f897c1f8d30 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f8964034040 tx=0x7f8964004730 comp rx=0 tx=0).stop 2024-04-25T01:19:03.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.920+0000 7f897be00640 1 -- 172.21.15.90:0/1707484551 shutdown_connections 2024-04-25T01:19:03.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.920+0000 7f897be00640 1 --2- 172.21.15.90:0/1707484551 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f895806be60 0x7f895806e320 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.920+0000 7f897be00640 1 --2- 172.21.15.90:0/1707484551 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f897c0b3010 0x7f897c0b3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.920+0000 7f897be00640 1 --2- 172.21.15.90:0/1707484551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897c1579c0 0x7f897c1f8d30 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.920+0000 7f897be00640 1 --2- 172.21.15.90:0/1707484551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8960003f30 0x7f897c1f9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:03.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.920+0000 7f897be00640 1 -- 172.21.15.90:0/1707484551 >> 172.21.15.90:0/1707484551 conn(0x7f897c0b1560 msgr2=0x7f897c0b2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:03.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.920+0000 7f897be00640 1 -- 172.21.15.90:0/1707484551 shutdown_connections 2024-04-25T01:19:03.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:03.920+0000 7f897be00640 1 -- 172.21.15.90:0/1707484551 wait complete. 2024-04-25T01:19:03.922 INFO:tasks.ceph:Scrubbing osd.4 2024-04-25T01:19:03.922 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-25T01:19:04.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.144+0000 7fb296400640 1 Processor -- start 2024-04-25T01:19:04.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.145+0000 7fb296400640 1 -- start start 2024-04-25T01:19:04.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.146+0000 7fb296400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fb29015e2c0 0x7fb2900d46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:04.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.146+0000 7fb296400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb290161cd0 0x7fb290162110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:04.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.146+0000 7fb296400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2900d2b80 con 0x7fb290162650 2024-04-25T01:19:04.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.146+0000 7fb296400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fb2900b3010 con 0x7fb290161cd0 2024-04-25T01:19:04.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.146+0000 7fb296400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fb29015fad0 con 0x7fb29015e2c0 2024-04-25T01:19:04.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.146+0000 7fb28fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb290161cd0 0x7fb290162110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:04.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.146+0000 7fb295a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fb290162650 0x7fb29015e6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:42160/0 (socket says 172.21.15.90:42160) 2024-04-25T01:19:04.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.147+0000 7fb295a00640 1 -- 172.21.15.90:0/2061220033 learned_addr learned my addr 172.21.15.90:0/2061220033 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:19:04.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.147+0000 7fb295000640 1 --2- 172.21.15.90:0/2061220033 >> v2:172.21.15.103:3300/0 conn(0x7fb29015e2c0 0x7fb2900d46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:04.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.147+0000 7fb28f400640 1 -- 172.21.15.90:0/2061220033 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 739836741 0 0) 0x7fb2900d2b80 con 0x7fb290162650 2024-04-25T01:19:04.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.147+0000 7fb28f400640 1 -- 172.21.15.90:0/2061220033 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb278003520 con 0x7fb290162650 2024-04-25T01:19:04.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.147+0000 7fb28fe00640 1 -- 172.21.15.90:0/2061220033 >> v2:172.21.15.103:3300/0 conn(0x7fb29015e2c0 msgr2=0x7fb2900d46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:04.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.147+0000 7fb28fe00640 1 --2- 172.21.15.90:0/2061220033 >> v2:172.21.15.103:3300/0 conn(0x7fb29015e2c0 0x7fb2900d46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:04.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.147+0000 7fb28fe00640 1 -- 172.21.15.90:0/2061220033 >> v1:172.21.15.90:6789/0 conn(0x7fb290162650 legacy=0x7fb29015e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:04.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.147+0000 7fb28fe00640 1 -- 172.21.15.90:0/2061220033 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2900d2e60 con 0x7fb290161cd0 2024-04-25T01:19:04.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.148+0000 7fb28fe00640 1 --2- 172.21.15.90:0/2061220033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb290161cd0 0x7fb290162110 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fb290067e40 tx=0x7fb27c02a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb49343818e5d5ba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:04.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.148+0000 7fb28f400640 1 -- 172.21.15.90:0/2061220033 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27c037070 con 0x7fb290161cd0 2024-04-25T01:19:04.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.148+0000 7fb28f400640 1 -- 172.21.15.90:0/2061220033 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb27c02add0 con 0x7fb290161cd0 2024-04-25T01:19:04.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.148+0000 7fb28f400640 1 -- 172.21.15.90:0/2061220033 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27c032660 con 0x7fb290161cd0 2024-04-25T01:19:04.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.149+0000 7fb296400640 1 -- 172.21.15.90:0/2061220033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb290161cd0 msgr2=0x7fb290162110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:04.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.149+0000 7fb296400640 1 --2- 172.21.15.90:0/2061220033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb290161cd0 0x7fb290162110 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fb290067e40 tx=0x7fb27c02a080 comp rx=0 tx=0).stop 2024-04-25T01:19:04.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.149+0000 7fb296400640 1 -- 172.21.15.90:0/2061220033 shutdown_connections 2024-04-25T01:19:04.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.149+0000 7fb296400640 1 --2- 172.21.15.90:0/2061220033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb290161cd0 0x7fb290162110 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:04.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.149+0000 7fb296400640 1 --2- 172.21.15.90:0/2061220033 >> v2:172.21.15.103:3300/0 conn(0x7fb29015e2c0 0x7fb2900d46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:04.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.149+0000 7fb296400640 1 -- 172.21.15.90:0/2061220033 >> 172.21.15.90:0/2061220033 conn(0x7fb29009cff0 msgr2=0x7fb2900d3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:04.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.149+0000 7fb296400640 1 -- 172.21.15.90:0/2061220033 shutdown_connections 2024-04-25T01:19:04.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.149+0000 7fb296400640 1 -- 172.21.15.90:0/2061220033 wait complete. 2024-04-25T01:19:04.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.151+0000 7fb296400640 1 Processor -- start 2024-04-25T01:19:04.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.151+0000 7fb296400640 1 -- start start 2024-04-25T01:19:04.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.152+0000 7fb296400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb29015e2c0 0x7fb2900c9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:04.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.152+0000 7fb296400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb290161cd0 0x7fb29006c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:04.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.152+0000 7fb296400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb290162650 0x7fb29006c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:04.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.152+0000 7fb296400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb290161530 con 0x7fb29015e2c0 2024-04-25T01:19:04.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.152+0000 7fb296400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb290162ae0 con 0x7fb290161cd0 2024-04-25T01:19:04.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.152+0000 7fb296400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fb2901616b0 con 0x7fb290162650 2024-04-25T01:19:04.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.152+0000 7fb295000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb29015e2c0 0x7fb2900c9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:04.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.152+0000 7fb295000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb29015e2c0 0x7fb2900c9870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45454/0 (socket says 172.21.15.90:45454) 2024-04-25T01:19:04.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.152+0000 7fb295000640 1 -- 172.21.15.90:0/3899276671 learned_addr learned my addr 172.21.15.90:0/3899276671 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:19:04.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.153+0000 7fb28fe00640 1 --2- 172.21.15.90:0/3899276671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb290161cd0 0x7fb29006c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:04.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.153+0000 7fb295000640 1 -- 172.21.15.90:0/3899276671 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb290162650 msgr2=0x7fb29006c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:04.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.153+0000 7fb295000640 1 --2- 172.21.15.90:0/3899276671 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fb290162650 0x7fb29006c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:04.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.153+0000 7fb295000640 1 -- 172.21.15.90:0/3899276671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb290161cd0 msgr2=0x7fb29006c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:04.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.153+0000 7fb295000640 1 --2- 172.21.15.90:0/3899276671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb290161cd0 0x7fb29006c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:04.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.153+0000 7fb295000640 1 -- 172.21.15.90:0/3899276671 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2900b4ea0 con 0x7fb29015e2c0 2024-04-25T01:19:04.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.153+0000 7fb28fe00640 1 --2- 172.21.15.90:0/3899276671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb290161cd0 0x7fb29006c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:19:04.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.153+0000 7fb295000640 1 --2- 172.21.15.90:0/3899276671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb29015e2c0 0x7fb2900c9870 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fb28400d5f0 tx=0x7fb28400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:04.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.153+0000 7fb28cc00640 1 -- 172.21.15.90:0/3899276671 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2840125f0 con 0x7fb29015e2c0 2024-04-25T01:19:04.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.153+0000 7fb28cc00640 1 -- 172.21.15.90:0/3899276671 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb284019070 con 0x7fb29015e2c0 2024-04-25T01:19:04.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.153+0000 7fb28cc00640 1 -- 172.21.15.90:0/3899276671 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2840129d0 con 0x7fb29015e2c0 2024-04-25T01:19:04.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.153+0000 7fb296400640 1 -- 172.21.15.90:0/3899276671 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2901f4f70 con 0x7fb29015e2c0 2024-04-25T01:19:04.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.154+0000 7fb296400640 1 -- 172.21.15.90:0/3899276671 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2900c9db0 con 0x7fb29015e2c0 2024-04-25T01:19:04.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.155+0000 7fb296400640 1 -- 172.21.15.90:0/3899276671 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb2901f5630 con 0x7fb29015e2c0 2024-04-25T01:19:04.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.156+0000 7fb28cc00640 1 -- 172.21.15.90:0/3899276671 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb284012b70 con 0x7fb29015e2c0 2024-04-25T01:19:04.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.157+0000 7fb28cc00640 1 --2- 172.21.15.90:0/3899276671 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb27006be30 0x7fb27006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:04.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.157+0000 7fb28cc00640 1 -- 172.21.15.90:0/3899276671 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5421+0+0 (secure 0 0 0) 0x7fb28408b0c0 con 0x7fb29015e2c0 2024-04-25T01:19:04.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.157+0000 7fb28fe00640 1 --2- 172.21.15.90:0/3899276671 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb27006be30 0x7fb27006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:04.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.157+0000 7fb28cc00640 1 --2- 172.21.15.90:0/3899276671 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x7fb270072370 0x7fb2700747b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:04.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.157+0000 7fb28cc00640 1 -- 172.21.15.90:0/3899276671 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb270074e80 con 0x7fb270072370 2024-04-25T01:19:04.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.157+0000 7fb28cc00640 1 -- 172.21.15.90:0/3899276671 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7fb284092050 con 0x7fb29015e2c0 2024-04-25T01:19:04.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.158+0000 7fb28fe00640 1 --2- 172.21.15.90:0/3899276671 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb27006be30 0x7fb27006e2f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb27c034000 tx=0x7fb27c02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:04.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.158+0000 7fb295a00640 1 --2- 172.21.15.90:0/3899276671 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x7fb270072370 0x7fb2700747b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:04.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.159+0000 7fb295a00640 1 --2- 172.21.15.90:0/3899276671 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x7fb270072370 0x7fb2700747b0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:04.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.161+0000 7fb28cc00640 1 -- 172.21.15.90:0/3899276671 <== osd.4 v2:172.21.15.103:6821/1971061680 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb270074e80 con 0x7fb270072370 2024-04-25T01:19:04.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.175+0000 7fb296400640 1 -- 172.21.15.90:0/3899276671 --> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fb29004f500 con 0x7fb270072370 2024-04-25T01:19:04.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.176+0000 7fb28cc00640 1 -- 172.21.15.90:0/3899276671 <== osd.4 v2:172.21.15.103:6821/1971061680 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fb29004f500 con 0x7fb270072370 2024-04-25T01:19:04.159 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T01:19:04.159 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T01:19:04.159 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T01:19:04.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.177+0000 7fb296400640 1 -- 172.21.15.90:0/3899276671 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x7fb270072370 msgr2=0x7fb2700747b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:04.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.177+0000 7fb296400640 1 --2- 172.21.15.90:0/3899276671 >> [v2:172.21.15.103:6821/1971061680,v1:172.21.15.103:6822/1971061680] conn(0x7fb270072370 0x7fb2700747b0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:04.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.177+0000 7fb296400640 1 -- 172.21.15.90:0/3899276671 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb27006be30 msgr2=0x7fb27006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:04.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.177+0000 7fb296400640 1 --2- 172.21.15.90:0/3899276671 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fb27006be30 0x7fb27006e2f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb27c034000 tx=0x7fb27c02e000 comp rx=0 tx=0).stop 2024-04-25T01:19:04.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.177+0000 7fb296400640 1 -- 172.21.15.90:0/3899276671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb29015e2c0 msgr2=0x7fb2900c9870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:04.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.177+0000 7fb296400640 1 --2- 172.21.15.90:0/3899276671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb29015e2c0 0x7fb2900c9870 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fb28400d5f0 tx=0x7fb28400dab0 comp rx=0 tx=0).stop 2024-04-25T01:19:04.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.177+0000 7fb295a00640 1 -- 172.21.15.90:0/3899276671 reap_dead start 2024-04-25T01:19:04.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.177+0000 7fb296400640 1 -- 172.21.15.90:0/3899276671 shutdown_connections 2024-04-25T01:19:04.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.177+0000 7fb296400640 1 -- 172.21.15.90:0/3899276671 >> 172.21.15.90:0/3899276671 conn(0x7fb29009cff0 msgr2=0x7fb2900d4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:04.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.177+0000 7fb296400640 1 -- 172.21.15.90:0/3899276671 shutdown_connections 2024-04-25T01:19:04.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.177+0000 7fb296400640 1 -- 172.21.15.90:0/3899276671 wait complete. 2024-04-25T01:19:04.179 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-25T01:19:04.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.397+0000 7fecb1a00640 1 Processor -- start 2024-04-25T01:19:04.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.397+0000 7fecb1a00640 1 -- start start 2024-04-25T01:19:04.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.399+0000 7fecb1a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7fecac161cb0 0x7fecac162100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:04.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.399+0000 7fecb1a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecac0af210 con 0x7fecac0aee30 2024-04-25T01:19:04.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.399+0000 7fecb1a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecac072c60 con 0x7fecac072680 2024-04-25T01:19:04.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.399+0000 7fecb1a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7fecac162640 con 0x7fecac161cb0 2024-04-25T01:19:04.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.399+0000 7fecaaa00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fecac0aee30 0x7fecac0b3550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:42164/0 (socket says 172.21.15.90:42164) 2024-04-25T01:19:04.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.399+0000 7fecaaa00640 1 -- 172.21.15.90:0/1405734877 learned_addr learned my addr 172.21.15.90:0/1405734877 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:19:04.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.400+0000 7fecab400640 1 --2- 172.21.15.90:0/1405734877 >> v2:172.21.15.103:3300/0 conn(0x7fecac161cb0 0x7fecac162100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:04.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.400+0000 7feca9600640 1 -- 172.21.15.90:0/1405734877 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 41213063 0 0) 0x7fecac0af210 con 0x7fecac0aee30 2024-04-25T01:19:04.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.400+0000 7feca9600640 1 -- 172.21.15.90:0/1405734877 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec8c003520 con 0x7fecac0aee30 2024-04-25T01:19:04.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.400+0000 7feca9600640 1 -- 172.21.15.90:0/1405734877 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3122175262 0 0) 0x7fec8c003520 con 0x7fecac0aee30 2024-04-25T01:19:04.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.400+0000 7feca9600640 1 -- 172.21.15.90:0/1405734877 >> v2:172.21.15.103:3300/0 conn(0x7fecac161cb0 msgr2=0x7fecac162100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:04.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.400+0000 7feca9600640 1 --2- 172.21.15.90:0/1405734877 >> v2:172.21.15.103:3300/0 conn(0x7fecac161cb0 0x7fecac162100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:04.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.400+0000 7feca9600640 1 -- 172.21.15.90:0/1405734877 >> v1:172.21.15.103:6789/0 conn(0x7fecac072680 legacy=0x7fecac071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:04.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.400+0000 7feca9600640 1 -- 172.21.15.90:0/1405734877 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecac071040 con 0x7fecac0aee30 2024-04-25T01:19:04.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.400+0000 7feca9600640 1 -- 172.21.15.90:0/1405734877 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7fec9c001670 con 0x7fecac0aee30 2024-04-25T01:19:04.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.400+0000 7feca9600640 1 -- 172.21.15.90:0/1405734877 >> v1:172.21.15.90:6789/0 conn(0x7fecac0aee30 legacy=0x7fecac0b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:04.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.400+0000 7feca9600640 1 --2- 172.21.15.90:0/1405734877 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fec8c003a00 0x7fec8c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:04.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.400+0000 7feca9600640 1 --2- 172.21.15.90:0/1405734877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec8c004800 0x7fec8c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:04.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.401+0000 7feca9600640 1 --2- 172.21.15.90:0/1405734877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec8c005210 0x7fec8c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:04.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.401+0000 7feca9600640 1 -- 172.21.15.90:0/1405734877 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fecac162640 con 0x7fec8c004800 2024-04-25T01:19:04.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.401+0000 7feca9600640 1 -- 172.21.15.90:0/1405734877 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fec8c00c160 con 0x7fec8c005210 2024-04-25T01:19:04.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.401+0000 7feca9600640 1 -- 172.21.15.90:0/1405734877 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fec8c00c360 con 0x7fec8c003a00 2024-04-25T01:19:04.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.401+0000 7fecaa000640 1 --2- 172.21.15.90:0/1405734877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec8c004800 0x7fec8c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:04.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.401+0000 7fecaaa00640 1 --2- 172.21.15.90:0/1405734877 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fec8c003a00 0x7fec8c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:04.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.401+0000 7fecab400640 1 --2- 172.21.15.90:0/1405734877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec8c005210 0x7fec8c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:04.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.401+0000 7fecaa000640 1 -- 172.21.15.90:0/1405734877 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fec8c003a00 msgr2=0x7fec8c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:04.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.401+0000 7fecaa000640 1 --2- 172.21.15.90:0/1405734877 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fec8c003a00 0x7fec8c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:04.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.401+0000 7fecaa000640 1 -- 172.21.15.90:0/1405734877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec8c005210 msgr2=0x7fec8c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:04.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.401+0000 7fecaa000640 1 --2- 172.21.15.90:0/1405734877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec8c005210 0x7fec8c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:04.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.401+0000 7fecaa000640 1 -- 172.21.15.90:0/1405734877 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec8c003520 con 0x7fec8c004800 2024-04-25T01:19:04.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.401+0000 7fecab400640 1 -- 172.21.15.90:0/1405734877 reap_dead start 2024-04-25T01:19:04.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.402+0000 7fecaa000640 1 --2- 172.21.15.90:0/1405734877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec8c004800 0x7fec8c004c80 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fec94001e20 tx=0x7fec8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21f66cf7eafecc2d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:04.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.403+0000 7feca9600640 1 -- 172.21.15.90:0/1405734877 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec94001a70 con 0x7fec8c004800 2024-04-25T01:19:04.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.403+0000 7feca9600640 1 -- 172.21.15.90:0/1405734877 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec9402ae10 con 0x7fec8c004800 2024-04-25T01:19:04.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.403+0000 7feca9600640 1 -- 172.21.15.90:0/1405734877 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec94003bb0 con 0x7fec8c004800 2024-04-25T01:19:04.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.403+0000 7fecb1a00640 1 -- 172.21.15.90:0/1405734877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec8c004800 msgr2=0x7fec8c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:04.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.403+0000 7fecb1a00640 1 --2- 172.21.15.90:0/1405734877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec8c004800 0x7fec8c004c80 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fec94001e20 tx=0x7fec8c005830 comp rx=0 tx=0).stop 2024-04-25T01:19:04.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.403+0000 7fecb1a00640 1 -- 172.21.15.90:0/1405734877 shutdown_connections 2024-04-25T01:19:04.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.403+0000 7fecb1a00640 1 --2- 172.21.15.90:0/1405734877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec8c004800 0x7fec8c004c80 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:04.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.403+0000 7fecb1a00640 1 -- 172.21.15.90:0/1405734877 >> 172.21.15.90:0/1405734877 conn(0x7fecac06b140 msgr2=0x7fecac06b550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:04.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.403+0000 7fecb1a00640 1 -- 172.21.15.90:0/1405734877 shutdown_connections 2024-04-25T01:19:04.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.403+0000 7fecb1a00640 1 -- 172.21.15.90:0/1405734877 wait complete. 2024-04-25T01:19:04.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.405+0000 7fecb1a00640 1 Processor -- start 2024-04-25T01:19:04.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.405+0000 7fecb1a00640 1 -- start start 2024-04-25T01:19:04.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.406+0000 7fecb1a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec8c004800 0x7fecac15ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:04.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.406+0000 7fecb1a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecac15fba0 0x7fecac0b0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:04.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.406+0000 7fecb1a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fecac149270 0x7fecac1496f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:04.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.406+0000 7fecb1a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fecac162100 con 0x7fec8c004800 2024-04-25T01:19:04.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.406+0000 7fecb1a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fecac070d10 con 0x7fecac15fba0 2024-04-25T01:19:04.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.406+0000 7fecb1a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fecac162280 con 0x7fecac149270 2024-04-25T01:19:04.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.406+0000 7fecaaa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec8c004800 0x7fecac15ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:04.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.406+0000 7fecaaa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec8c004800 0x7fecac15ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45476/0 (socket says 172.21.15.90:45476) 2024-04-25T01:19:04.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.406+0000 7fecaaa00640 1 -- 172.21.15.90:0/2362934631 learned_addr learned my addr 172.21.15.90:0/2362934631 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:19:04.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.407+0000 7fecab400640 1 --2- 172.21.15.90:0/2362934631 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fecac149270 0x7fecac1496f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:04.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.407+0000 7fecaa000640 1 --2- 172.21.15.90:0/2362934631 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecac15fba0 0x7fecac0b0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:04.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.407+0000 7fecaaa00640 1 -- 172.21.15.90:0/2362934631 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fecac149270 msgr2=0x7fecac1496f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:04.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.407+0000 7fecaaa00640 1 --2- 172.21.15.90:0/2362934631 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fecac149270 0x7fecac1496f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:04.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.407+0000 7fecaaa00640 1 -- 172.21.15.90:0/2362934631 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecac15fba0 msgr2=0x7fecac0b0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:04.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.407+0000 7fecaaa00640 1 --2- 172.21.15.90:0/2362934631 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecac15fba0 0x7fecac0b0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:04.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.407+0000 7fecaaa00640 1 -- 172.21.15.90:0/2362934631 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecac149c30 con 0x7fec8c004800 2024-04-25T01:19:04.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.407+0000 7fecaa000640 1 --2- 172.21.15.90:0/2362934631 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecac15fba0 0x7fecac0b0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:19:04.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.407+0000 7fecaaa00640 1 --2- 172.21.15.90:0/2362934631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec8c004800 0x7fecac15ee80 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fec9c00e780 tx=0x7fec9c00ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:04.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.408+0000 7feca2a00640 1 -- 172.21.15.90:0/2362934631 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec9c00c660 con 0x7fec8c004800 2024-04-25T01:19:04.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.408+0000 7feca2a00640 1 -- 172.21.15.90:0/2362934631 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec9c018070 con 0x7fec8c004800 2024-04-25T01:19:04.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.408+0000 7feca2a00640 1 -- 172.21.15.90:0/2362934631 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec9c00ca40 con 0x7fec8c004800 2024-04-25T01:19:04.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.408+0000 7fecb1a00640 1 -- 172.21.15.90:0/2362934631 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecac1602f0 con 0x7fec8c004800 2024-04-25T01:19:04.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.408+0000 7fecb1a00640 1 -- 172.21.15.90:0/2362934631 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecac1608b0 con 0x7fec8c004800 2024-04-25T01:19:04.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.409+0000 7fecb1a00640 1 -- 172.21.15.90:0/2362934631 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecac1541b0 con 0x7fec8c004800 2024-04-25T01:19:04.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.410+0000 7feca2a00640 1 -- 172.21.15.90:0/2362934631 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fec9c00cbe0 con 0x7fec8c004800 2024-04-25T01:19:04.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.411+0000 7feca2a00640 1 --2- 172.21.15.90:0/2362934631 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fec8806be30 0x7fec8806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:04.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.411+0000 7feca2a00640 1 -- 172.21.15.90:0/2362934631 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5421+0+0 (secure 0 0 0) 0x7fec9c08aa90 con 0x7fec8c004800 2024-04-25T01:19:04.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.411+0000 7fecaa000640 1 --2- 172.21.15.90:0/2362934631 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fec8806be30 0x7fec8806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:04.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.412+0000 7fecaa000640 1 --2- 172.21.15.90:0/2362934631 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fec8806be30 0x7fec8806e2f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fec94032000 tx=0x7fec9402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:04.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.415+0000 7feca2a00640 1 -- 172.21.15.90:0/2362934631 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fecac1541b0 con 0x7fec8c004800 2024-04-25T01:19:04.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.615+0000 7fecb1a00640 1 -- 172.21.15.90:0/2362934631 --> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7fecac0cbb70 con 0x7fec8806be30 2024-04-25T01:19:04.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.615+0000 7feca2a00640 1 -- 172.21.15.90:0/2362934631 <== mgr.4104 v2:172.21.15.90:6832/2198086733 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fecac0cbb70 con 0x7fec8806be30 2024-04-25T01:19:04.598 INFO:teuthology.orchestra.run.smithi090.stderr:instructed osd(s) 4 to deep-scrub 2024-04-25T01:19:04.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.618+0000 7fecb1a00640 1 -- 172.21.15.90:0/2362934631 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fec8806be30 msgr2=0x7fec8806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:04.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.618+0000 7fecb1a00640 1 --2- 172.21.15.90:0/2362934631 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fec8806be30 0x7fec8806e2f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fec94032000 tx=0x7fec9402e000 comp rx=0 tx=0).stop 2024-04-25T01:19:04.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.618+0000 7fecb1a00640 1 -- 172.21.15.90:0/2362934631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec8c004800 msgr2=0x7fecac15ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:04.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.618+0000 7fecb1a00640 1 --2- 172.21.15.90:0/2362934631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec8c004800 0x7fecac15ee80 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fec9c00e780 tx=0x7fec9c00ec40 comp rx=0 tx=0).stop 2024-04-25T01:19:04.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.618+0000 7fecb1a00640 1 -- 172.21.15.90:0/2362934631 shutdown_connections 2024-04-25T01:19:04.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.618+0000 7fecb1a00640 1 --2- 172.21.15.90:0/2362934631 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fec8806be30 0x7fec8806e2f0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:04.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.618+0000 7fecb1a00640 1 --2- 172.21.15.90:0/2362934631 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fecac149270 0x7fecac1496f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:04.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.618+0000 7fecb1a00640 1 --2- 172.21.15.90:0/2362934631 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecac15fba0 0x7fecac0b0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:04.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.618+0000 7fecb1a00640 1 --2- 172.21.15.90:0/2362934631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec8c004800 0x7fecac15ee80 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:04.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.618+0000 7fecb1a00640 1 -- 172.21.15.90:0/2362934631 >> 172.21.15.90:0/2362934631 conn(0x7fecac06b140 msgr2=0x7fecac06b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:04.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.618+0000 7fecb1a00640 1 -- 172.21.15.90:0/2362934631 shutdown_connections 2024-04-25T01:19:04.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.618+0000 7fecb1a00640 1 -- 172.21.15.90:0/2362934631 wait complete. 2024-04-25T01:19:04.621 INFO:tasks.ceph:Scrubbing osd.5 2024-04-25T01:19:04.621 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-25T01:19:04.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.846+0000 7fd38de00640 1 Processor -- start 2024-04-25T01:19:04.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.846+0000 7fd38de00640 1 -- start start 2024-04-25T01:19:04.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.848+0000 7fd38de00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd388161710 0x7fd3880d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:04.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.848+0000 7fd38de00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3880d29d0 0x7fd3880b30e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:04.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.848+0000 7fd38de00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3880ceb20 con 0x7fd3880d1ac0 2024-04-25T01:19:04.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.848+0000 7fd38de00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fd3880ce8f0 con 0x7fd388161710 2024-04-25T01:19:04.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.848+0000 7fd38de00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd3880d10d0 con 0x7fd3880d29d0 2024-04-25T01:19:04.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.848+0000 7fd38ca00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd388161710 0x7fd3880d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:04.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.848+0000 7fd38ca00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd388161710 0x7fd3880d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49018/0 (socket says 172.21.15.90:49018) 2024-04-25T01:19:04.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.848+0000 7fd38ca00640 1 -- 172.21.15.90:0/3286425930 learned_addr learned my addr 172.21.15.90:0/3286425930 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:19:04.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.849+0000 7fd38ca00640 1 -- 172.21.15.90:0/3286425930 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3880d29d0 msgr2=0x7fd3880b30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:04.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.849+0000 7fd38d400640 1 --2- 172.21.15.90:0/3286425930 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3880d29d0 0x7fd3880b30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:04.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.849+0000 7fd38ca00640 1 --2- 172.21.15.90:0/3286425930 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3880d29d0 0x7fd3880b30e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:04.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.849+0000 7fd38ca00640 1 -- 172.21.15.90:0/3286425930 >> v1:172.21.15.103:6789/0 conn(0x7fd3880d1ac0 legacy=0x7fd3880d22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:04.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.849+0000 7fd38ca00640 1 -- 172.21.15.90:0/3286425930 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3880cedc0 con 0x7fd388161710 2024-04-25T01:19:04.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.849+0000 7fd38d400640 1 --2- 172.21.15.90:0/3286425930 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3880d29d0 0x7fd3880b30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:19:04.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.850+0000 7fd38ca00640 1 --2- 172.21.15.90:0/3286425930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd388161710 0x7fd3880d4800 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fd37c009870 tx=0x7fd37c02a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c0e51b70ff4f463 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:04.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.850+0000 7fd387400640 1 -- 172.21.15.90:0/3286425930 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd37c037070 con 0x7fd388161710 2024-04-25T01:19:04.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.850+0000 7fd387400640 1 -- 172.21.15.90:0/3286425930 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd37c032440 con 0x7fd388161710 2024-04-25T01:19:04.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.850+0000 7fd387400640 1 -- 172.21.15.90:0/3286425930 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd37c032810 con 0x7fd388161710 2024-04-25T01:19:04.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.851+0000 7fd38de00640 1 -- 172.21.15.90:0/3286425930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd388161710 msgr2=0x7fd3880d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:04.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.851+0000 7fd38de00640 1 --2- 172.21.15.90:0/3286425930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd388161710 0x7fd3880d4800 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fd37c009870 tx=0x7fd37c02a200 comp rx=0 tx=0).stop 2024-04-25T01:19:04.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.851+0000 7fd38de00640 1 -- 172.21.15.90:0/3286425930 shutdown_connections 2024-04-25T01:19:04.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.851+0000 7fd38de00640 1 --2- 172.21.15.90:0/3286425930 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3880d29d0 0x7fd3880b30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:04.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.851+0000 7fd38de00640 1 --2- 172.21.15.90:0/3286425930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd388161710 0x7fd3880d4800 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:04.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.851+0000 7fd38de00640 1 -- 172.21.15.90:0/3286425930 >> 172.21.15.90:0/3286425930 conn(0x7fd3880d3ce0 msgr2=0x7fd3880d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:04.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.851+0000 7fd38de00640 1 -- 172.21.15.90:0/3286425930 shutdown_connections 2024-04-25T01:19:04.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.851+0000 7fd38de00640 1 -- 172.21.15.90:0/3286425930 wait complete. 2024-04-25T01:19:04.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.853+0000 7fd38de00640 1 Processor -- start 2024-04-25T01:19:04.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.853+0000 7fd38de00640 1 -- start start 2024-04-25T01:19:04.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.854+0000 7fd38de00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3880d1ac0 0x7fd3880ca280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:04.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.854+0000 7fd38de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3880d29d0 0x7fd3881520f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:04.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.854+0000 7fd38de00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd388161710 0x7fd38815ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:04.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.854+0000 7fd38de00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd3880cf660 con 0x7fd3880d29d0 2024-04-25T01:19:04.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.854+0000 7fd38de00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd3880cf380 con 0x7fd388161710 2024-04-25T01:19:04.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.854+0000 7fd38de00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fd388161af0 con 0x7fd3880d1ac0 2024-04-25T01:19:04.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.854+0000 7fd387e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3880d29d0 0x7fd3881520f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:04.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.854+0000 7fd387e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3880d29d0 0x7fd3881520f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49034/0 (socket says 172.21.15.90:49034) 2024-04-25T01:19:04.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.854+0000 7fd387e00640 1 -- 172.21.15.90:0/2775863515 learned_addr learned my addr 172.21.15.90:0/2775863515 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:19:04.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.855+0000 7fd38ca00640 1 --2- 172.21.15.90:0/2775863515 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3880d1ac0 0x7fd3880ca280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:04.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.855+0000 7fd387e00640 1 -- 172.21.15.90:0/2775863515 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3880d1ac0 msgr2=0x7fd3880ca280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:04.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.855+0000 7fd387e00640 1 --2- 172.21.15.90:0/2775863515 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fd3880d1ac0 0x7fd3880ca280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:04.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.855+0000 7fd387e00640 1 -- 172.21.15.90:0/2775863515 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd388161710 msgr2=0x7fd38815ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:04.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.855+0000 7fd387e00640 1 --2- 172.21.15.90:0/2775863515 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd388161710 0x7fd38815ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:04.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.855+0000 7fd387e00640 1 -- 172.21.15.90:0/2775863515 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd38815ee60 con 0x7fd3880d29d0 2024-04-25T01:19:04.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.855+0000 7fd387e00640 1 --2- 172.21.15.90:0/2775863515 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3880d29d0 0x7fd3881520f0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fd37400c430 tx=0x7fd37400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:04.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.855+0000 7fd384c00640 1 -- 172.21.15.90:0/2775863515 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd374011790 con 0x7fd3880d29d0 2024-04-25T01:19:04.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.855+0000 7fd384c00640 1 -- 172.21.15.90:0/2775863515 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd374017070 con 0x7fd3880d29d0 2024-04-25T01:19:04.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.855+0000 7fd384c00640 1 -- 172.21.15.90:0/2775863515 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd374011bb0 con 0x7fd3880d29d0 2024-04-25T01:19:04.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.855+0000 7fd38de00640 1 -- 172.21.15.90:0/2775863515 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd38815f140 con 0x7fd3880d29d0 2024-04-25T01:19:04.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.855+0000 7fd38de00640 1 -- 172.21.15.90:0/2775863515 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd38815f700 con 0x7fd3880d29d0 2024-04-25T01:19:04.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.856+0000 7fd38de00640 1 -- 172.21.15.90:0/2775863515 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd388068b00 con 0x7fd3880d29d0 2024-04-25T01:19:04.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.857+0000 7fd384c00640 1 -- 172.21.15.90:0/2775863515 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd37401c460 con 0x7fd3880d29d0 2024-04-25T01:19:04.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.858+0000 7fd384c00640 1 --2- 172.21.15.90:0/2775863515 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd36806be30 0x7fd36806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:04.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.858+0000 7fd384c00640 1 -- 172.21.15.90:0/2775863515 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5421+0+0 (secure 0 0 0) 0x7fd37408acb0 con 0x7fd3880d29d0 2024-04-25T01:19:04.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.859+0000 7fd384c00640 1 --2- 172.21.15.90:0/2775863515 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x7fd368072370 0x7fd3680747b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:04.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.859+0000 7fd384c00640 1 -- 172.21.15.90:0/2775863515 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd374020ec0 con 0x7fd368072370 2024-04-25T01:19:04.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.859+0000 7fd384c00640 1 -- 172.21.15.90:0/2775863515 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7fd37408d250 con 0x7fd3880d29d0 2024-04-25T01:19:04.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.859+0000 7fd38ca00640 1 --2- 172.21.15.90:0/2775863515 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd36806be30 0x7fd36806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:04.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.859+0000 7fd38ca00640 1 --2- 172.21.15.90:0/2775863515 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd36806be30 0x7fd36806e2f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd37c034000 tx=0x7fd37c02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:04.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.860+0000 7fd38d400640 1 --2- 172.21.15.90:0/2775863515 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x7fd368072370 0x7fd3680747b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:04.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.861+0000 7fd38d400640 1 --2- 172.21.15.90:0/2775863515 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x7fd368072370 0x7fd3680747b0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:04.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.863+0000 7fd384c00640 1 -- 172.21.15.90:0/2775863515 <== osd.5 v2:172.21.15.103:6801/1348846168 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd374020ec0 con 0x7fd368072370 2024-04-25T01:19:04.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.877+0000 7fd38de00640 1 -- 172.21.15.90:0/2775863515 --> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fd38804f500 con 0x7fd368072370 2024-04-25T01:19:04.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.878+0000 7fd384c00640 1 -- 172.21.15.90:0/2775863515 <== osd.5 v2:172.21.15.103:6801/1348846168 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fd38804f500 con 0x7fd368072370 2024-04-25T01:19:04.861 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T01:19:04.861 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T01:19:04.861 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T01:19:04.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.879+0000 7fd38de00640 1 -- 172.21.15.90:0/2775863515 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x7fd368072370 msgr2=0x7fd3680747b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:04.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.879+0000 7fd38de00640 1 --2- 172.21.15.90:0/2775863515 >> [v2:172.21.15.103:6801/1348846168,v1:172.21.15.103:6803/1348846168] conn(0x7fd368072370 0x7fd3680747b0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:04.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.879+0000 7fd38de00640 1 -- 172.21.15.90:0/2775863515 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd36806be30 msgr2=0x7fd36806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:04.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.879+0000 7fd38de00640 1 --2- 172.21.15.90:0/2775863515 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fd36806be30 0x7fd36806e2f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd37c034000 tx=0x7fd37c02e000 comp rx=0 tx=0).stop 2024-04-25T01:19:04.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.879+0000 7fd38de00640 1 -- 172.21.15.90:0/2775863515 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3880d29d0 msgr2=0x7fd3881520f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:04.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.879+0000 7fd38de00640 1 --2- 172.21.15.90:0/2775863515 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3880d29d0 0x7fd3881520f0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fd37400c430 tx=0x7fd37400c8f0 comp rx=0 tx=0).stop 2024-04-25T01:19:04.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.879+0000 7fd38d400640 1 -- 172.21.15.90:0/2775863515 reap_dead start 2024-04-25T01:19:04.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.879+0000 7fd38de00640 1 -- 172.21.15.90:0/2775863515 shutdown_connections 2024-04-25T01:19:04.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.879+0000 7fd38de00640 1 -- 172.21.15.90:0/2775863515 >> 172.21.15.90:0/2775863515 conn(0x7fd3880d3ce0 msgr2=0x7fd3880b1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:04.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.880+0000 7fd38de00640 1 -- 172.21.15.90:0/2775863515 shutdown_connections 2024-04-25T01:19:04.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:04.880+0000 7fd38de00640 1 -- 172.21.15.90:0/2775863515 wait complete. 2024-04-25T01:19:04.882 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-25T01:19:05.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.099+0000 7ff272400640 1 Processor -- start 2024-04-25T01:19:05.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.100+0000 7ff272400640 1 -- start start 2024-04-25T01:19:05.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.101+0000 7ff272400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff26c15f340 0x7ff26c0cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:05.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.101+0000 7ff272400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7ff26c0b1410 0x7ff26c15fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.101+0000 7ff272400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff26c0cda00 con 0x7ff26c15f720 2024-04-25T01:19:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.101+0000 7ff272400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7ff26c0cd160 con 0x7ff26c0b1410 2024-04-25T01:19:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.101+0000 7ff272400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff26c0cd440 con 0x7ff26c15f340 2024-04-25T01:19:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.101+0000 7ff26be00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff26c15f720 0x7ff26c0d0180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52240/0 (socket says 172.21.15.90:52240) 2024-04-25T01:19:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.101+0000 7ff26be00640 1 -- 172.21.15.90:0/305080780 learned_addr learned my addr 172.21.15.90:0/305080780 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:19:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.102+0000 7ff271a00640 1 --2- 172.21.15.90:0/305080780 >> v2:172.21.15.103:3300/0 conn(0x7ff26c0b1410 0x7ff26c15fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.102+0000 7ff271000640 1 --2- 172.21.15.90:0/305080780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff26c15f340 0x7ff26c0cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.102+0000 7ff26b400640 1 -- 172.21.15.90:0/305080780 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 502112620 0 0) 0x7ff26c0cda00 con 0x7ff26c15f720 2024-04-25T01:19:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.102+0000 7ff26b400640 1 -- 172.21.15.90:0/305080780 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff250003520 con 0x7ff26c15f720 2024-04-25T01:19:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.102+0000 7ff26b400640 1 -- 172.21.15.90:0/305080780 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 207018700 0 0) 0x7ff250003520 con 0x7ff26c15f720 2024-04-25T01:19:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.102+0000 7ff26b400640 1 -- 172.21.15.90:0/305080780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff26c15f340 msgr2=0x7ff26c0cfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.102+0000 7ff26b400640 1 --2- 172.21.15.90:0/305080780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff26c15f340 0x7ff26c0cfc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.102+0000 7ff26b400640 1 -- 172.21.15.90:0/305080780 >> v2:172.21.15.103:3300/0 conn(0x7ff26c0b1410 msgr2=0x7ff26c15fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:05.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.102+0000 7ff26b400640 1 --2- 172.21.15.90:0/305080780 >> v2:172.21.15.103:3300/0 conn(0x7ff26c0b1410 0x7ff26c15fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:05.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.102+0000 7ff26b400640 1 -- 172.21.15.90:0/305080780 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff26c0cd720 con 0x7ff26c15f720 2024-04-25T01:19:05.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.102+0000 7ff26b400640 1 -- 172.21.15.90:0/305080780 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7ff258001670 con 0x7ff26c15f720 2024-04-25T01:19:05.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.102+0000 7ff26b400640 1 -- 172.21.15.90:0/305080780 >> v1:172.21.15.90:6789/0 conn(0x7ff26c15f720 legacy=0x7ff26c0d0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:05.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.103+0000 7ff26b400640 1 --2- 172.21.15.90:0/305080780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff26c15f340 0x7ff2500039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:05.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.103+0000 7ff26b400640 1 --2- 172.21.15.90:0/305080780 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff250003f80 0x7ff250004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:05.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.103+0000 7ff26b400640 1 --2- 172.21.15.90:0/305080780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff250004d40 0x7ff2500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:05.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.103+0000 7ff26b400640 1 -- 172.21.15.90:0/305080780 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff26c0cd160 con 0x7ff250003f80 2024-04-25T01:19:05.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.103+0000 7ff26b400640 1 -- 172.21.15.90:0/305080780 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff26c0cd440 con 0x7ff250004d40 2024-04-25T01:19:05.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.103+0000 7ff26b400640 1 -- 172.21.15.90:0/305080780 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff25000ba70 con 0x7ff26c15f340 2024-04-25T01:19:05.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.103+0000 7ff26be00640 1 --2- 172.21.15.90:0/305080780 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff250003f80 0x7ff250004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:05.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.103+0000 7ff271000640 1 --2- 172.21.15.90:0/305080780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff26c15f340 0x7ff2500039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:05.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.103+0000 7ff271a00640 1 --2- 172.21.15.90:0/305080780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff250004d40 0x7ff2500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:05.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.103+0000 7ff26be00640 1 -- 172.21.15.90:0/305080780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff26c15f340 msgr2=0x7ff2500039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:05.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.103+0000 7ff26be00640 1 --2- 172.21.15.90:0/305080780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff26c15f340 0x7ff2500039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:05.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.103+0000 7ff26be00640 1 -- 172.21.15.90:0/305080780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff250004d40 msgr2=0x7ff2500051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:05.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.103+0000 7ff26be00640 1 --2- 172.21.15.90:0/305080780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff250004d40 0x7ff2500051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:05.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.103+0000 7ff26be00640 1 -- 172.21.15.90:0/305080780 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff250003520 con 0x7ff250003f80 2024-04-25T01:19:05.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.103+0000 7ff271a00640 1 --2- 172.21.15.90:0/305080780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff250004d40 0x7ff2500051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:19:05.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.103+0000 7ff271000640 1 --2- 172.21.15.90:0/305080780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff26c15f340 0x7ff2500039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:19:05.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.104+0000 7ff26be00640 1 --2- 172.21.15.90:0/305080780 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff250003f80 0x7ff250004800 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7ff25800c160 tx=0x7ff250005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c0dc1e54633f246 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:05.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.105+0000 7ff26b400640 1 -- 172.21.15.90:0/305080780 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff258007c60 con 0x7ff250003f80 2024-04-25T01:19:05.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.105+0000 7ff26b400640 1 -- 172.21.15.90:0/305080780 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff258005af0 con 0x7ff250003f80 2024-04-25T01:19:05.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.105+0000 7ff26b400640 1 -- 172.21.15.90:0/305080780 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff258007070 con 0x7ff250003f80 2024-04-25T01:19:05.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.105+0000 7ff272400640 1 -- 172.21.15.90:0/305080780 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff250003f80 msgr2=0x7ff250004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:05.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.105+0000 7ff272400640 1 --2- 172.21.15.90:0/305080780 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff250003f80 0x7ff250004800 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7ff25800c160 tx=0x7ff250005830 comp rx=0 tx=0).stop 2024-04-25T01:19:05.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.105+0000 7ff271a00640 1 -- 172.21.15.90:0/305080780 reap_dead start 2024-04-25T01:19:05.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.105+0000 7ff272400640 1 -- 172.21.15.90:0/305080780 shutdown_connections 2024-04-25T01:19:05.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.105+0000 7ff272400640 1 -- 172.21.15.90:0/305080780 >> 172.21.15.90:0/305080780 conn(0x7ff26c0b3600 msgr2=0x7ff26c0b3a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:05.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.106+0000 7ff272400640 1 -- 172.21.15.90:0/305080780 shutdown_connections 2024-04-25T01:19:05.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.106+0000 7ff272400640 1 -- 172.21.15.90:0/305080780 wait complete. 2024-04-25T01:19:05.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.107+0000 7ff272400640 1 Processor -- start 2024-04-25T01:19:05.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.107+0000 7ff272400640 1 -- start start 2024-04-25T01:19:05.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.108+0000 7ff272400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff250003f80 0x7ff26c14dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:05.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.108+0000 7ff272400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff26c0c8b80 0x7ff26c14e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:05.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.108+0000 7ff272400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff26c0af810 0x7ff26c0afc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:05.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.108+0000 7ff272400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff26c0d0890 con 0x7ff26c0af810 2024-04-25T01:19:05.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.108+0000 7ff272400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff26c0b2d80 con 0x7ff26c0c8b80 2024-04-25T01:19:05.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.108+0000 7ff272400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7ff26c0d0a10 con 0x7ff250003f80 2024-04-25T01:19:05.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.109+0000 7ff271a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff26c0af810 0x7ff26c0afc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:05.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.109+0000 7ff271a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff26c0af810 0x7ff26c0afc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49040/0 (socket says 172.21.15.90:49040) 2024-04-25T01:19:05.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.109+0000 7ff271a00640 1 -- 172.21.15.90:0/1049985382 learned_addr learned my addr 172.21.15.90:0/1049985382 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:19:05.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.109+0000 7ff271000640 1 --2- 172.21.15.90:0/1049985382 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff250003f80 0x7ff26c14dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:05.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.109+0000 7ff26be00640 1 --2- 172.21.15.90:0/1049985382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff26c0c8b80 0x7ff26c14e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:05.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.109+0000 7ff271a00640 1 -- 172.21.15.90:0/1049985382 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff250003f80 msgr2=0x7ff26c14dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:05.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.109+0000 7ff271a00640 1 --2- 172.21.15.90:0/1049985382 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff250003f80 0x7ff26c14dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:05.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.109+0000 7ff271a00640 1 -- 172.21.15.90:0/1049985382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff26c0c8b80 msgr2=0x7ff26c14e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:05.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.109+0000 7ff271a00640 1 --2- 172.21.15.90:0/1049985382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff26c0c8b80 0x7ff26c14e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:05.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.109+0000 7ff271a00640 1 -- 172.21.15.90:0/1049985382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff26c0af150 con 0x7ff26c0af810 2024-04-25T01:19:05.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.109+0000 7ff26be00640 1 --2- 172.21.15.90:0/1049985382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff26c0c8b80 0x7ff26c14e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:19:05.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.109+0000 7ff271000640 1 --2- 172.21.15.90:0/1049985382 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff250003f80 0x7ff26c14dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:19:05.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.109+0000 7ff271a00640 1 --2- 172.21.15.90:0/1049985382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff26c0af810 0x7ff26c0afc90 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7ff26c0d1710 tx=0x7ff26400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:05.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.110+0000 7ff268c00640 1 -- 172.21.15.90:0/1049985382 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff26401c070 con 0x7ff26c0af810 2024-04-25T01:19:05.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.110+0000 7ff268c00640 1 -- 172.21.15.90:0/1049985382 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff264017400 con 0x7ff26c0af810 2024-04-25T01:19:05.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.110+0000 7ff268c00640 1 -- 172.21.15.90:0/1049985382 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2640177d0 con 0x7ff26c0af810 2024-04-25T01:19:05.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.110+0000 7ff272400640 1 -- 172.21.15.90:0/1049985382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff26c0c8fe0 con 0x7ff26c0af810 2024-04-25T01:19:05.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.110+0000 7ff272400640 1 -- 172.21.15.90:0/1049985382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff26c0c7ff0 con 0x7ff26c0af810 2024-04-25T01:19:05.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.111+0000 7ff272400640 1 -- 172.21.15.90:0/1049985382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff26c04f500 con 0x7ff26c0af810 2024-04-25T01:19:05.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.112+0000 7ff268c00640 1 -- 172.21.15.90:0/1049985382 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff264005d10 con 0x7ff26c0af810 2024-04-25T01:19:05.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.113+0000 7ff268c00640 1 --2- 172.21.15.90:0/1049985382 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff24c06be30 0x7ff24c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:05.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.113+0000 7ff268c00640 1 -- 172.21.15.90:0/1049985382 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5421+0+0 (secure 0 0 0) 0x7ff26408aff0 con 0x7ff26c0af810 2024-04-25T01:19:05.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.113+0000 7ff271000640 1 --2- 172.21.15.90:0/1049985382 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff24c06be30 0x7ff24c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:05.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.114+0000 7ff271000640 1 --2- 172.21.15.90:0/1049985382 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff24c06be30 0x7ff24c06e2f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff25c0099c0 tx=0x7ff25c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:05.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.117+0000 7ff268c00640 1 -- 172.21.15.90:0/1049985382 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff264012070 con 0x7ff26c0af810 2024-04-25T01:19:05.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.315+0000 7ff272400640 1 -- 172.21.15.90:0/1049985382 --> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7ff26c0b7a30 con 0x7ff24c06be30 2024-04-25T01:19:05.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.316+0000 7ff268c00640 1 -- 172.21.15.90:0/1049985382 <== mgr.4104 v2:172.21.15.90:6832/2198086733 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ff26c0b7a30 con 0x7ff24c06be30 2024-04-25T01:19:05.298 INFO:teuthology.orchestra.run.smithi090.stderr:instructed osd(s) 5 to deep-scrub 2024-04-25T01:19:05.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.318+0000 7ff272400640 1 -- 172.21.15.90:0/1049985382 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff24c06be30 msgr2=0x7ff24c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:05.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.318+0000 7ff272400640 1 --2- 172.21.15.90:0/1049985382 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff24c06be30 0x7ff24c06e2f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff25c0099c0 tx=0x7ff25c008040 comp rx=0 tx=0).stop 2024-04-25T01:19:05.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.318+0000 7ff272400640 1 -- 172.21.15.90:0/1049985382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff26c0af810 msgr2=0x7ff26c0afc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:05.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.318+0000 7ff272400640 1 --2- 172.21.15.90:0/1049985382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff26c0af810 0x7ff26c0afc90 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7ff26c0d1710 tx=0x7ff26400f4a0 comp rx=0 tx=0).stop 2024-04-25T01:19:05.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.318+0000 7ff272400640 1 -- 172.21.15.90:0/1049985382 shutdown_connections 2024-04-25T01:19:05.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.318+0000 7ff272400640 1 --2- 172.21.15.90:0/1049985382 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7ff24c06be30 0x7ff24c06e2f0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:05.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.318+0000 7ff272400640 1 --2- 172.21.15.90:0/1049985382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff26c0af810 0x7ff26c0afc90 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:05.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.318+0000 7ff272400640 1 --2- 172.21.15.90:0/1049985382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff26c0c8b80 0x7ff26c14e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:05.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.318+0000 7ff272400640 1 --2- 172.21.15.90:0/1049985382 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7ff250003f80 0x7ff26c14dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:05.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.318+0000 7ff272400640 1 -- 172.21.15.90:0/1049985382 >> 172.21.15.90:0/1049985382 conn(0x7ff26c0b3600 msgr2=0x7ff26c0b3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:05.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.318+0000 7ff272400640 1 -- 172.21.15.90:0/1049985382 shutdown_connections 2024-04-25T01:19:05.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.318+0000 7ff272400640 1 -- 172.21.15.90:0/1049985382 wait complete. 2024-04-25T01:19:05.319 INFO:tasks.ceph:Scrubbing osd.6 2024-04-25T01:19:05.320 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-25T01:19:05.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.542+0000 7f1c9be00640 1 Processor -- start 2024-04-25T01:19:05.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.542+0000 7f1c9be00640 1 -- start start 2024-04-25T01:19:05.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.544+0000 7f1c9be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c9c0cfbf0 0x7f1c9c0b2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:05.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.544+0000 7f1c9be00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1c9c06d010 0x7f1c9c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:05.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.544+0000 7f1c9be00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c9c15e980 con 0x7f1c9c0ce360 2024-04-25T01:19:05.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.544+0000 7f1c9be00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1c9c0b3450 con 0x7f1c9c06d010 2024-04-25T01:19:05.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.544+0000 7f1c9be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1c9c0b1190 con 0x7f1c9c0cfbf0 2024-04-25T01:19:05.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.544+0000 7f1c9aa00640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f1c9c0ce360 0x7f1c9c0ce740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.90:47382/0 (socket says 172.21.15.90:47382) 2024-04-25T01:19:05.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.544+0000 7f1c92000640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c9c0cfbf0 0x7f1c9c0b2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:05.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.544+0000 7f1c9aa00640 1 -- 172.21.15.90:0/1437130753 learned_addr learned my addr 172.21.15.90:0/1437130753 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:19:05.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.545+0000 7f1c9b400640 1 --2- 172.21.15.90:0/1437130753 >> v2:172.21.15.103:3300/0 conn(0x7f1c9c06d010 0x7f1c9c06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:05.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.545+0000 7f1c9a000640 1 -- 172.21.15.90:0/1437130753 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 7598106 0 0) 0x7f1c9c15e980 con 0x7f1c9c0ce360 2024-04-25T01:19:05.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.545+0000 7f1c9a000640 1 -- 172.21.15.90:0/1437130753 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c80003520 con 0x7f1c9c0ce360 2024-04-25T01:19:05.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.545+0000 7f1c9b400640 1 -- 172.21.15.90:0/1437130753 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c9c0cfbf0 msgr2=0x7f1c9c0b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:05.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.545+0000 7f1c9b400640 1 --2- 172.21.15.90:0/1437130753 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c9c0cfbf0 0x7f1c9c0b2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:05.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.545+0000 7f1c9b400640 1 -- 172.21.15.90:0/1437130753 >> v1:172.21.15.103:6789/0 conn(0x7f1c9c0ce360 legacy=0x7f1c9c0ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:05.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.545+0000 7f1c9b400640 1 -- 172.21.15.90:0/1437130753 --> v2:172.21.15.103:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c9c0b25f0 con 0x7f1c9c06d010 2024-04-25T01:19:05.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.545+0000 7f1c92000640 1 --2- 172.21.15.90:0/1437130753 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c9c0cfbf0 0x7f1c9c0b2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:19:05.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.547+0000 7f1c9b400640 1 --2- 172.21.15.90:0/1437130753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c9c06d010 0x7f1c9c06d4a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f1c940097a0 tx=0x7f1c9402a0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac26c5db9a7741b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:05.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.547+0000 7f1c9a000640 1 -- 172.21.15.90:0/1437130753 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c94037070 con 0x7f1c9c06d010 2024-04-25T01:19:05.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.547+0000 7f1c9a000640 1 -- 172.21.15.90:0/1437130753 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c9402ae40 con 0x7f1c9c06d010 2024-04-25T01:19:05.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.547+0000 7f1c9a000640 1 -- 172.21.15.90:0/1437130753 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c94032660 con 0x7f1c9c06d010 2024-04-25T01:19:05.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.548+0000 7f1c9be00640 1 -- 172.21.15.90:0/1437130753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c9c06d010 msgr2=0x7f1c9c06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:05.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.548+0000 7f1c9be00640 1 --2- 172.21.15.90:0/1437130753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c9c06d010 0x7f1c9c06d4a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f1c940097a0 tx=0x7f1c9402a0f0 comp rx=0 tx=0).stop 2024-04-25T01:19:05.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.548+0000 7f1c9be00640 1 -- 172.21.15.90:0/1437130753 shutdown_connections 2024-04-25T01:19:05.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.548+0000 7f1c9be00640 1 --2- 172.21.15.90:0/1437130753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c9c06d010 0x7f1c9c06d4a0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:05.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.548+0000 7f1c9be00640 1 --2- 172.21.15.90:0/1437130753 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c9c0cfbf0 0x7f1c9c0b2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:05.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.548+0000 7f1c9be00640 1 -- 172.21.15.90:0/1437130753 >> 172.21.15.90:0/1437130753 conn(0x7f1c9c0d1ae0 msgr2=0x7f1c9c0d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:05.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.548+0000 7f1c9be00640 1 -- 172.21.15.90:0/1437130753 shutdown_connections 2024-04-25T01:19:05.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.548+0000 7f1c9be00640 1 -- 172.21.15.90:0/1437130753 wait complete. 2024-04-25T01:19:05.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.550+0000 7f1c9be00640 1 Processor -- start 2024-04-25T01:19:05.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.550+0000 7f1c9be00640 1 -- start start 2024-04-25T01:19:05.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.551+0000 7f1c9be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c9c06d010 0x7f1c9c15d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:05.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.551+0000 7f1c9be00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c9c0ce360 0x7f1c9c0ae770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:05.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.551+0000 7f1c9be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c9c0cfbf0 0x7f1c9c0b5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:05.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.551+0000 7f1c9be00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c06caf0 con 0x7f1c9c06d010 2024-04-25T01:19:05.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.551+0000 7f1c9be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c0b2890 con 0x7f1c9c0cfbf0 2024-04-25T01:19:05.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.551+0000 7f1c9be00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1c9c0cddb0 con 0x7f1c9c0ce360 2024-04-25T01:19:05.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.551+0000 7f1c9aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c9c06d010 0x7f1c9c15d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:05.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.551+0000 7f1c9aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c9c06d010 0x7f1c9c15d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49046/0 (socket says 172.21.15.90:49046) 2024-04-25T01:19:05.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.551+0000 7f1c9aa00640 1 -- 172.21.15.90:0/2412319462 learned_addr learned my addr 172.21.15.90:0/2412319462 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:19:05.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.552+0000 7f1c92000640 1 --2- 172.21.15.90:0/2412319462 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c9c0ce360 0x7f1c9c0ae770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:05.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.552+0000 7f1c9b400640 1 --2- 172.21.15.90:0/2412319462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c9c0cfbf0 0x7f1c9c0b5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:05.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.552+0000 7f1c9aa00640 1 -- 172.21.15.90:0/2412319462 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c9c0ce360 msgr2=0x7f1c9c0ae770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:05.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.552+0000 7f1c9aa00640 1 --2- 172.21.15.90:0/2412319462 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1c9c0ce360 0x7f1c9c0ae770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:05.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.552+0000 7f1c9aa00640 1 -- 172.21.15.90:0/2412319462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c9c0cfbf0 msgr2=0x7f1c9c0b5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:05.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.552+0000 7f1c9aa00640 1 --2- 172.21.15.90:0/2412319462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c9c0cfbf0 0x7f1c9c0b5ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:05.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.552+0000 7f1c9aa00640 1 -- 172.21.15.90:0/2412319462 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c9c0aeee0 con 0x7f1c9c06d010 2024-04-25T01:19:05.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.552+0000 7f1c9aa00640 1 --2- 172.21.15.90:0/2412319462 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c9c06d010 0x7f1c9c15d510 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c00caa0 tx=0x7f1c8c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:05.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.552+0000 7f1c93400640 1 -- 172.21.15.90:0/2412319462 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c017070 con 0x7f1c9c06d010 2024-04-25T01:19:05.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.552+0000 7f1c93400640 1 -- 172.21.15.90:0/2412319462 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c8c00ac00 con 0x7f1c9c06d010 2024-04-25T01:19:05.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.552+0000 7f1c93400640 1 -- 172.21.15.90:0/2412319462 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c012400 con 0x7f1c9c06d010 2024-04-25T01:19:05.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.552+0000 7f1c9be00640 1 -- 172.21.15.90:0/2412319462 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c9c0af1c0 con 0x7f1c9c06d010 2024-04-25T01:19:05.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.553+0000 7f1c9be00640 1 -- 172.21.15.90:0/2412319462 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c9c0af780 con 0x7f1c9c06d010 2024-04-25T01:19:05.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.553+0000 7f1c9be00640 1 -- 172.21.15.90:0/2412319462 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1c9c0b6220 con 0x7f1c9c06d010 2024-04-25T01:19:05.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.555+0000 7f1c93400640 1 -- 172.21.15.90:0/2412319462 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1c8c01a070 con 0x7f1c9c06d010 2024-04-25T01:19:05.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.556+0000 7f1c93400640 1 --2- 172.21.15.90:0/2412319462 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1c7806be30 0x7f1c7806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:05.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.556+0000 7f1c93400640 1 -- 172.21.15.90:0/2412319462 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5421+0+0 (secure 0 0 0) 0x7f1c8c08b2f0 con 0x7f1c9c06d010 2024-04-25T01:19:05.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.556+0000 7f1c92000640 1 --2- 172.21.15.90:0/2412319462 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1c7806be30 0x7f1c7806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:05.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.556+0000 7f1c93400640 1 --2- 172.21.15.90:0/2412319462 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x7f1c78072370 0x7f1c780747b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:05.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.556+0000 7f1c93400640 1 -- 172.21.15.90:0/2412319462 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1c8c016ec0 con 0x7f1c78072370 2024-04-25T01:19:05.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.556+0000 7f1c93400640 1 -- 172.21.15.90:0/2412319462 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f1c8c08b6a0 con 0x7f1c9c06d010 2024-04-25T01:19:05.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.557+0000 7f1c92000640 1 --2- 172.21.15.90:0/2412319462 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1c7806be30 0x7f1c7806e2f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1c84009560 tx=0x7f1c84008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:05.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.557+0000 7f1c9b400640 1 --2- 172.21.15.90:0/2412319462 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x7f1c78072370 0x7f1c780747b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:05.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.558+0000 7f1c9b400640 1 --2- 172.21.15.90:0/2412319462 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x7f1c78072370 0x7f1c780747b0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:05.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.560+0000 7f1c93400640 1 -- 172.21.15.90:0/2412319462 <== osd.6 v2:172.21.15.103:6816/2008388906 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1c8c016ec0 con 0x7f1c78072370 2024-04-25T01:19:05.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.574+0000 7f1c9be00640 1 -- 172.21.15.90:0/2412319462 --> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f1c9c04f500 con 0x7f1c78072370 2024-04-25T01:19:05.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.576+0000 7f1c93400640 1 -- 172.21.15.90:0/2412319462 <== osd.6 v2:172.21.15.103:6816/2008388906 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f1c9c04f500 con 0x7f1c78072370 2024-04-25T01:19:05.559 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T01:19:05.559 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T01:19:05.559 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T01:19:05.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.576+0000 7f1c9be00640 1 -- 172.21.15.90:0/2412319462 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x7f1c78072370 msgr2=0x7f1c780747b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:05.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.576+0000 7f1c9be00640 1 --2- 172.21.15.90:0/2412319462 >> [v2:172.21.15.103:6816/2008388906,v1:172.21.15.103:6817/2008388906] conn(0x7f1c78072370 0x7f1c780747b0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:05.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.576+0000 7f1c9be00640 1 -- 172.21.15.90:0/2412319462 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1c7806be30 msgr2=0x7f1c7806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:05.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.576+0000 7f1c9be00640 1 --2- 172.21.15.90:0/2412319462 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1c7806be30 0x7f1c7806e2f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1c84009560 tx=0x7f1c84008040 comp rx=0 tx=0).stop 2024-04-25T01:19:05.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.576+0000 7f1c9be00640 1 -- 172.21.15.90:0/2412319462 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c9c06d010 msgr2=0x7f1c9c15d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:05.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.576+0000 7f1c9be00640 1 --2- 172.21.15.90:0/2412319462 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c9c06d010 0x7f1c9c15d510 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c00caa0 tx=0x7f1c8c00cf60 comp rx=0 tx=0).stop 2024-04-25T01:19:05.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.576+0000 7f1c9b400640 1 -- 172.21.15.90:0/2412319462 reap_dead start 2024-04-25T01:19:05.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.576+0000 7f1c9be00640 1 -- 172.21.15.90:0/2412319462 shutdown_connections 2024-04-25T01:19:05.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.576+0000 7f1c9be00640 1 -- 172.21.15.90:0/2412319462 >> 172.21.15.90:0/2412319462 conn(0x7f1c9c0d1ae0 msgr2=0x7f1c9c157310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:05.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.576+0000 7f1c9be00640 1 -- 172.21.15.90:0/2412319462 shutdown_connections 2024-04-25T01:19:05.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.576+0000 7f1c9be00640 1 -- 172.21.15.90:0/2412319462 wait complete. 2024-04-25T01:19:05.576 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-25T01:19:05.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.790+0000 7f0073e00640 1 Processor -- start 2024-04-25T01:19:05.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.790+0000 7f0073e00640 1 -- start start 2024-04-25T01:19:05.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.792+0000 7f0073e00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f00740af840 0x7f00740d0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:05.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.792+0000 7f0073e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f007406eed0 0x7f007406f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:05.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.792+0000 7f0073e00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f007406f880 con 0x7f007415be40 2024-04-25T01:19:05.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.792+0000 7f0073e00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f00741608d0 con 0x7f00740af840 2024-04-25T01:19:05.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.792+0000 7f0073e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f007415d230 con 0x7f007406eed0 2024-04-25T01:19:05.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.792+0000 7f0073400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f007406eed0 0x7f007406f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:05.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.792+0000 7f0073400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f007406eed0 0x7f007406f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3301/0 says I am v2:172.21.15.90:34956/0 (socket says 172.21.15.90:34956) 2024-04-25T01:19:05.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.792+0000 7f0073400640 1 -- 172.21.15.90:0/2965229981 learned_addr learned my addr 172.21.15.90:0/2965229981 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:19:05.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.792+0000 7f006a000640 1 --2- 172.21.15.90:0/2965229981 >> v2:172.21.15.103:3300/0 conn(0x7f00740af840 0x7f00740d0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:05.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.793+0000 7f0073400640 1 -- 172.21.15.90:0/2965229981 >> v2:172.21.15.103:3300/0 conn(0x7f00740af840 msgr2=0x7f00740d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:05.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.793+0000 7f0073400640 1 --2- 172.21.15.90:0/2965229981 >> v2:172.21.15.103:3300/0 conn(0x7f00740af840 0x7f00740d0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:05.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.793+0000 7f0073400640 1 -- 172.21.15.90:0/2965229981 >> v1:172.21.15.103:6789/0 conn(0x7f007415be40 legacy=0x7f00741601c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:05.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.793+0000 7f0073400640 1 -- 172.21.15.90:0/2965229981 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00740afcd0 con 0x7f007406eed0 2024-04-25T01:19:05.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.794+0000 7f0073400640 1 --2- 172.21.15.90:0/2965229981 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f007406eed0 0x7f007406f340 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f006c009e50 tx=0x7f006c02a6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47e94b14aadc81c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:05.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.794+0000 7f0072000640 1 -- 172.21.15.90:0/2965229981 <== mon.2 v2:172.21.15.103:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f006c036070 con 0x7f007406eed0 2024-04-25T01:19:05.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.794+0000 7f0072000640 1 -- 172.21.15.90:0/2965229981 <== mon.2 v2:172.21.15.103:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f006c031600 con 0x7f007406eed0 2024-04-25T01:19:05.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.794+0000 7f0072000640 1 -- 172.21.15.90:0/2965229981 <== mon.2 v2:172.21.15.103:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f006c0319d0 con 0x7f007406eed0 2024-04-25T01:19:05.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.795+0000 7f0073e00640 1 -- 172.21.15.90:0/2965229981 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f007406eed0 msgr2=0x7f007406f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:05.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.795+0000 7f0073e00640 1 --2- 172.21.15.90:0/2965229981 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f007406eed0 0x7f007406f340 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f006c009e50 tx=0x7f006c02a6c0 comp rx=0 tx=0).stop 2024-04-25T01:19:05.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.795+0000 7f0073e00640 1 -- 172.21.15.90:0/2965229981 shutdown_connections 2024-04-25T01:19:05.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.795+0000 7f0073e00640 1 --2- 172.21.15.90:0/2965229981 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f007406eed0 0x7f007406f340 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:05.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.795+0000 7f0073e00640 1 --2- 172.21.15.90:0/2965229981 >> v2:172.21.15.103:3300/0 conn(0x7f00740af840 0x7f00740d0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:05.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.795+0000 7f0073e00640 1 -- 172.21.15.90:0/2965229981 >> 172.21.15.90:0/2965229981 conn(0x7f00740b25e0 msgr2=0x7f00740b29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:05.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.795+0000 7f0073e00640 1 -- 172.21.15.90:0/2965229981 shutdown_connections 2024-04-25T01:19:05.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.795+0000 7f0073e00640 1 -- 172.21.15.90:0/2965229981 wait complete. 2024-04-25T01:19:05.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.797+0000 7f0073e00640 1 Processor -- start 2024-04-25T01:19:05.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.797+0000 7f0073e00640 1 -- start start 2024-04-25T01:19:05.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.798+0000 7f0073e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f007406eed0 0x7f00741f5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:05.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.798+0000 7f0073e00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f007415be40 0x7f0074157950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:05.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.798+0000 7f0073e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f007414a590 0x7f007415e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:05.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.798+0000 7f0073e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f007415fca0 con 0x7f007414a590 2024-04-25T01:19:05.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.798+0000 7f0073e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f007415f9c0 con 0x7f007406eed0 2024-04-25T01:19:05.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.798+0000 7f0073e00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f00740b0290 con 0x7f007415be40 2024-04-25T01:19:05.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.799+0000 7f0073400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f007414a590 0x7f007415e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:05.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.799+0000 7f0073400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f007414a590 0x7f007415e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49048/0 (socket says 172.21.15.90:49048) 2024-04-25T01:19:05.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.799+0000 7f0073400640 1 -- 172.21.15.90:0/1904044901 learned_addr learned my addr 172.21.15.90:0/1904044901 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:19:05.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.799+0000 7f006a000640 1 --2- 172.21.15.90:0/1904044901 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f007415be40 0x7f0074157950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:05.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.799+0000 7f0072a00640 1 --2- 172.21.15.90:0/1904044901 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f007406eed0 0x7f00741f5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:05.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.799+0000 7f0073400640 1 -- 172.21.15.90:0/1904044901 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f007415be40 msgr2=0x7f0074157950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:05.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.799+0000 7f0073400640 1 --2- 172.21.15.90:0/1904044901 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f007415be40 0x7f0074157950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:05.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.799+0000 7f0073400640 1 -- 172.21.15.90:0/1904044901 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f007406eed0 msgr2=0x7f00741f5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:05.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.799+0000 7f0073400640 1 --2- 172.21.15.90:0/1904044901 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f007406eed0 0x7f00741f5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:05.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.799+0000 7f0073400640 1 -- 172.21.15.90:0/1904044901 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0074155f50 con 0x7f007414a590 2024-04-25T01:19:05.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.799+0000 7f006a000640 1 --2- 172.21.15.90:0/1904044901 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f007415be40 0x7f0074157950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:19:05.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.799+0000 7f0073400640 1 --2- 172.21.15.90:0/1904044901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f007414a590 0x7f007415e420 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f006c009f70 tx=0x7f006c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:05.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.800+0000 7f006b400640 1 -- 172.21.15.90:0/1904044901 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f006c03f020 con 0x7f007414a590 2024-04-25T01:19:05.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.800+0000 7f006b400640 1 -- 172.21.15.90:0/1904044901 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f006c037040 con 0x7f007414a590 2024-04-25T01:19:05.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.800+0000 7f006b400640 1 -- 172.21.15.90:0/1904044901 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f006c036070 con 0x7f007414a590 2024-04-25T01:19:05.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.800+0000 7f0073e00640 1 -- 172.21.15.90:0/1904044901 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00741561c0 con 0x7f007414a590 2024-04-25T01:19:05.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.800+0000 7f0073e00640 1 -- 172.21.15.90:0/1904044901 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0074156780 con 0x7f007414a590 2024-04-25T01:19:05.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.801+0000 7f0073e00640 1 -- 172.21.15.90:0/1904044901 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f007404f500 con 0x7f007414a590 2024-04-25T01:19:05.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.802+0000 7f006b400640 1 -- 172.21.15.90:0/1904044901 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f006c042020 con 0x7f007414a590 2024-04-25T01:19:05.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.803+0000 7f006b400640 1 --2- 172.21.15.90:0/1904044901 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f005006be30 0x7f005006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:05.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.803+0000 7f006b400640 1 -- 172.21.15.90:0/1904044901 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5421+0+0 (secure 0 0 0) 0x7f006c0ab2a0 con 0x7f007414a590 2024-04-25T01:19:05.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.803+0000 7f0072a00640 1 --2- 172.21.15.90:0/1904044901 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f005006be30 0x7f005006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:05.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.804+0000 7f0072a00640 1 --2- 172.21.15.90:0/1904044901 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f005006be30 0x7f005006e2f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0064006fd0 tx=0x7f0064008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:05.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:05.807+0000 7f006b400640 1 -- 172.21.15.90:0/1904044901 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f006c049030 con 0x7f007414a590 2024-04-25T01:19:05.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.004+0000 7f0073e00640 1 -- 172.21.15.90:0/1904044901 --> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f00740b6920 con 0x7f005006be30 2024-04-25T01:19:05.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.005+0000 7f006b400640 1 -- 172.21.15.90:0/1904044901 <== mgr.4104 v2:172.21.15.90:6832/2198086733 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f00740b6920 con 0x7f005006be30 2024-04-25T01:19:05.988 INFO:teuthology.orchestra.run.smithi090.stderr:instructed osd(s) 6 to deep-scrub 2024-04-25T01:19:05.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.008+0000 7f0073e00640 1 -- 172.21.15.90:0/1904044901 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f005006be30 msgr2=0x7f005006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:05.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.008+0000 7f0073e00640 1 --2- 172.21.15.90:0/1904044901 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f005006be30 0x7f005006e2f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0064006fd0 tx=0x7f0064008040 comp rx=0 tx=0).stop 2024-04-25T01:19:05.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.008+0000 7f0073e00640 1 -- 172.21.15.90:0/1904044901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f007414a590 msgr2=0x7f007415e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:05.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.008+0000 7f0073e00640 1 --2- 172.21.15.90:0/1904044901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f007414a590 0x7f007415e420 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f006c009f70 tx=0x7f006c004a40 comp rx=0 tx=0).stop 2024-04-25T01:19:05.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.008+0000 7f0073e00640 1 -- 172.21.15.90:0/1904044901 shutdown_connections 2024-04-25T01:19:05.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.008+0000 7f0073e00640 1 --2- 172.21.15.90:0/1904044901 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f005006be30 0x7f005006e2f0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:05.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.008+0000 7f0073e00640 1 --2- 172.21.15.90:0/1904044901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f007414a590 0x7f007415e420 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:05.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.008+0000 7f0073e00640 1 --2- 172.21.15.90:0/1904044901 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f007415be40 0x7f0074157950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:05.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.008+0000 7f0073e00640 1 --2- 172.21.15.90:0/1904044901 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f007406eed0 0x7f00741f5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:05.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.008+0000 7f0073e00640 1 -- 172.21.15.90:0/1904044901 >> 172.21.15.90:0/1904044901 conn(0x7f00740b25e0 msgr2=0x7f007406bab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:05.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.008+0000 7f0073e00640 1 -- 172.21.15.90:0/1904044901 shutdown_connections 2024-04-25T01:19:05.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.008+0000 7f0073e00640 1 -- 172.21.15.90:0/1904044901 wait complete. 2024-04-25T01:19:06.009 INFO:tasks.ceph:Scrubbing osd.7 2024-04-25T01:19:06.009 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-25T01:19:06.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.227+0000 7f2864a00640 1 Processor -- start 2024-04-25T01:19:06.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.228+0000 7f2864a00640 1 -- start start 2024-04-25T01:19:06.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.229+0000 7f2864a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2860071f10 0x7f28600cc200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:06.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.229+0000 7f2864a00640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f286006e230 0x7f286006e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:06.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.229+0000 7f2864a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f286006dfe0 con 0x7f286006db30 2024-04-25T01:19:06.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.229+0000 7f2864a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f286015d2a0 con 0x7f2860071f10 2024-04-25T01:19:06.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.229+0000 7f2864a00640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f28600722f0 con 0x7f286006e230 2024-04-25T01:19:06.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.229+0000 7f285f400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2860071f10 0x7f28600cc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:06.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.229+0000 7f2857e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f286006db30 0x7f286015cb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52250/0 (socket says 172.21.15.90:52250) 2024-04-25T01:19:06.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.229+0000 7f2857e00640 1 -- 172.21.15.90:0/2379871337 learned_addr learned my addr 172.21.15.90:0/2379871337 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:19:06.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.230+0000 7f285fe00640 1 --2- 172.21.15.90:0/2379871337 >> v2:172.21.15.103:3300/0 conn(0x7f286006e230 0x7f286006e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:06.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.230+0000 7f285f400640 1 -- 172.21.15.90:0/2379871337 >> v2:172.21.15.103:3300/0 conn(0x7f286006e230 msgr2=0x7f286006e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:06.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.230+0000 7f285f400640 1 --2- 172.21.15.90:0/2379871337 >> v2:172.21.15.103:3300/0 conn(0x7f286006e230 0x7f286006e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.230+0000 7f285f400640 1 -- 172.21.15.90:0/2379871337 >> v1:172.21.15.90:6789/0 conn(0x7f286006db30 legacy=0x7f286015cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:06.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.230+0000 7f285f400640 1 -- 172.21.15.90:0/2379871337 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2860160420 con 0x7f2860071f10 2024-04-25T01:19:06.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.230+0000 7f285ea00640 1 -- 172.21.15.90:0/2379871337 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3957374532 0 0) 0x7f286006dfe0 con 0x7f286006db30 2024-04-25T01:19:06.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.230+0000 7f285fe00640 1 --2- 172.21.15.90:0/2379871337 >> v2:172.21.15.103:3300/0 conn(0x7f286006e230 0x7f286006e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:19:06.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.231+0000 7f285f400640 1 --2- 172.21.15.90:0/2379871337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2860071f10 0x7f28600cc200 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f28600b2250 tx=0x7f285002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48462b254c5db86d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:06.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.231+0000 7f285ea00640 1 -- 172.21.15.90:0/2379871337 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2850037070 con 0x7f2860071f10 2024-04-25T01:19:06.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.231+0000 7f285ea00640 1 -- 172.21.15.90:0/2379871337 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2850032440 con 0x7f2860071f10 2024-04-25T01:19:06.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.231+0000 7f285ea00640 1 -- 172.21.15.90:0/2379871337 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2850032810 con 0x7f2860071f10 2024-04-25T01:19:06.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.232+0000 7f2864a00640 1 -- 172.21.15.90:0/2379871337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2860071f10 msgr2=0x7f28600cc200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:06.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.232+0000 7f2864a00640 1 --2- 172.21.15.90:0/2379871337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2860071f10 0x7f28600cc200 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f28600b2250 tx=0x7f285002a1a0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.232+0000 7f2864a00640 1 -- 172.21.15.90:0/2379871337 shutdown_connections 2024-04-25T01:19:06.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.232+0000 7f2864a00640 1 --2- 172.21.15.90:0/2379871337 >> v2:172.21.15.103:3300/0 conn(0x7f286006e230 0x7f286006e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.232+0000 7f2864a00640 1 --2- 172.21.15.90:0/2379871337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2860071f10 0x7f28600cc200 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.232+0000 7f2864a00640 1 -- 172.21.15.90:0/2379871337 >> 172.21.15.90:0/2379871337 conn(0x7f28600b1460 msgr2=0x7f28600b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:06.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.232+0000 7f2864a00640 1 -- 172.21.15.90:0/2379871337 shutdown_connections 2024-04-25T01:19:06.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.232+0000 7f2864a00640 1 -- 172.21.15.90:0/2379871337 wait complete. 2024-04-25T01:19:06.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.234+0000 7f2864a00640 1 Processor -- start 2024-04-25T01:19:06.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.234+0000 7f2864a00640 1 -- start start 2024-04-25T01:19:06.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.236+0000 7f2864a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f286006db30 0x7f28600b2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:06.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.236+0000 7f2864a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f286006e230 0x7f28601f3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:06.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.236+0000 7f2864a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2860071f10 0x7f28601f3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:06.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.236+0000 7f2864a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2860159ce0 con 0x7f286006db30 2024-04-25T01:19:06.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.236+0000 7f2864a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f286015c700 con 0x7f2860071f10 2024-04-25T01:19:06.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.236+0000 7f2864a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f286006eba0 con 0x7f286006e230 2024-04-25T01:19:06.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.236+0000 7f285f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f286006db30 0x7f28600b2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:06.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.236+0000 7f285f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f286006db30 0x7f28600b2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49058/0 (socket says 172.21.15.90:49058) 2024-04-25T01:19:06.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.236+0000 7f285f400640 1 -- 172.21.15.90:0/2286964780 learned_addr learned my addr 172.21.15.90:0/2286964780 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:19:06.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.236+0000 7f285f400640 1 -- 172.21.15.90:0/2286964780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f286006e230 msgr2=0x7f28601f3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:06.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.236+0000 7f285f400640 1 --2- 172.21.15.90:0/2286964780 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f286006e230 0x7f28601f3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.236+0000 7f285f400640 1 -- 172.21.15.90:0/2286964780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2860071f10 msgr2=0x7f28601f3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:06.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.236+0000 7f285f400640 1 --2- 172.21.15.90:0/2286964780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2860071f10 0x7f28601f3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.236+0000 7f285f400640 1 -- 172.21.15.90:0/2286964780 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28600b3240 con 0x7f286006db30 2024-04-25T01:19:06.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.237+0000 7f285f400640 1 --2- 172.21.15.90:0/2286964780 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f286006db30 0x7f28600b2860 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f2850000c00 tx=0x7f28500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:06.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.237+0000 7f2857400640 1 -- 172.21.15.90:0/2286964780 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2850037050 con 0x7f286006db30 2024-04-25T01:19:06.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.237+0000 7f2857400640 1 -- 172.21.15.90:0/2286964780 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2850038040 con 0x7f286006db30 2024-04-25T01:19:06.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.237+0000 7f2857400640 1 -- 172.21.15.90:0/2286964780 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2850004450 con 0x7f286006db30 2024-04-25T01:19:06.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.237+0000 7f2864a00640 1 -- 172.21.15.90:0/2286964780 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28600b3520 con 0x7f286006db30 2024-04-25T01:19:06.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.237+0000 7f2864a00640 1 -- 172.21.15.90:0/2286964780 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28600b3ae0 con 0x7f286006db30 2024-04-25T01:19:06.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.238+0000 7f2864a00640 1 -- 172.21.15.90:0/2286964780 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f286004f500 con 0x7f286006db30 2024-04-25T01:19:06.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.239+0000 7f2857400640 1 -- 172.21.15.90:0/2286964780 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2850030980 con 0x7f286006db30 2024-04-25T01:19:06.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.240+0000 7f2857400640 1 --2- 172.21.15.90:0/2286964780 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f283c06be30 0x7f283c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:06.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.240+0000 7f2857400640 1 -- 172.21.15.90:0/2286964780 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5421+0+0 (secure 0 0 0) 0x7f28500ab990 con 0x7f286006db30 2024-04-25T01:19:06.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.240+0000 7f2857e00640 1 --2- 172.21.15.90:0/2286964780 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f283c06be30 0x7f283c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:06.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.241+0000 7f2857400640 1 --2- 172.21.15.90:0/2286964780 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x7f283c072370 0x7f283c0747b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:06.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.241+0000 7f2857400640 1 -- 172.21.15.90:0/2286964780 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f285000af30 con 0x7f283c072370 2024-04-25T01:19:06.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.241+0000 7f2857400640 1 -- 172.21.15.90:0/2286964780 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f28500b2050 con 0x7f286006db30 2024-04-25T01:19:06.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.241+0000 7f2857e00640 1 --2- 172.21.15.90:0/2286964780 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f283c06be30 0x7f283c06e2f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2848007fd0 tx=0x7f2848007920 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:06.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.241+0000 7f285fe00640 1 --2- 172.21.15.90:0/2286964780 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x7f283c072370 0x7f283c0747b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:06.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.242+0000 7f285fe00640 1 --2- 172.21.15.90:0/2286964780 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x7f283c072370 0x7f283c0747b0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:06.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.244+0000 7f2857400640 1 -- 172.21.15.90:0/2286964780 <== osd.7 v2:172.21.15.103:6800/1145679154 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f285000af30 con 0x7f283c072370 2024-04-25T01:19:06.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.259+0000 7f2864a00640 1 -- 172.21.15.90:0/2286964780 --> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f28600724a0 con 0x7f283c072370 2024-04-25T01:19:06.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.260+0000 7f2857400640 1 -- 172.21.15.90:0/2286964780 <== osd.7 v2:172.21.15.103:6800/1145679154 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f28600724a0 con 0x7f283c072370 2024-04-25T01:19:06.243 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T01:19:06.243 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T01:19:06.243 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T01:19:06.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.260+0000 7f2864a00640 1 -- 172.21.15.90:0/2286964780 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x7f283c072370 msgr2=0x7f283c0747b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:06.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.260+0000 7f2864a00640 1 --2- 172.21.15.90:0/2286964780 >> [v2:172.21.15.103:6800/1145679154,v1:172.21.15.103:6802/1145679154] conn(0x7f283c072370 0x7f283c0747b0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.260+0000 7f2864a00640 1 -- 172.21.15.90:0/2286964780 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f283c06be30 msgr2=0x7f283c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:06.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.260+0000 7f2864a00640 1 --2- 172.21.15.90:0/2286964780 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f283c06be30 0x7f283c06e2f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2848007fd0 tx=0x7f2848007920 comp rx=0 tx=0).stop 2024-04-25T01:19:06.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.260+0000 7f2864a00640 1 -- 172.21.15.90:0/2286964780 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f286006db30 msgr2=0x7f28600b2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:06.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.260+0000 7f2864a00640 1 --2- 172.21.15.90:0/2286964780 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f286006db30 0x7f28600b2860 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f2850000c00 tx=0x7f28500046c0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.260+0000 7f285fe00640 1 -- 172.21.15.90:0/2286964780 reap_dead start 2024-04-25T01:19:06.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.261+0000 7f2864a00640 1 -- 172.21.15.90:0/2286964780 shutdown_connections 2024-04-25T01:19:06.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.261+0000 7f2864a00640 1 -- 172.21.15.90:0/2286964780 >> 172.21.15.90:0/2286964780 conn(0x7f28600b1460 msgr2=0x7f286006c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:06.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.261+0000 7f2864a00640 1 -- 172.21.15.90:0/2286964780 shutdown_connections 2024-04-25T01:19:06.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.261+0000 7f2864a00640 1 -- 172.21.15.90:0/2286964780 wait complete. 2024-04-25T01:19:06.261 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-25T01:19:06.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.471+0000 7f1e06400640 1 Processor -- start 2024-04-25T01:19:06.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.471+0000 7f1e06400640 1 -- start start 2024-04-25T01:19:06.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.473+0000 7f1e06400640 1 --2- >> v2:172.21.15.103:3300/0 conn(0x7f1e0015bdc0 0x7f1e00072690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:06.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.473+0000 7f1e06400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e0015c1a0 con 0x7f1e0015cd80 2024-04-25T01:19:06.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.473+0000 7f1e06400640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e0015d210 con 0x7f1e000aef40 2024-04-25T01:19:06.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.473+0000 7f1e06400640 1 -- --> v2:172.21.15.103:3300/0 -- mon_getmap magic: 0 -- 0x7f1e000cc5d0 con 0x7f1e0015bdc0 2024-04-25T01:19:06.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.473+0000 7f1df7e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f1e0015cd80 0x7f1e000ae830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52266/0 (socket says 172.21.15.90:52266) 2024-04-25T01:19:06.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.473+0000 7f1df7e00640 1 -- 172.21.15.90:0/3011186767 learned_addr learned my addr 172.21.15.90:0/3011186767 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:19:06.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.474+0000 7f1e05000640 1 --2- 172.21.15.90:0/3011186767 >> v2:172.21.15.103:3300/0 conn(0x7f1e0015bdc0 0x7f1e00072690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:06.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.474+0000 7f1dffe00640 1 -- 172.21.15.90:0/3011186767 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 367134798 0 0) 0x7f1e0015c1a0 con 0x7f1e0015cd80 2024-04-25T01:19:06.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.474+0000 7f1dffe00640 1 -- 172.21.15.90:0/3011186767 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1de4003520 con 0x7f1e0015cd80 2024-04-25T01:19:06.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.474+0000 7f1dffe00640 1 -- 172.21.15.90:0/3011186767 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2075566392 0 0) 0x7f1de4003520 con 0x7f1e0015cd80 2024-04-25T01:19:06.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.474+0000 7f1dffe00640 1 -- 172.21.15.90:0/3011186767 >> v2:172.21.15.103:3300/0 conn(0x7f1e0015bdc0 msgr2=0x7f1e00072690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:06.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.474+0000 7f1dffe00640 1 --2- 172.21.15.90:0/3011186767 >> v2:172.21.15.103:3300/0 conn(0x7f1e0015bdc0 0x7f1e00072690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.474+0000 7f1dffe00640 1 -- 172.21.15.90:0/3011186767 >> v1:172.21.15.103:6789/0 conn(0x7f1e000aef40 legacy=0x7f1e000cdc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:06.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.474+0000 7f1dffe00640 1 -- 172.21.15.90:0/3011186767 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e00072c70 con 0x7f1e0015cd80 2024-04-25T01:19:06.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.474+0000 7f1dffe00640 1 -- 172.21.15.90:0/3011186767 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7f1de8001670 con 0x7f1e0015cd80 2024-04-25T01:19:06.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.474+0000 7f1dffe00640 1 -- 172.21.15.90:0/3011186767 >> v1:172.21.15.90:6789/0 conn(0x7f1e0015cd80 legacy=0x7f1e000ae830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:06.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.474+0000 7f1dffe00640 1 --2- 172.21.15.90:0/3011186767 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1de40039b0 0x7f1de4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:06.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.475+0000 7f1dffe00640 1 --2- 172.21.15.90:0/3011186767 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1de4004800 0x7f1de4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:06.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.475+0000 7f1dffe00640 1 --2- 172.21.15.90:0/3011186767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1de40051c0 0x7f1de400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:06.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.475+0000 7f1dffe00640 1 -- 172.21.15.90:0/3011186767 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1e000cc5d0 con 0x7f1de40051c0 2024-04-25T01:19:06.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.475+0000 7f1dffe00640 1 -- 172.21.15.90:0/3011186767 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1de400c160 con 0x7f1de4004800 2024-04-25T01:19:06.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.475+0000 7f1dffe00640 1 -- 172.21.15.90:0/3011186767 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1de400c360 con 0x7f1de40039b0 2024-04-25T01:19:06.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.475+0000 7f1e05a00640 1 --2- 172.21.15.90:0/3011186767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1de40051c0 0x7f1de400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:06.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.475+0000 7f1df7e00640 1 --2- 172.21.15.90:0/3011186767 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1de4004800 0x7f1de4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:06.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.475+0000 7f1e05000640 1 --2- 172.21.15.90:0/3011186767 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1de40039b0 0x7f1de4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:06.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.475+0000 7f1e05a00640 1 -- 172.21.15.90:0/3011186767 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1de40039b0 msgr2=0x7f1de4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:06.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.475+0000 7f1e05a00640 1 --2- 172.21.15.90:0/3011186767 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1de40039b0 0x7f1de4003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.475+0000 7f1e05a00640 1 -- 172.21.15.90:0/3011186767 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1de4004800 msgr2=0x7f1de4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:06.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.475+0000 7f1e05a00640 1 --2- 172.21.15.90:0/3011186767 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1de4004800 0x7f1de4004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.475+0000 7f1e05a00640 1 -- 172.21.15.90:0/3011186767 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1de4003520 con 0x7f1de40051c0 2024-04-25T01:19:06.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.476+0000 7f1e05a00640 1 -- 172.21.15.90:0/3011186767 reap_dead start 2024-04-25T01:19:06.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.476+0000 7f1e05a00640 1 --2- 172.21.15.90:0/3011186767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1de40051c0 0x7f1de400b9c0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f1df8001e50 tx=0x7f1de4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=605ea05c476b19b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:06.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.477+0000 7f1dffe00640 1 -- 172.21.15.90:0/3011186767 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df8001a70 con 0x7f1de40051c0 2024-04-25T01:19:06.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.477+0000 7f1dffe00640 1 -- 172.21.15.90:0/3011186767 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1df8003970 con 0x7f1de40051c0 2024-04-25T01:19:06.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.477+0000 7f1dffe00640 1 -- 172.21.15.90:0/3011186767 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df8003d60 con 0x7f1de40051c0 2024-04-25T01:19:06.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.477+0000 7f1e06400640 1 -- 172.21.15.90:0/3011186767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1de40051c0 msgr2=0x7f1de400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:06.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.477+0000 7f1e06400640 1 --2- 172.21.15.90:0/3011186767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1de40051c0 0x7f1de400b9c0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f1df8001e50 tx=0x7f1de4005830 comp rx=0 tx=0).stop 2024-04-25T01:19:06.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.477+0000 7f1e06400640 1 -- 172.21.15.90:0/3011186767 shutdown_connections 2024-04-25T01:19:06.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.477+0000 7f1e06400640 1 --2- 172.21.15.90:0/3011186767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1de40051c0 0x7f1de400b9c0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.477+0000 7f1e06400640 1 -- 172.21.15.90:0/3011186767 >> 172.21.15.90:0/3011186767 conn(0x7f1e000b1560 msgr2=0x7f1e000b1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:06.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.477+0000 7f1e06400640 1 -- 172.21.15.90:0/3011186767 shutdown_connections 2024-04-25T01:19:06.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.477+0000 7f1e06400640 1 -- 172.21.15.90:0/3011186767 wait complete. 2024-04-25T01:19:06.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.479+0000 7f1e06400640 1 Processor -- start 2024-04-25T01:19:06.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.479+0000 7f1e06400640 1 -- start start 2024-04-25T01:19:06.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.480+0000 7f1e06400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1de40051c0 0x7f1e00146a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:06.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.480+0000 7f1e06400640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e00146f40 0x7f1e001e7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:06.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.480+0000 7f1e06400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e001e7eb0 0x7f1e001495a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:06.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.480+0000 7f1e06400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1e00070de0 con 0x7f1de40051c0 2024-04-25T01:19:06.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.480+0000 7f1e06400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1e000cf6b0 con 0x7f1e001e7eb0 2024-04-25T01:19:06.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.480+0000 7f1e06400640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f1e000710c0 con 0x7f1e00146f40 2024-04-25T01:19:06.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.481+0000 7f1e05000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1de40051c0 0x7f1e00146a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:06.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.481+0000 7f1e05000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1de40051c0 0x7f1e00146a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49088/0 (socket says 172.21.15.90:49088) 2024-04-25T01:19:06.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.481+0000 7f1e05000640 1 -- 172.21.15.90:0/2006556365 learned_addr learned my addr 172.21.15.90:0/2006556365 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:19:06.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.481+0000 7f1e05a00640 1 --2- 172.21.15.90:0/2006556365 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e001e7eb0 0x7f1e001495a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:06.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.481+0000 7f1df7e00640 1 --2- 172.21.15.90:0/2006556365 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e00146f40 0x7f1e001e7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:06.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.481+0000 7f1e05000640 1 -- 172.21.15.90:0/2006556365 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e00146f40 msgr2=0x7f1e001e7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:06.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.481+0000 7f1e05000640 1 --2- 172.21.15.90:0/2006556365 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e00146f40 0x7f1e001e7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.481+0000 7f1e05000640 1 -- 172.21.15.90:0/2006556365 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e001e7eb0 msgr2=0x7f1e001495a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:06.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.481+0000 7f1e05000640 1 --2- 172.21.15.90:0/2006556365 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e001e7eb0 0x7f1e001495a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.481+0000 7f1e05000640 1 -- 172.21.15.90:0/2006556365 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e000cc3c0 con 0x7f1de40051c0 2024-04-25T01:19:06.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.481+0000 7f1df7e00640 1 --2- 172.21.15.90:0/2006556365 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e00146f40 0x7f1e001e7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:19:06.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.481+0000 7f1e05000640 1 --2- 172.21.15.90:0/2006556365 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1de40051c0 0x7f1e00146a00 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f1df0010860 tx=0x7f1df0010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:06.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.482+0000 7f1dfd600640 1 -- 172.21.15.90:0/2006556365 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df000f040 con 0x7f1de40051c0 2024-04-25T01:19:06.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.482+0000 7f1dfd600640 1 -- 172.21.15.90:0/2006556365 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1df000eb60 con 0x7f1de40051c0 2024-04-25T01:19:06.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.482+0000 7f1e06400640 1 -- 172.21.15.90:0/2006556365 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e0014a310 con 0x7f1de40051c0 2024-04-25T01:19:06.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.482+0000 7f1dfd600640 1 -- 172.21.15.90:0/2006556365 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df001c400 con 0x7f1de40051c0 2024-04-25T01:19:06.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.482+0000 7f1e06400640 1 -- 172.21.15.90:0/2006556365 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e00147710 con 0x7f1de40051c0 2024-04-25T01:19:06.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.483+0000 7f1e06400640 1 -- 172.21.15.90:0/2006556365 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e0004f500 con 0x7f1de40051c0 2024-04-25T01:19:06.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.484+0000 7f1dfd600640 1 -- 172.21.15.90:0/2006556365 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1df001c5a0 con 0x7f1de40051c0 2024-04-25T01:19:06.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.485+0000 7f1dfd600640 1 --2- 172.21.15.90:0/2006556365 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1ddc06be30 0x7f1ddc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:06.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.485+0000 7f1dfd600640 1 -- 172.21.15.90:0/2006556365 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5421+0+0 (secure 0 0 0) 0x7f1df008b440 con 0x7f1de40051c0 2024-04-25T01:19:06.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.486+0000 7f1df7e00640 1 --2- 172.21.15.90:0/2006556365 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1ddc06be30 0x7f1ddc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:06.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.486+0000 7f1df7e00640 1 --2- 172.21.15.90:0/2006556365 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1ddc06be30 0x7f1ddc06e2f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1e0015a4e0 tx=0x7f1de8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:06.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.490+0000 7f1dfd600640 1 -- 172.21.15.90:0/2006556365 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1df0018030 con 0x7f1de40051c0 2024-04-25T01:19:06.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.688+0000 7f1e06400640 1 -- 172.21.15.90:0/2006556365 --> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f1e000b2050 con 0x7f1ddc06be30 2024-04-25T01:19:06.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.689+0000 7f1dfd600640 1 -- 172.21.15.90:0/2006556365 <== mgr.4104 v2:172.21.15.90:6832/2198086733 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f1e000b2050 con 0x7f1ddc06be30 2024-04-25T01:19:06.672 INFO:teuthology.orchestra.run.smithi090.stderr:instructed osd(s) 7 to deep-scrub 2024-04-25T01:19:06.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.692+0000 7f1e06400640 1 -- 172.21.15.90:0/2006556365 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1ddc06be30 msgr2=0x7f1ddc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:06.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.692+0000 7f1e06400640 1 --2- 172.21.15.90:0/2006556365 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1ddc06be30 0x7f1ddc06e2f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1e0015a4e0 tx=0x7f1de8009040 comp rx=0 tx=0).stop 2024-04-25T01:19:06.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.692+0000 7f1e06400640 1 -- 172.21.15.90:0/2006556365 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1de40051c0 msgr2=0x7f1e00146a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:06.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.692+0000 7f1e06400640 1 --2- 172.21.15.90:0/2006556365 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1de40051c0 0x7f1e00146a00 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f1df0010860 tx=0x7f1df0010d20 comp rx=0 tx=0).stop 2024-04-25T01:19:06.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.692+0000 7f1e06400640 1 -- 172.21.15.90:0/2006556365 shutdown_connections 2024-04-25T01:19:06.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.692+0000 7f1e06400640 1 --2- 172.21.15.90:0/2006556365 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f1ddc06be30 0x7f1ddc06e2f0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.692+0000 7f1e06400640 1 --2- 172.21.15.90:0/2006556365 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e001e7eb0 0x7f1e001495a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.692+0000 7f1e06400640 1 --2- 172.21.15.90:0/2006556365 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f1e00146f40 0x7f1e001e7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.692+0000 7f1e06400640 1 --2- 172.21.15.90:0/2006556365 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1de40051c0 0x7f1e00146a00 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.692+0000 7f1e06400640 1 -- 172.21.15.90:0/2006556365 >> 172.21.15.90:0/2006556365 conn(0x7f1e000b1560 msgr2=0x7f1e0015d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:06.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.692+0000 7f1e06400640 1 -- 172.21.15.90:0/2006556365 shutdown_connections 2024-04-25T01:19:06.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.692+0000 7f1e06400640 1 -- 172.21.15.90:0/2006556365 wait complete. 2024-04-25T01:19:06.696 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:19:06.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.911+0000 7f0c97200640 1 Processor -- start 2024-04-25T01:19:06.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.911+0000 7f0c97200640 1 -- start start 2024-04-25T01:19:06.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.913+0000 7f0c97200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0c981561c0 0x7f0c980cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:06.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.913+0000 7f0c97200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c980709d0 0x7f0c9815cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:06.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.913+0000 7f0c97200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c98070f90 con 0x7f0c980cfb30 2024-04-25T01:19:06.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.913+0000 7f0c97200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f0c980b4310 con 0x7f0c981561c0 2024-04-25T01:19:06.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.913+0000 7f0c97200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0c981565a0 con 0x7f0c980709d0 2024-04-25T01:19:06.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.913+0000 7f0c95e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0c981561c0 0x7f0c980cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:06.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.913+0000 7f0c95e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0c981561c0 0x7f0c980cdbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49094/0 (socket says 172.21.15.90:49094) 2024-04-25T01:19:06.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.913+0000 7f0c95e00640 1 -- 172.21.15.90:0/4055674243 learned_addr learned my addr 172.21.15.90:0/4055674243 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:19:06.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.913+0000 7f0c95e00640 1 -- 172.21.15.90:0/4055674243 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c980709d0 msgr2=0x7f0c9815cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:06.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.913+0000 7f0c95e00640 1 --2- 172.21.15.90:0/4055674243 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c980709d0 0x7f0c9815cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.913+0000 7f0c95e00640 1 -- 172.21.15.90:0/4055674243 >> v1:172.21.15.103:6789/0 conn(0x7f0c980cfb30 legacy=0x7f0c980b2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:06.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.913+0000 7f0c95e00640 1 -- 172.21.15.90:0/4055674243 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c98071270 con 0x7f0c981561c0 2024-04-25T01:19:06.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.914+0000 7f0c95e00640 1 --2- 172.21.15.90:0/4055674243 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c981561c0 0x7f0c980cdbf0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f0c88009870 tx=0x7f0c8802a0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34d50618c7b347b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:06.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.915+0000 7f0c94a00640 1 -- 172.21.15.90:0/4055674243 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c88037070 con 0x7f0c981561c0 2024-04-25T01:19:06.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.915+0000 7f0c94a00640 1 -- 172.21.15.90:0/4055674243 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c8802adf0 con 0x7f0c981561c0 2024-04-25T01:19:06.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.915+0000 7f0c94a00640 1 -- 172.21.15.90:0/4055674243 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c88032660 con 0x7f0c981561c0 2024-04-25T01:19:06.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.915+0000 7f0c97200640 1 -- 172.21.15.90:0/4055674243 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c981561c0 msgr2=0x7f0c980cdbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:06.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.915+0000 7f0c97200640 1 --2- 172.21.15.90:0/4055674243 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c981561c0 0x7f0c980cdbf0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f0c88009870 tx=0x7f0c8802a0e0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.916+0000 7f0c97200640 1 -- 172.21.15.90:0/4055674243 shutdown_connections 2024-04-25T01:19:06.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.916+0000 7f0c97200640 1 --2- 172.21.15.90:0/4055674243 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c980709d0 0x7f0c9815cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.916+0000 7f0c97200640 1 --2- 172.21.15.90:0/4055674243 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c981561c0 0x7f0c980cdbf0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.916+0000 7f0c97200640 1 -- 172.21.15.90:0/4055674243 >> 172.21.15.90:0/4055674243 conn(0x7f0c9806b320 msgr2=0x7f0c9806b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:06.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.916+0000 7f0c97200640 1 -- 172.21.15.90:0/4055674243 shutdown_connections 2024-04-25T01:19:06.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.916+0000 7f0c97200640 1 -- 172.21.15.90:0/4055674243 wait complete. 2024-04-25T01:19:06.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.917+0000 7f0c97200640 1 Processor -- start 2024-04-25T01:19:06.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.918+0000 7f0c97200640 1 -- start start 2024-04-25T01:19:06.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.918+0000 7f0c97200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c980709d0 0x7f0c98150b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:06.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.919+0000 7f0c97200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c980cfb30 0x7f0c981f77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:06.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.919+0000 7f0c97200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c981561c0 0x7f0c9813ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:06.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.919+0000 7f0c97200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0c980cffa0 con 0x7f0c981561c0 2024-04-25T01:19:06.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.919+0000 7f0c97200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0c98071830 con 0x7f0c980cfb30 2024-04-25T01:19:06.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.919+0000 7f0c97200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f0c980d0120 con 0x7f0c980709d0 2024-04-25T01:19:06.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.919+0000 7f0c96800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c981561c0 0x7f0c9813ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:06.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.919+0000 7f0c96800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c981561c0 0x7f0c9813ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49104/0 (socket says 172.21.15.90:49104) 2024-04-25T01:19:06.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.919+0000 7f0c96800640 1 -- 172.21.15.90:0/2200446578 learned_addr learned my addr 172.21.15.90:0/2200446578 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:19:06.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.919+0000 7f0c95e00640 1 --2- 172.21.15.90:0/2200446578 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c980709d0 0x7f0c98150b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:06.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.919+0000 7f0c96800640 1 -- 172.21.15.90:0/2200446578 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c980709d0 msgr2=0x7f0c98150b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:06.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.919+0000 7f0c96800640 1 --2- 172.21.15.90:0/2200446578 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c980709d0 0x7f0c98150b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.919+0000 7f0c96800640 1 -- 172.21.15.90:0/2200446578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c980cfb30 msgr2=0x7f0c981f77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:06.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.919+0000 7f0c96800640 1 --2- 172.21.15.90:0/2200446578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c980cfb30 0x7f0c981f77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:06.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.919+0000 7f0c96800640 1 -- 172.21.15.90:0/2200446578 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c9814fd60 con 0x7f0c981561c0 2024-04-25T01:19:06.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.920+0000 7f0c96800640 1 --2- 172.21.15.90:0/2200446578 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c981561c0 0x7f0c9813ea20 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f0c9000c780 tx=0x7f0c9000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:06.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.920+0000 7f0c8e000640 1 -- 172.21.15.90:0/2200446578 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c90017070 con 0x7f0c981561c0 2024-04-25T01:19:06.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.920+0000 7f0c8e000640 1 -- 172.21.15.90:0/2200446578 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c90003da0 con 0x7f0c981561c0 2024-04-25T01:19:06.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.920+0000 7f0c8e000640 1 -- 172.21.15.90:0/2200446578 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c900052b0 con 0x7f0c981561c0 2024-04-25T01:19:06.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.920+0000 7f0c97200640 1 -- 172.21.15.90:0/2200446578 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c98150040 con 0x7f0c981561c0 2024-04-25T01:19:06.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.920+0000 7f0c97200640 1 -- 172.21.15.90:0/2200446578 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c9814fa30 con 0x7f0c981561c0 2024-04-25T01:19:06.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.921+0000 7f0c97200640 1 -- 172.21.15.90:0/2200446578 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c9804f500 con 0x7f0c981561c0 2024-04-25T01:19:06.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.922+0000 7f0c8e000640 1 -- 172.21.15.90:0/2200446578 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0c90005450 con 0x7f0c981561c0 2024-04-25T01:19:06.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.923+0000 7f0c8e000640 1 --2- 172.21.15.90:0/2200446578 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0c7406be30 0x7f0c7406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:06.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.923+0000 7f0c8e000640 1 -- 172.21.15.90:0/2200446578 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5421+0+0 (secure 0 0 0) 0x7f0c9008a240 con 0x7f0c981561c0 2024-04-25T01:19:06.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.923+0000 7f0c95e00640 1 --2- 172.21.15.90:0/2200446578 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0c7406be30 0x7f0c7406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:06.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.924+0000 7f0c95e00640 1 --2- 172.21.15.90:0/2200446578 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0c7406be30 0x7f0c7406e2f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0c88034000 tx=0x7f0c8802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:06.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:06.927+0000 7f0c8e000640 1 -- 172.21.15.90:0/2200446578 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0c90058d00 con 0x7f0c981561c0 2024-04-25T01:19:07.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.120+0000 7f0c97200640 1 -- 172.21.15.90:0/2200446578 --> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0c98068620 con 0x7f0c7406be30 2024-04-25T01:19:07.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.122+0000 7f0c8e000640 1 -- 172.21.15.90:0/2200446578 <== mgr.4104 v2:172.21.15.90:6832/2198086733 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13383 (secure 0 0 0) 0x7f0c98068620 con 0x7f0c7406be30 2024-04-25T01:19:07.105 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:19:07.106 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-25T01:19:07.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.124+0000 7f0c97200640 1 -- 172.21.15.90:0/2200446578 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0c7406be30 msgr2=0x7f0c7406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:07.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.124+0000 7f0c97200640 1 --2- 172.21.15.90:0/2200446578 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0c7406be30 0x7f0c7406e2f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0c88034000 tx=0x7f0c8802e000 comp rx=0 tx=0).stop 2024-04-25T01:19:07.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.124+0000 7f0c97200640 1 -- 172.21.15.90:0/2200446578 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c981561c0 msgr2=0x7f0c9813ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:07.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.124+0000 7f0c97200640 1 --2- 172.21.15.90:0/2200446578 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c981561c0 0x7f0c9813ea20 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f0c9000c780 tx=0x7f0c9000cc40 comp rx=0 tx=0).stop 2024-04-25T01:19:07.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.124+0000 7f0c97200640 1 -- 172.21.15.90:0/2200446578 shutdown_connections 2024-04-25T01:19:07.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.124+0000 7f0c97200640 1 --2- 172.21.15.90:0/2200446578 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f0c7406be30 0x7f0c7406e2f0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:07.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.124+0000 7f0c97200640 1 --2- 172.21.15.90:0/2200446578 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c981561c0 0x7f0c9813ea20 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:07.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.124+0000 7f0c97200640 1 --2- 172.21.15.90:0/2200446578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c980cfb30 0x7f0c981f77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:07.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.124+0000 7f0c97200640 1 --2- 172.21.15.90:0/2200446578 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f0c980709d0 0x7f0c98150b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:07.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.125+0000 7f0c97200640 1 -- 172.21.15.90:0/2200446578 >> 172.21.15.90:0/2200446578 conn(0x7f0c9806b320 msgr2=0x7f0c980ce570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:07.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.125+0000 7f0c97200640 1 -- 172.21.15.90:0/2200446578 shutdown_connections 2024-04-25T01:19:07.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.125+0000 7f0c97200640 1 -- 172.21.15.90:0/2200446578 wait complete. 2024-04-25T01:19:07.128 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2024-04-25T01:19:05.876186+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":230676,"kb_used_data":6184,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754744044,"statfs":{"total":773094113280,"available":772857901056,"internally_reserved":0,"allocated":6332416,"data_stored":3505728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":229756465},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.800989"},"pg_stats":[{"pgid":"1.0","version":"54'119","reported_seq":240,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T01:19:02.781010+0000","last_change":"2024-04-25T01:19:02.781044+0000","last_active":"2024-04-25T01:19:02.781010+0000","last_peered":"2024-04-25T01:19:02.781010+0000","last_clean":"2024-04-25T01:19:02.781010+0000","last_became_active":"2024-04-25T01:16:17.465687+0000","last_became_peered":"2024-04-25T01:16:17.465687+0000","last_unstale":"2024-04-25T01:19:02.781010+0000","last_undegraded":"2024-04-25T01:19:02.781010+0000","last_fullsized":"2024-04-25T01:19:02.781010+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'119","last_scrub_stamp":"2024-04-25T01:19:02.780923+0000","last_deep_scrub":"54'119","last_deep_scrub_stamp":"2024-04-25T01:19:02.780923+0000","last_clean_scrub_stamp":"2024-04-25T01:19:02.780923+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:11.934429+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542181,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27656,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":643072,"data_stored":290624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542180,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27656,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":643072,"data_stored":290624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542181,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32384,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339456,"statfs":{"total":96636764160,"available":96603602944,"internally_reserved":0,"allocated":1236992,"data_stored":880992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542180,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27652,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344188,"statfs":{"total":96636764160,"available":96608448512,"internally_reserved":0,"allocated":643072,"data_stored":290624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542180,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27648,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":643072,"data_stored":290624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542182,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27648,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":643072,"data_stored":290624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542180,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32384,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339456,"statfs":{"total":96636764160,"available":96603602944,"internally_reserved":0,"allocated":1236992,"data_stored":880992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542181,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27648,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":643072,"data_stored":290624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:19:07.131 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-25T01:19:07.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.342+0000 7f7ca0a00640 1 Processor -- start 2024-04-25T01:19:07.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.342+0000 7f7ca0a00640 1 -- start start 2024-04-25T01:19:07.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.344+0000 7f7ca0a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c9c15c840 0x7f7c9c0b4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:07.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.344+0000 7f7ca0a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7c9c070e60 0x7f7c9c0712f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:07.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.344+0000 7f7ca0a00640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c9c06e3f0 con 0x7f7c9c1627f0 2024-04-25T01:19:07.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.344+0000 7f7ca0a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f7c9c166010 con 0x7f7c9c070e60 2024-04-25T01:19:07.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.344+0000 7f7ca0a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7c9c15cc20 con 0x7f7c9c15c840 2024-04-25T01:19:07.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.345+0000 7f7c9b200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7c9c070e60 0x7f7c9c0712f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:07.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.345+0000 7f7c9b200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7c9c070e60 0x7f7c9c0712f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49120/0 (socket says 172.21.15.90:49120) 2024-04-25T01:19:07.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.345+0000 7f7c9b200640 1 -- 172.21.15.90:0/1989426698 learned_addr learned my addr 172.21.15.90:0/1989426698 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:19:07.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.345+0000 7f7c9a800640 1 --2- 172.21.15.90:0/1989426698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c9c15c840 0x7f7c9c0b4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:07.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.345+0000 7f7c9b200640 1 -- 172.21.15.90:0/1989426698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c9c15c840 msgr2=0x7f7c9c0b4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:07.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.345+0000 7f7c9b200640 1 --2- 172.21.15.90:0/1989426698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c9c15c840 0x7f7c9c0b4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:07.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.345+0000 7f7c9b200640 1 -- 172.21.15.90:0/1989426698 >> v1:172.21.15.103:6789/0 conn(0x7f7c9c1627f0 legacy=0x7f7c9c1664a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:07.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.345+0000 7f7c9b200640 1 -- 172.21.15.90:0/1989426698 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c9c162f60 con 0x7f7c9c070e60 2024-04-25T01:19:07.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.346+0000 7f7c9b200640 1 --2- 172.21.15.90:0/1989426698 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c9c070e60 0x7f7c9c0712f0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f7c94009870 tx=0x7f7c9402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a675a8c85377571 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:07.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.346+0000 7f7c99e00640 1 -- 172.21.15.90:0/1989426698 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c94037070 con 0x7f7c9c070e60 2024-04-25T01:19:07.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.346+0000 7f7c99e00640 1 -- 172.21.15.90:0/1989426698 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c94032440 con 0x7f7c9c070e60 2024-04-25T01:19:07.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.347+0000 7f7c99e00640 1 -- 172.21.15.90:0/1989426698 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c94032810 con 0x7f7c9c070e60 2024-04-25T01:19:07.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.347+0000 7f7ca0a00640 1 -- 172.21.15.90:0/1989426698 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c9c070e60 msgr2=0x7f7c9c0712f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:07.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.347+0000 7f7ca0a00640 1 --2- 172.21.15.90:0/1989426698 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c9c070e60 0x7f7c9c0712f0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f7c94009870 tx=0x7f7c9402a210 comp rx=0 tx=0).stop 2024-04-25T01:19:07.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.347+0000 7f7ca0a00640 1 -- 172.21.15.90:0/1989426698 shutdown_connections 2024-04-25T01:19:07.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.347+0000 7f7ca0a00640 1 --2- 172.21.15.90:0/1989426698 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c9c070e60 0x7f7c9c0712f0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:07.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.347+0000 7f7ca0a00640 1 --2- 172.21.15.90:0/1989426698 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c9c15c840 0x7f7c9c0b4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:07.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.347+0000 7f7ca0a00640 1 -- 172.21.15.90:0/1989426698 >> 172.21.15.90:0/1989426698 conn(0x7f7c9c06b560 msgr2=0x7f7c9c06b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:07.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.347+0000 7f7ca0a00640 1 -- 172.21.15.90:0/1989426698 shutdown_connections 2024-04-25T01:19:07.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.347+0000 7f7ca0a00640 1 -- 172.21.15.90:0/1989426698 wait complete. 2024-04-25T01:19:07.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.349+0000 7f7ca0a00640 1 Processor -- start 2024-04-25T01:19:07.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.349+0000 7f7ca0a00640 1 -- start start 2024-04-25T01:19:07.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.350+0000 7f7ca0a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c9c070e60 0x7f7c9c15f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:07.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.350+0000 7f7ca0a00640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c9c15c840 0x7f7c9c161c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:07.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.350+0000 7f7ca0a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c9c1627f0 0x7f7c9c147910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:07.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.350+0000 7f7ca0a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7c9c162c70 con 0x7f7c9c070e60 2024-04-25T01:19:07.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.350+0000 7f7ca0a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7c9c163520 con 0x7f7c9c1627f0 2024-04-25T01:19:07.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.350+0000 7f7ca0a00640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7f7c9c166bb0 con 0x7f7c9c15c840 2024-04-25T01:19:07.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.350+0000 7f7c9a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c9c070e60 0x7f7c9c15f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:07.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.351+0000 7f7c9a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c9c070e60 0x7f7c9c15f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49122/0 (socket says 172.21.15.90:49122) 2024-04-25T01:19:07.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.351+0000 7f7c9a800640 1 -- 172.21.15.90:0/1223287968 learned_addr learned my addr 172.21.15.90:0/1223287968 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:19:07.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.351+0000 7f7c9b200640 1 --2- 172.21.15.90:0/1223287968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c9c1627f0 0x7f7c9c147910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:07.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.351+0000 7f7c91e00640 1 --2- 172.21.15.90:0/1223287968 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c9c15c840 0x7f7c9c161c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:07.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.351+0000 7f7c9a800640 1 -- 172.21.15.90:0/1223287968 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c9c15c840 msgr2=0x7f7c9c161c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:07.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.351+0000 7f7c9a800640 1 --2- 172.21.15.90:0/1223287968 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c9c15c840 0x7f7c9c161c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:07.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.351+0000 7f7c9a800640 1 -- 172.21.15.90:0/1223287968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c9c1627f0 msgr2=0x7f7c9c147910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:07.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.351+0000 7f7c9a800640 1 --2- 172.21.15.90:0/1223287968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c9c1627f0 0x7f7c9c147910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:07.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.351+0000 7f7c9a800640 1 -- 172.21.15.90:0/1223287968 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c9c160160 con 0x7f7c9c070e60 2024-04-25T01:19:07.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.351+0000 7f7c9b200640 1 --2- 172.21.15.90:0/1223287968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c9c1627f0 0x7f7c9c147910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:19:07.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.351+0000 7f7c91e00640 1 --2- 172.21.15.90:0/1223287968 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c9c15c840 0x7f7c9c161c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:19:07.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.351+0000 7f7c9a800640 1 --2- 172.21.15.90:0/1223287968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c9c070e60 0x7f7c9c15f780 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c00cd00 tx=0x7f7c8c00a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:07.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.351+0000 7f7c93400640 1 -- 172.21.15.90:0/1223287968 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c003990 con 0x7f7c9c070e60 2024-04-25T01:19:07.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.352+0000 7f7c93400640 1 -- 172.21.15.90:0/1223287968 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c8c003b30 con 0x7f7c9c070e60 2024-04-25T01:19:07.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.352+0000 7f7c93400640 1 -- 172.21.15.90:0/1223287968 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c004020 con 0x7f7c9c070e60 2024-04-25T01:19:07.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.352+0000 7f7ca0a00640 1 -- 172.21.15.90:0/1223287968 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c9c15fe80 con 0x7f7c9c070e60 2024-04-25T01:19:07.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.352+0000 7f7ca0a00640 1 -- 172.21.15.90:0/1223287968 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c9c158870 con 0x7f7c9c070e60 2024-04-25T01:19:07.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.353+0000 7f7ca0a00640 1 -- 172.21.15.90:0/1223287968 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c9c04f500 con 0x7f7c9c070e60 2024-04-25T01:19:07.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.354+0000 7f7c93400640 1 -- 172.21.15.90:0/1223287968 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7c8c0041c0 con 0x7f7c9c070e60 2024-04-25T01:19:07.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.355+0000 7f7c93400640 1 --2- 172.21.15.90:0/1223287968 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7c7806be30 0x7f7c7806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:07.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.355+0000 7f7c93400640 1 -- 172.21.15.90:0/1223287968 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5421+0+0 (secure 0 0 0) 0x7f7c8c08b230 con 0x7f7c9c070e60 2024-04-25T01:19:07.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.355+0000 7f7c91e00640 1 --2- 172.21.15.90:0/1223287968 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7c7806be30 0x7f7c7806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:07.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.356+0000 7f7c91e00640 1 --2- 172.21.15.90:0/1223287968 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7c7806be30 0x7f7c7806e2f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c157470 tx=0x7f7c84008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:07.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.359+0000 7f7c93400640 1 -- 172.21.15.90:0/1223287968 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7c8c014030 con 0x7f7c9c070e60 2024-04-25T01:19:07.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.558+0000 7f7ca0a00640 1 -- 172.21.15.90:0/1223287968 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f7c9c06e6f0 con 0x7f7c9c070e60 2024-04-25T01:19:07.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.565+0000 7f7c93400640 1 -- 172.21.15.90:0/1223287968 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v14)=0 v14) ==== 123+0+0 (secure 0 0 0) 0x7f7c8c059cf0 con 0x7f7c9c070e60 2024-04-25T01:19:07.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.567+0000 7f7c93400640 1 -- 172.21.15.90:0/1223287968 <== mon.0 v2:172.21.15.90:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f7c8c0595f0 con 0x7f7c9c070e60 2024-04-25T01:19:07.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.568+0000 7f7ca0a00640 1 -- 172.21.15.90:0/1223287968 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7c7806be30 msgr2=0x7f7c7806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:07.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.568+0000 7f7ca0a00640 1 --2- 172.21.15.90:0/1223287968 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7c7806be30 0x7f7c7806e2f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c157470 tx=0x7f7c84008040 comp rx=0 tx=0).stop 2024-04-25T01:19:07.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.568+0000 7f7ca0a00640 1 -- 172.21.15.90:0/1223287968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c9c070e60 msgr2=0x7f7c9c15f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:07.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.568+0000 7f7ca0a00640 1 --2- 172.21.15.90:0/1223287968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c9c070e60 0x7f7c9c15f780 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c00cd00 tx=0x7f7c8c00a5c0 comp rx=0 tx=0).stop 2024-04-25T01:19:07.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.569+0000 7f7ca0a00640 1 -- 172.21.15.90:0/1223287968 shutdown_connections 2024-04-25T01:19:07.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.569+0000 7f7ca0a00640 1 --2- 172.21.15.90:0/1223287968 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7f7c7806be30 0x7f7c7806e2f0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:07.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.569+0000 7f7ca0a00640 1 --2- 172.21.15.90:0/1223287968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c9c1627f0 0x7f7c9c147910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:07.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.569+0000 7f7ca0a00640 1 --2- 172.21.15.90:0/1223287968 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7f7c9c15c840 0x7f7c9c161c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:07.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.569+0000 7f7ca0a00640 1 --2- 172.21.15.90:0/1223287968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c9c070e60 0x7f7c9c15f780 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:07.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.569+0000 7f7ca0a00640 1 -- 172.21.15.90:0/1223287968 >> 172.21.15.90:0/1223287968 conn(0x7f7c9c06b560 msgr2=0x7f7c9c06b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:07.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.569+0000 7f7ca0a00640 1 -- 172.21.15.90:0/1223287968 shutdown_connections 2024-04-25T01:19:07.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.569+0000 7f7ca0a00640 1 -- 172.21.15.90:0/1223287968 wait complete. 2024-04-25T01:19:07.569 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T01:19:07.569 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:19:07.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.780+0000 7fa860200640 1 Processor -- start 2024-04-25T01:19:07.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.780+0000 7fa860200640 1 -- start start 2024-04-25T01:19:07.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.781+0000 7fa860200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa858145640 0x7fa85815ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:07.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.781+0000 7fa860200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8580cdf70 con 0x7fa8580d03a0 2024-04-25T01:19:07.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.781+0000 7fa860200640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa85815d360 con 0x7fa8580b2a20 2024-04-25T01:19:07.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.781+0000 7fa860200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa858145a20 con 0x7fa858145640 2024-04-25T01:19:07.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.782+0000 7fa85e200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa8580d03a0 0x7fa8580ce450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52270/0 (socket says 172.21.15.90:52270) 2024-04-25T01:19:07.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.782+0000 7fa85e200640 1 -- 172.21.15.90:0/34585725 learned_addr learned my addr 172.21.15.90:0/34585725 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-04-25T01:19:07.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.782+0000 7fa857e00640 1 -- 172.21.15.90:0/34585725 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1863562830 0 0) 0x7fa8580cdf70 con 0x7fa8580d03a0 2024-04-25T01:19:07.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.782+0000 7fa85d800640 1 --2- 172.21.15.90:0/34585725 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa858145640 0x7fa85815ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:07.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.782+0000 7fa857e00640 1 -- 172.21.15.90:0/34585725 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa840003520 con 0x7fa8580d03a0 2024-04-25T01:19:07.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.782+0000 7fa857e00640 1 -- 172.21.15.90:0/34585725 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3855315679 0 0) 0x7fa840003520 con 0x7fa8580d03a0 2024-04-25T01:19:07.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.783+0000 7fa857e00640 1 -- 172.21.15.90:0/34585725 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa858145640 msgr2=0x7fa85815ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:07.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.783+0000 7fa857e00640 1 --2- 172.21.15.90:0/34585725 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa858145640 0x7fa85815ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:07.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.783+0000 7fa857e00640 1 -- 172.21.15.90:0/34585725 >> v1:172.21.15.103:6789/0 conn(0x7fa8580b2a20 legacy=0x7fa8580cfc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:07.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.783+0000 7fa857e00640 1 -- 172.21.15.90:0/34585725 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa85806d140 con 0x7fa8580d03a0 2024-04-25T01:19:07.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.783+0000 7fa857e00640 1 -- 172.21.15.90:0/34585725 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2875042004 0 0) 0x7fa8500019f0 con 0x7fa8580d03a0 2024-04-25T01:19:07.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.783+0000 7fa857e00640 1 -- 172.21.15.90:0/34585725 >> v1:172.21.15.90:6789/0 conn(0x7fa8580d03a0 legacy=0x7fa8580ce450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:07.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.783+0000 7fa857e00640 1 --2- 172.21.15.90:0/34585725 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa858145640 0x7fa8400039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:07.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.783+0000 7fa857e00640 1 --2- 172.21.15.90:0/34585725 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa840003ff0 0x7fa840004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:07.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.783+0000 7fa857e00640 1 --2- 172.21.15.90:0/34585725 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa840004d40 0x7fa8400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:07.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.783+0000 7fa857e00640 1 -- 172.21.15.90:0/34585725 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa858145a20 con 0x7fa840003ff0 2024-04-25T01:19:07.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.783+0000 7fa857e00640 1 -- 172.21.15.90:0/34585725 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa84000bb50 con 0x7fa840004d40 2024-04-25T01:19:07.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.783+0000 7fa857e00640 1 -- 172.21.15.90:0/34585725 --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa84000bd50 con 0x7fa858145640 2024-04-25T01:19:07.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.783+0000 7fa85ce00640 1 --2- 172.21.15.90:0/34585725 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa840003ff0 0x7fa840004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:07.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.783+0000 7fa85d800640 1 --2- 172.21.15.90:0/34585725 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa858145640 0x7fa8400039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:07.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.783+0000 7fa85e200640 1 --2- 172.21.15.90:0/34585725 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa840004d40 0x7fa8400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:07.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.783+0000 7fa85ce00640 1 -- 172.21.15.90:0/34585725 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa858145640 msgr2=0x7fa8400039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:07.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.783+0000 7fa85ce00640 1 --2- 172.21.15.90:0/34585725 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa858145640 0x7fa8400039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:07.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.783+0000 7fa85ce00640 1 -- 172.21.15.90:0/34585725 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa840004d40 msgr2=0x7fa8400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:19:07.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.783+0000 7fa85ce00640 1 --2- 172.21.15.90:0/34585725 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa840004d40 0x7fa8400051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:07.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.783+0000 7fa85ce00640 1 -- 172.21.15.90:0/34585725 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa840003520 con 0x7fa840003ff0 2024-04-25T01:19:07.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.784+0000 7fa85ce00640 1 --2- 172.21.15.90:0/34585725 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa840003ff0 0x7fa840004800 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fa844003d80 tx=0x7fa840005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a70c1db37cb628ed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:07.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.785+0000 7fa857e00640 1 -- 172.21.15.90:0/34585725 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa844037070 con 0x7fa840003ff0 2024-04-25T01:19:07.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.785+0000 7fa857e00640 1 -- 172.21.15.90:0/34585725 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fa84402a9d0 con 0x7fa840003ff0 2024-04-25T01:19:07.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.785+0000 7fa857e00640 1 -- 172.21.15.90:0/34585725 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa844003980 con 0x7fa840003ff0 2024-04-25T01:19:07.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.785+0000 7fa860200640 1 -- 172.21.15.90:0/34585725 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa840003ff0 msgr2=0x7fa840004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:07.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.785+0000 7fa860200640 1 --2- 172.21.15.90:0/34585725 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa840003ff0 0x7fa840004800 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fa844003d80 tx=0x7fa840005830 comp rx=0 tx=0).stop 2024-04-25T01:19:07.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.785+0000 7fa85e200640 1 -- 172.21.15.90:0/34585725 reap_dead start 2024-04-25T01:19:07.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.785+0000 7fa860200640 1 -- 172.21.15.90:0/34585725 shutdown_connections 2024-04-25T01:19:07.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.785+0000 7fa860200640 1 -- 172.21.15.90:0/34585725 >> 172.21.15.90:0/34585725 conn(0x7fa85806b320 msgr2=0x7fa85806b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:07.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.785+0000 7fa860200640 1 -- 172.21.15.90:0/34585725 shutdown_connections 2024-04-25T01:19:07.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.786+0000 7fa860200640 1 -- 172.21.15.90:0/34585725 wait complete. 2024-04-25T01:19:07.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.787+0000 7fa860200640 1 Processor -- start 2024-04-25T01:19:07.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.787+0000 7fa860200640 1 -- start start 2024-04-25T01:19:07.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.788+0000 7fa860200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa840003ff0 0x7fa858158d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:07.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.788+0000 7fa860200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa85813c620 0x7fa85813caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:07.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.788+0000 7fa860200640 1 --2- >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa85815ed20 0x7fa85815f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:07.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.788+0000 7fa860200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa85806d9e0 con 0x7fa840003ff0 2024-04-25T01:19:07.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.788+0000 7fa860200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa85806d700 con 0x7fa85813c620 2024-04-25T01:19:07.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.788+0000 7fa860200640 1 -- --> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] -- mon_getmap magic: 0 -- 0x7fa85806cbc0 con 0x7fa85815ed20 2024-04-25T01:19:07.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.788+0000 7fa85d800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa840003ff0 0x7fa858158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:07.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.789+0000 7fa85d800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa840003ff0 0x7fa858158d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49146/0 (socket says 172.21.15.90:49146) 2024-04-25T01:19:07.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.789+0000 7fa85d800640 1 -- 172.21.15.90:0/1175843405 learned_addr learned my addr 172.21.15.90:0/1175843405 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T01:19:07.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.789+0000 7fa85d800640 1 -- 172.21.15.90:0/1175843405 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa85815ed20 msgr2=0x7fa85815f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:07.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.789+0000 7fa85d800640 1 --2- 172.21.15.90:0/1175843405 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa85815ed20 0x7fa85815f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:07.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.789+0000 7fa85d800640 1 -- 172.21.15.90:0/1175843405 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa85813c620 msgr2=0x7fa85813caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:07.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.789+0000 7fa85d800640 1 --2- 172.21.15.90:0/1175843405 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa85813c620 0x7fa85813caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:07.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.789+0000 7fa85d800640 1 -- 172.21.15.90:0/1175843405 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa85813e2a0 con 0x7fa840003ff0 2024-04-25T01:19:07.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.789+0000 7fa85d800640 1 --2- 172.21.15.90:0/1175843405 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa840003ff0 0x7fa858158d50 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fa84c00e7e0 tx=0x7fa84c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:07.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.789+0000 7fa855600640 1 -- 172.21.15.90:0/1175843405 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa84c017070 con 0x7fa840003ff0 2024-04-25T01:19:07.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.789+0000 7fa855600640 1 -- 172.21.15.90:0/1175843405 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fa84c00c6d0 con 0x7fa840003ff0 2024-04-25T01:19:07.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.789+0000 7fa855600640 1 -- 172.21.15.90:0/1175843405 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa84c00f040 con 0x7fa840003ff0 2024-04-25T01:19:07.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.789+0000 7fa860200640 1 -- 172.21.15.90:0/1175843405 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8581447f0 con 0x7fa840003ff0 2024-04-25T01:19:07.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.790+0000 7fa860200640 1 -- 172.21.15.90:0/1175843405 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa85813a120 con 0x7fa840003ff0 2024-04-25T01:19:07.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.791+0000 7fa860200640 1 -- 172.21.15.90:0/1175843405 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa858068190 con 0x7fa840003ff0 2024-04-25T01:19:07.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.792+0000 7fa855600640 1 -- 172.21.15.90:0/1175843405 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa84c01a070 con 0x7fa840003ff0 2024-04-25T01:19:07.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.792+0000 7fa855600640 1 --2- 172.21.15.90:0/1175843405 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa83806be30 0x7fa83806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:19:07.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.793+0000 7fa855600640 1 -- 172.21.15.90:0/1175843405 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5421+0+0 (secure 0 0 0) 0x7fa84c08aa40 con 0x7fa840003ff0 2024-04-25T01:19:07.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.793+0000 7fa85ce00640 1 --2- 172.21.15.90:0/1175843405 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa83806be30 0x7fa83806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:19:07.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.794+0000 7fa85ce00640 1 --2- 172.21.15.90:0/1175843405 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa83806be30 0x7fa83806e2f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fa844003d80 tx=0x7fa84402c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:19:07.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:07.797+0000 7fa855600640 1 -- 172.21.15.90:0/1175843405 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa85813e2a0 con 0x7fa840003ff0 2024-04-25T01:19:08.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:08.036+0000 7fa860200640 1 -- 172.21.15.90:0/1175843405 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8580af130 con 0x7fa840003ff0 2024-04-25T01:19:08.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:08.037+0000 7fa855600640 1 -- 172.21.15.90:0/1175843405 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+3875 (secure 0 0 0) 0x7fa84c08ae20 con 0x7fa840003ff0 2024-04-25T01:19:08.020 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T01:19:08.020 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":32,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6834/1837027468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1837027468},{"type":"v1","addr":"172.21.15.103:6834","nonce":1837027468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":4966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1873497946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6833","nonce":1873497946},{"type":"v1","addr":"172.21.15.103:6835","nonce":1873497946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5595,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/3206865830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":3206865830},{"type":"v1","addr":"172.21.15.90:6835","nonce":3206865830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5598,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2208594231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2208594231},{"type":"v1","addr":"172.21.15.90:6837","nonce":2208594231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[]} 2024-04-25T01:19:08.020 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 32 2024-04-25T01:19:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:08.040+0000 7fa860200640 1 -- 172.21.15.90:0/1175843405 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa83806be30 msgr2=0x7fa83806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:08.040+0000 7fa860200640 1 --2- 172.21.15.90:0/1175843405 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa83806be30 0x7fa83806e2f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fa844003d80 tx=0x7fa84402c000 comp rx=0 tx=0).stop 2024-04-25T01:19:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:08.040+0000 7fa860200640 1 -- 172.21.15.90:0/1175843405 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa840003ff0 msgr2=0x7fa858158d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:19:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:08.040+0000 7fa860200640 1 --2- 172.21.15.90:0/1175843405 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa840003ff0 0x7fa858158d50 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fa84c00e7e0 tx=0x7fa84c00eca0 comp rx=0 tx=0).stop 2024-04-25T01:19:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:08.040+0000 7fa860200640 1 -- 172.21.15.90:0/1175843405 shutdown_connections 2024-04-25T01:19:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:08.040+0000 7fa860200640 1 --2- 172.21.15.90:0/1175843405 >> [v2:172.21.15.90:6832/2198086733,v1:172.21.15.90:6833/2198086733] conn(0x7fa83806be30 0x7fa83806e2f0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:08.040+0000 7fa860200640 1 --2- 172.21.15.90:0/1175843405 >> [v2:172.21.15.103:3301/0,v1:172.21.15.103:6790/0] conn(0x7fa85815ed20 0x7fa85815f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:08.040+0000 7fa860200640 1 --2- 172.21.15.90:0/1175843405 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa85813c620 0x7fa85813caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:08.040+0000 7fa860200640 1 --2- 172.21.15.90:0/1175843405 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa840003ff0 0x7fa858158d50 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:19:08.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:08.040+0000 7fa860200640 1 -- 172.21.15.90:0/1175843405 >> 172.21.15.90:0/1175843405 conn(0x7fa85806b320 msgr2=0x7fa85806b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:19:08.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:08.040+0000 7fa860200640 1 -- 172.21.15.90:0/1175843405 shutdown_connections 2024-04-25T01:19:08.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T01:19:08.040+0000 7fa860200640 1 -- 172.21.15.90:0/1175843405 wait complete. 2024-04-25T01:19:08.041 DEBUG:tasks.cephfs.filesystem:already dead... 2024-04-25T01:19:08.042 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-25T01:19:08.042 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T01:19:08.042 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:19:08.058 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T01:19:08.058 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T01:19:08.058 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:19:08.117 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T01:19:08.118 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T01:19:08.118 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:19:08.142 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T01:19:08.142 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T01:19:08.142 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:19:08.205 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T01:19:08.205 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-25T01:19:08.205 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-25T01:19:08.205 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:19:08.267 INFO:tasks.ceph.osd.0:Stopped 2024-04-25T01:19:08.267 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-25T01:19:08.267 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:19:08.345 INFO:tasks.ceph.osd.1:Stopped 2024-04-25T01:19:08.345 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-25T01:19:08.345 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:19:08.407 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-04-25T01:19:08.425 INFO:tasks.ceph.osd.2:Stopped 2024-04-25T01:19:08.426 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-25T01:19:08.426 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:19:08.496 INFO:tasks.ceph.osd.3:Stopped 2024-04-25T01:19:08.496 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-25T01:19:08.496 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:19:08.577 INFO:tasks.ceph.osd.4:Stopped 2024-04-25T01:19:08.577 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-25T01:19:08.577 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:19:08.628 INFO:tasks.ceph.osd.5:Stopped 2024-04-25T01:19:08.628 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-25T01:19:08.628 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:19:08.703 INFO:tasks.ceph.osd.6:Stopped 2024-04-25T01:19:08.703 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-25T01:19:08.703 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:19:08.774 INFO:tasks.ceph.osd.7:Stopped 2024-04-25T01:19:08.774 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-25T01:19:08.774 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-25T01:19:08.774 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:19:08.862 INFO:tasks.ceph.mgr.y:Stopped 2024-04-25T01:19:08.863 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-25T01:19:08.863 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:19:08.939 INFO:tasks.ceph.mgr.x:Stopped 2024-04-25T01:19:08.939 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-25T01:19:08.940 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-25T01:19:08.940 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:19:08.990 INFO:tasks.ceph.mon.a:Stopped 2024-04-25T01:19:08.991 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-25T01:19:08.991 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:19:09.023 INFO:tasks.ceph.mon.b:Stopped 2024-04-25T01:19:09.023 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-25T01:19:09.024 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:19:09.108 INFO:tasks.ceph.mon.c:Stopped 2024-04-25T01:19:09.109 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-25T01:19:09.109 DEBUG:teuthology.orchestra.run.smithi090:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'bad backtrace' | egrep -v 'object missing on disk' | egrep -v 'error reading table object' | egrep -v 'error reading sessionmap' | egrep -v 'unmatched fragstat' | egrep -v 'unmatched rstat' | egrep -v 'was unreadable, recreating it now' | egrep -v 'Scrub error on inode' | egrep -v 'Metadata damage detected' | egrep -v MDS_FAILED | egrep -v MDS_DAMAGE | head -n 1 2024-04-25T01:19:09.221 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi090.front.sepia.ceph.com 2024-04-25T01:19:09.221 DEBUG:teuthology.orchestra.run.smithi090:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-25T01:19:09.634 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi090.front.sepia.ceph.com 2024-04-25T01:19:09.634 DEBUG:teuthology.orchestra.run.smithi090:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-25T01:19:09.770 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi090.front.sepia.ceph.com 2024-04-25T01:19:09.770 DEBUG:teuthology.orchestra.run.smithi090:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-25T01:19:09.901 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi090.front.sepia.ceph.com 2024-04-25T01:19:09.902 DEBUG:teuthology.orchestra.run.smithi090:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-25T01:19:10.066 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-25T01:19:10.066 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-25T01:19:10.290 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-25T01:19:10.290 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-25T01:19:10.421 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-25T01:19:10.421 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-25T01:19:10.552 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-25T01:19:10.553 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-25T01:19:10.696 INFO:tasks.ceph:Archiving mon data... 2024-04-25T01:19:10.698 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672309/data/mon.a.tgz 2024-04-25T01:19:10.698 DEBUG:teuthology.orchestra.run.smithi090:> mktemp 2024-04-25T01:19:10.757 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/tmp.3BFlruHMcC 2024-04-25T01:19:10.758 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.3BFlruHMcC 2024-04-25T01:19:10.932 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0666 /tmp/tmp.3BFlruHMcC 2024-04-25T01:19:11.075 DEBUG:teuthology.orchestra.remote:smithi090:/tmp/tmp.3BFlruHMcC is 343KB 2024-04-25T01:19:11.183 DEBUG:teuthology.orchestra.run.smithi090:> rm -fr /tmp/tmp.3BFlruHMcC 2024-04-25T01:19:11.239 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672309/data/mon.b.tgz 2024-04-25T01:19:11.240 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2024-04-25T01:19:11.299 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.B9gwy4LKtL 2024-04-25T01:19:11.299 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.B9gwy4LKtL 2024-04-25T01:19:11.530 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.B9gwy4LKtL 2024-04-25T01:19:11.667 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.B9gwy4LKtL is 456KB 2024-04-25T01:19:11.774 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.B9gwy4LKtL 2024-04-25T01:19:11.831 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672309/data/mon.c.tgz 2024-04-25T01:19:11.832 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2024-04-25T01:19:11.881 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.Bpufl19cPf 2024-04-25T01:19:11.882 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.Bpufl19cPf 2024-04-25T01:19:12.059 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.Bpufl19cPf 2024-04-25T01:19:12.186 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.Bpufl19cPf is 455KB 2024-04-25T01:19:12.292 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.Bpufl19cPf 2024-04-25T01:19:12.349 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-25T01:19:12.349 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-25T01:19:12.357 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-25T01:19:12.496 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-25T01:19:12.553 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-25T01:19:12.553 INFO:tasks.ceph:Archiving crash dumps... 2024-04-25T01:19:12.555 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672309/remote/smithi090/crash 2024-04-25T01:19:12.556 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-25T01:19:12.642 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672309/remote/smithi103/crash 2024-04-25T01:19:12.643 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-25T01:19:12.726 INFO:tasks.ceph:Shutting down logrotate 2024-04-25T01:19:12.726 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:19:12.734 DEBUG:teuthology.orchestra.run.smithi103:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T01:19:12.847 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-25T01:19:12.933 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-25T01:19:13.020 INFO:tasks.ceph:Compressing logs... 2024-04-25T01:19:13.021 DEBUG:teuthology.orchestra.run.smithi090:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T01:19:13.029 DEBUG:teuthology.orchestra.run.smithi103:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T01:19:13.105 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47690.log 2024-04-25T01:19:13.106 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41101.log 2024-04-25T01:19:13.107 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.47690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47957.log 2024-04-25T01:19:13.107 INFO:teuthology.orchestra.run.smithi090.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47690.log.gz 2024-04-25T01:19:13.108 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53329.log 2024-04-25T01:19:13.109 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.38426.log 2024-04-25T01:19:13.109 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41101.log.gz 2024-04-25T01:19:13.110 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.47957.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47957.log.gz 2024-04-25T01:19:13.110 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51400.log 2024-04-25T01:19:13.111 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.53329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48041.log 2024-04-25T01:19:13.111 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53329.log.gz 2024-04-25T01:19:13.112 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.38844.log 2024-04-25T01:19:13.112 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.1.38426.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.38426.log.gz 2024-04-25T01:19:13.112 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.38445.log 2024-04-25T01:19:13.112 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.3.38844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.38844.log.gz 2024-04-25T01:19:13.112 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33218.log 2024-04-25T01:19:13.114 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.1.38445.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.38445.log.gz 2024-04-25T01:19:13.114 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-25T01:19:13.115 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52257.log 2024-04-25T01:19:13.116 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51400.log.gz 2024-04-25T01:19:13.116 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49677.log 2024-04-25T01:19:13.116 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48041.log.gz 2024-04-25T01:19:13.116 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52257.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52257.log.gz 2024-04-25T01:19:13.116 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48697.log 2024-04-25T01:19:13.116 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44799.log 2024-04-25T01:19:13.116 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49677.log.gz 2024-04-25T01:19:13.116 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46542.log 2024-04-25T01:19:13.120 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48697.log.gz 2024-04-25T01:19:13.120 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46416.log 2024-04-25T01:19:13.120 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44799.log.gz 2024-04-25T01:19:13.120 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50355.log 2024-04-25T01:19:13.120 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46542.log.gz 2024-04-25T01:19:13.121 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48209.log 2024-04-25T01:19:13.121 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46416.log.gz 2024-04-25T01:19:13.121 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45051.log 2024-04-25T01:19:13.121 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50355.log.gz 2024-04-25T01:19:13.122 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.33218.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.33218.log.gz 2024-04-25T01:19:13.122 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33378.log 2024-04-25T01:19:13.122 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-25T01:19:13.122 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.33378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33338.log 2024-04-25T01:19:13.122 INFO:teuthology.orchestra.run.smithi103.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33378.log.gz 2024-04-25T01:19:13.122 INFO:teuthology.orchestra.run.smithi103.stderr: 91.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-25T01:19:13.123 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-25T01:19:13.123 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.33338.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 87.9% 2024-04-25T01:19:13.123 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33338.log.gz 2024-04-25T01:19:13.123 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-osd.7.log: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-25T01:19:13.123 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-25T01:19:13.123 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-25T01:19:13.123 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-osd.5.log: 91.4% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-04-25T01:19:13.123 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.38840.log 2024-04-25T01:19:13.124 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33418.log 2024-04-25T01:19:13.127 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49635.log 2024-04-25T01:19:13.127 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45051.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.48209.log.gz 2024-04-25T01:19:13.127 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44505.log 2024-04-25T01:19:13.127 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45051.log.gz 2024-04-25T01:19:13.127 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48275.log 2024-04-25T01:19:13.127 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49635.log.gz 2024-04-25T01:19:13.127 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.43390.log 2024-04-25T01:19:13.127 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44505.log.gz 2024-04-25T01:19:13.128 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48083.log 2024-04-25T01:19:13.128 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48275.log.gz 2024-04-25T01:19:13.128 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.0.43390.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.43390.log.gz 2024-04-25T01:19:13.128 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44253.log 2024-04-25T01:19:13.128 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43983.log 2024-04-25T01:19:13.128 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48083.log.gz 2024-04-25T01:19:13.131 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.3.38840.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.38840.log.gz 2024-04-25T01:19:13.131 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-25T01:19:13.131 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.33418.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-25T01:19:13.131 INFO:teuthology.orchestra.run.smithi103.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33418.log.gz 2024-04-25T01:19:13.131 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-25T01:19:13.131 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33127.log 2024-04-25T01:19:13.132 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44841.log 2024-04-25T01:19:13.132 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44253.log.gz 2024-04-25T01:19:13.132 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47352.log 2024-04-25T01:19:13.133 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43983.log.gz 2024-04-25T01:19:13.133 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47734.log 2024-04-25T01:19:13.133 INFO:teuthology.orchestra.run.smithi090.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44841.log.gz 2024-04-25T01:19:13.133 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.47352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53161.log 2024-04-25T01:19:13.133 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47352.log.gz 2024-04-25T01:19:13.133 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.47734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52783.log 2024-04-25T01:19:13.133 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47734.log.gz 2024-04-25T01:19:13.133 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.53161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41017.log 2024-04-25T01:19:13.135 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33171.log 2024-04-25T01:19:13.135 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.33127.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33127.log.gz 2024-04-25T01:19:13.135 INFO:teuthology.orchestra.run.smithi103.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-25T01:19:13.135 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.33171.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33171.log.gz 2024-04-25T01:19:13.135 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33258.log 2024-04-25T01:19:13.136 INFO:teuthology.orchestra.run.smithi090.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53161.log.gz 2024-04-25T01:19:13.136 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45744.log 2024-04-25T01:19:13.136 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52783.log.gz 2024-04-25T01:19:13.136 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42865.log 2024-04-25T01:19:13.136 INFO:teuthology.orchestra.run.smithi090.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41017.log.gz 2024-04-25T01:19:13.136 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48782.log 2024-04-25T01:19:13.136 INFO:teuthology.orchestra.run.smithi090.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45744.log.gz 2024-04-25T01:19:13.137 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-04-25T01:19:13.138 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46836.log 2024-04-25T01:19:13.138 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42865.log.gz 2024-04-25T01:19:13.138 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47022.log 2024-04-25T01:19:13.138 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48782.log.gz 2024-04-25T01:19:13.140 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.33258.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-25T01:19:13.140 INFO:teuthology.orchestra.run.smithi103.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33258.log.gz 2024-04-25T01:19:13.140 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33298.log 2024-04-25T01:19:13.141 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52951.log 2024-04-25T01:19:13.141 INFO:teuthology.orchestra.run.smithi090.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46836.log.gz 2024-04-25T01:19:13.141 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.47022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50397.log 2024-04-25T01:19:13.141 INFO:teuthology.orchestra.run.smithi090.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47022.log.gz 2024-04-25T01:19:13.141 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53455.log 2024-04-25T01:19:13.141 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52951.log.gz 2024-04-25T01:19:13.142 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53497.log 2024-04-25T01:19:13.142 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50397.log.gz 2024-04-25T01:19:13.143 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.33298.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33298.log.gz 2024-04-25T01:19:13.143 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.53455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48740.log 2024-04-25T01:19:13.143 INFO:teuthology.orchestra.run.smithi090.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53455.log.gz 2024-04-25T01:19:13.144 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.53497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50120.log 2024-04-25T01:19:13.144 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53497.log.gz 2024-04-25T01:19:13.145 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41655.log 2024-04-25T01:19:13.145 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48740.log.gz 2024-04-25T01:19:13.145 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41316.log 2024-04-25T01:19:13.147 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41655.log: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.50120.log.gz 2024-04-25T01:19:13.147 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41655.log.gz 2024-04-25T01:19:13.147 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46794.log 2024-04-25T01:19:13.148 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53203.log 2024-04-25T01:19:13.148 INFO:teuthology.orchestra.run.smithi090.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41316.log.gz 2024-04-25T01:19:13.149 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41661.log 2024-04-25T01:19:13.149 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46794.log.gz 2024-04-25T01:19:13.150 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.53203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33547.log 2024-04-25T01:19:13.150 INFO:teuthology.orchestra.run.smithi090.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53203.log.gz 2024-04-25T01:19:13.150 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41661.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-25T01:19:13.151 INFO:teuthology.orchestra.run.smithi090.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41661.log.gz 2024-04-25T01:19:13.152 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44379.log 2024-04-25T01:19:13.152 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33547.log.gz 2024-04-25T01:19:13.153 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52155.log 2024-04-25T01:19:13.153 INFO:teuthology.orchestra.run.smithi090.stderr: 85.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-25T01:19:13.154 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53371.log 2024-04-25T01:19:13.154 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44379.log.gz 2024-04-25T01:19:13.155 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44169.log 2024-04-25T01:19:13.155 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52155.log.gz 2024-04-25T01:19:13.156 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.53371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39010.log 2024-04-25T01:19:13.156 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53371.log.gz 2024-04-25T01:19:13.157 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51358.log 2024-04-25T01:19:13.157 INFO:teuthology.orchestra.run.smithi090.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44169.log.gz 2024-04-25T01:19:13.158 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39010.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-25T01:19:13.158 INFO:teuthology.orchestra.run.smithi090.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39010.log.gz 2024-04-25T01:19:13.159 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47064.log 2024-04-25T01:19:13.159 INFO:teuthology.orchestra.run.smithi090.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51358.log.gz 2024-04-25T01:19:13.160 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46668.log 2024-04-25T01:19:13.161 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.47064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53035.log 2024-04-25T01:19:13.161 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47064.log.gz 2024-04-25T01:19:13.163 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42907.log 2024-04-25T01:19:13.163 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46668.log.gz 2024-04-25T01:19:13.163 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.53035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51096.log 2024-04-25T01:19:13.164 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53035.log.gz 2024-04-25T01:19:13.164 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52593.log 2024-04-25T01:19:13.165 INFO:teuthology.orchestra.run.smithi090.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.42907.log.gz 2024-04-25T01:19:13.166 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39052.log 2024-04-25T01:19:13.166 INFO:teuthology.orchestra.run.smithi090.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51096.log.gz 2024-04-25T01:19:13.167 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38640.log 2024-04-25T01:19:13.167 INFO:teuthology.orchestra.run.smithi090.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52593.log.gz 2024-04-25T01:19:13.168 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44757.log 2024-04-25T01:19:13.168 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39052.log.gz 2024-04-25T01:19:13.169 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50078.log 2024-04-25T01:19:13.169 INFO:teuthology.orchestra.run.smithi090.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38640.log.gz 2024-04-25T01:19:13.170 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40459.log 2024-04-25T01:19:13.170 INFO:teuthology.orchestra.run.smithi090.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44757.log.gz 2024-04-25T01:19:13.171 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46500.log 2024-04-25T01:19:13.172 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40459.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40459.log.gz 2024-04-25T01:19:13.172 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-25T01:19:13.174 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46500.log: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.50078.log.gz 2024-04-25T01:19:13.174 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52299.log 2024-04-25T01:19:13.174 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46500.log.gz 2024-04-25T01:19:13.175 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41322.log 2024-04-25T01:19:13.176 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52299.log: 88.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-25T01:19:13.176 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52299.log.gz 2024-04-25T01:19:13.176 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51012.log 2024-04-25T01:19:13.177 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49845.log 2024-04-25T01:19:13.177 INFO:teuthology.orchestra.run.smithi090.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41322.log.gz 2024-04-25T01:19:13.178 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51012.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51012.log.gz 2024-04-25T01:19:13.178 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47478.log 2024-04-25T01:19:13.179 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39138.log 2024-04-25T01:19:13.179 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49845.log.gz 2024-04-25T01:19:13.180 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.47478.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47478.log.gzgzip 2024-04-25T01:19:13.180 INFO:teuthology.orchestra.run.smithi090.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47225.log 2024-04-25T01:19:13.181 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39138.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39138.log.gz 2024-04-25T01:19:13.181 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41910.log 2024-04-25T01:19:13.182 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.47225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51778.log 2024-04-25T01:19:13.183 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47225.log.gz 2024-04-25T01:19:13.184 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47436.log 2024-04-25T01:19:13.184 INFO:teuthology.orchestra.run.smithi090.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41910.log.gz 2024-04-25T01:19:13.185 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47562.log 2024-04-25T01:19:13.185 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51778.log.gz 2024-04-25T01:19:13.186 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.47436.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47436.log.gz 2024-04-25T01:19:13.186 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53705.log 2024-04-25T01:19:13.187 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.47562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48463.log 2024-04-25T01:19:13.187 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47562.log.gz 2024-04-25T01:19:13.188 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.53705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52215.log 2024-04-25T01:19:13.188 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53705.log.gz 2024-04-25T01:19:13.188 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.0.48463.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.48463.log.gz 2024-04-25T01:19:13.189 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49449.log 2024-04-25T01:19:13.190 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49321.log 2024-04-25T01:19:13.190 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52215.log.gz 2024-04-25T01:19:13.191 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46878.log 2024-04-25T01:19:13.191 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49449.log.gz 2024-04-25T01:19:13.192 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49321.log: gzip -5 --verbose -- 88.0% /var/log/ceph/ceph-client.admin.46086.log -- replaced with /var/log/ceph/ceph-client.admin.49321.log.gz 2024-04-25T01:19:13.192 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-25T01:19:13.193 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46938.log 2024-04-25T01:19:13.193 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46878.log.gz 2024-04-25T01:19:13.194 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46086.log: gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46086.log.gz -5 2024-04-25T01:19:13.194 INFO:teuthology.orchestra.run.smithi090.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41325.log 2024-04-25T01:19:13.195 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46938.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-25T01:19:13.195 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46938.log.gz 2024-04-25T01:19:13.196 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52909.log 2024-04-25T01:19:13.196 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41325.log.gz 2024-04-25T01:19:13.197 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39264.log 2024-04-25T01:19:13.198 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41653.log 2024-04-25T01:19:13.198 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52909.log.gz 2024-04-25T01:19:13.199 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53287.log 2024-04-25T01:19:13.199 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39264.log.gz 2024-04-25T01:19:13.200 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41312.log 2024-04-25T01:19:13.201 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41653.log.gz 2024-04-25T01:19:13.202 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.53287.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53287.log.gz 2024-04-25T01:19:13.202 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.50847.log 2024-04-25T01:19:13.203 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50547.log 2024-04-25T01:19:13.203 INFO:teuthology.orchestra.run.smithi090.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41312.log.gz 2024-04-25T01:19:13.204 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.0.50847.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.50847.log.gz 2024-04-25T01:19:13.204 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40627.log 2024-04-25T01:19:13.205 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45414.log 2024-04-25T01:19:13.205 INFO:teuthology.orchestra.run.smithi090.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.50547.log.gz 2024-04-25T01:19:13.206 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40627.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40627.log.gz 2024-04-25T01:19:13.206 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51945.log 2024-04-25T01:19:13.207 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44631.log 2024-04-25T01:19:13.207 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45414.log.gz 2024-04-25T01:19:13.208 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45498.log 2024-04-25T01:19:13.208 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51945.log.gz 2024-04-25T01:19:13.209 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49407.log 2024-04-25T01:19:13.209 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44631.log.gz 2024-04-25T01:19:13.210 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52113.log 2024-04-25T01:19:13.210 INFO:teuthology.orchestra.run.smithi090.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.45498.log.gz 2024-04-25T01:19:13.211 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47310.log 2024-04-25T01:19:13.211 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49407.log.gz 2024-04-25T01:19:13.212 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42613.log 2024-04-25T01:19:13.212 INFO:teuthology.orchestra.run.smithi090.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52113.log.gz 2024-04-25T01:19:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.47310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49868.log 2024-04-25T01:19:13.213 INFO:teuthology.orchestra.run.smithi090.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47310.log.gz 2024-04-25T01:19:13.214 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41143.log 2024-04-25T01:19:13.215 INFO:teuthology.orchestra.run.smithi090.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42613.log.gz 2024-04-25T01:19:13.215 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49868.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-04-25T01:19:13.216 INFO:teuthology.orchestra.run.smithi090.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.49868.log.gz 2024-04-25T01:19:13.216 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41143.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41143.log.gz 2024-04-25T01:19:13.217 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39222.log 2024-04-25T01:19:13.217 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51862.log 2024-04-25T01:19:13.219 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39222.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39222.log.gz 2024-04-25T01:19:13.219 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44547.log 2024-04-25T01:19:13.220 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51862.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51862.log.gz 2024-04-25T01:19:13.221 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49193.log 2024-04-25T01:19:13.221 INFO:teuthology.orchestra.run.smithi103.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-04-25T01:19:13.221 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39094.log 2024-04-25T01:19:13.222 INFO:teuthology.orchestra.run.smithi090.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44547.log.gz/var/log/ceph/ceph-client.admin.49193.log: 2024-04-25T01:19:13.223 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41308.log 2024-04-25T01:19:13.223 INFO:teuthology.orchestra.run.smithi090.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49193.log.gz 2024-04-25T01:19:13.224 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39094.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39094.log.gz 2024-04-25T01:19:13.224 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48653.log 2024-04-25T01:19:13.225 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46458.log 2024-04-25T01:19:13.226 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41308.log.gz 2024-04-25T01:19:13.226 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53581.log 2024-04-25T01:19:13.227 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48653.log.gz 2024-04-25T01:19:13.228 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39390.log 2024-04-25T01:19:13.228 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46458.log.gz 2024-04-25T01:19:13.229 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.53581.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-25T01:19:13.229 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53581.log.gz 2024-04-25T01:19:13.230 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41912.log 2024-04-25T01:19:13.230 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39390.log.gz 2024-04-25T01:19:13.231 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-25T01:19:13.232 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52383.log 2024-04-25T01:19:13.233 INFO:teuthology.orchestra.run.smithi090.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41912.log.gz 2024-04-25T01:19:13.234 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46980.log 2024-04-25T01:19:13.236 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51138.log 2024-04-25T01:19:13.236 INFO:teuthology.orchestra.run.smithi090.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52383.log.gz 2024-04-25T01:19:13.237 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46980.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46980.log.gz 2024-04-25T01:19:13.237 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48167.log 2024-04-25T01:19:13.238 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51138.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51138.log.gz 2024-04-25T01:19:13.239 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53077.log 2024-04-25T01:19:13.240 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42823.log 2024-04-25T01:19:13.240 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48167.log.gz 2024-04-25T01:19:13.241 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.53077.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53077.log.gz 2024-04-25T01:19:13.241 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53413.log 2024-04-25T01:19:13.242 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42823.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42823.log.gz 2024-04-25T01:19:13.243 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-25T01:19:13.244 INFO:teuthology.orchestra.run.smithi103.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-25T01:19:13.244 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.53413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52425.log 2024-04-25T01:19:13.244 INFO:teuthology.orchestra.run.smithi090.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53413.log.gz 2024-04-25T01:19:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47915.log 2024-04-25T01:19:13.248 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45372.log 2024-04-25T01:19:13.248 INFO:teuthology.orchestra.run.smithi090.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52425.log.gz 2024-04-25T01:19:13.250 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.47915.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47915.log.gz 2024-04-25T01:19:13.250 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41994.log 2024-04-25T01:19:13.251 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45372.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45372.log.gz 2024-04-25T01:19:13.253 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41994.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41994.log.gz 2024-04-25T01:19:13.253 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41059.log 2024-04-25T01:19:13.255 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42361.log 2024-04-25T01:19:13.256 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41059.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41059.log.gz 2024-04-25T01:19:13.257 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42361.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42361.log.gz 2024-04-25T01:19:13.258 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50036.log 2024-04-25T01:19:13.259 INFO:teuthology.orchestra.run.smithi090.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-04-25T01:19:13.259 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46332.log 2024-04-25T01:19:13.261 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47520.log 2024-04-25T01:19:13.262 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46332.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.50036.log.gz 2024-04-25T01:19:13.262 INFO:teuthology.orchestra.run.smithi090.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46332.log.gz 2024-04-25T01:19:13.263 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.47520.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47520.log.gz 2024-04-25T01:19:13.264 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46206.log 2024-04-25T01:19:13.265 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47785.log 2024-04-25T01:19:13.266 INFO:teuthology.orchestra.run.smithi103.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-25T01:19:13.267 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41324.log 2024-04-25T01:19:13.267 INFO:teuthology.orchestra.run.smithi090.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46206.log.gz 2024-04-25T01:19:13.268 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.47785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39923.log 2024-04-25T01:19:13.268 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47785.log.gz 2024-04-25T01:19:13.269 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41324.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41324.log.gz 2024-04-25T01:19:13.270 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52341.log 2024-04-25T01:19:13.271 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49994.log 2024-04-25T01:19:13.273 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52341.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52341.log.gz 2024-04-25T01:19:13.273 INFO:teuthology.orchestra.run.smithi090.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.39923.log.gz 2024-04-25T01:19:13.273 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46248.log 2024-04-25T01:19:13.274 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49994.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.49994.log.gz 2024-04-25T01:19:13.274 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53663.log 2024-04-25T01:19:13.276 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50505.log 2024-04-25T01:19:13.276 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46248.log.gz 2024-04-25T01:19:13.277 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.53663.log: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49910.log 2024-04-25T01:19:13.277 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53663.log.gz 2024-04-25T01:19:13.278 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50505.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50505.log.gz 2024-04-25T01:19:13.279 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51180.log 2024-04-25T01:19:13.280 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45288.log 2024-04-25T01:19:13.280 INFO:teuthology.orchestra.run.smithi090.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.49910.log.gz 2024-04-25T01:19:13.282 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33647.log 2024-04-25T01:19:13.282 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51180.log.gz 2024-04-25T01:19:13.283 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45288.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45288.log.gz 2024-04-25T01:19:13.284 INFO:teuthology.orchestra.run.smithi090.stderr:gzip/var/log/ceph/ceph-client.admin.33647.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.51316.log 2024-04-25T01:19:13.284 INFO:teuthology.orchestra.run.smithi090.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.33647.log.gz 2024-04-25T01:19:13.285 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49593.log 2024-04-25T01:19:13.286 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42781.log 2024-04-25T01:19:13.286 INFO:teuthology.orchestra.run.smithi090.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51316.log.gz 2024-04-25T01:19:13.287 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49593.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49593.log.gz 2024-04-25T01:19:13.288 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52467.log 2024-04-25T01:19:13.289 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42781.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42781.log.gz 2024-04-25T01:19:13.289 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39432.log 2024-04-25T01:19:13.291 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44925.log 2024-04-25T01:19:13.291 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52467.log.gz 2024-04-25T01:19:13.293 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39432.log: 91.0%/var/log/ceph/ceph-client.admin.44925.log: -- replaced with /var/log/ceph/ceph-client.admin.39432.log.gz 2024-04-25T01:19:13.294 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44925.log.gz 2024-04-25T01:19:13.294 INFO:teuthology.orchestra.run.smithi090.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49803.log 2024-04-25T01:19:13.295 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52825.log 2024-04-25T01:19:13.296 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40501.log 2024-04-25T01:19:13.296 INFO:teuthology.orchestra.run.smithi090.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-25T01:19:13.297 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49803.log.gz 2024-04-25T01:19:13.298 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52825.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-25T01:19:13.298 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52825.log.gz 2024-04-25T01:19:13.299 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42193.log 2024-04-25T01:19:13.299 INFO:teuthology.orchestra.run.smithi090.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40501.log.gz 2024-04-25T01:19:13.300 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40191.log 2024-04-25T01:19:13.301 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45009.log 2024-04-25T01:19:13.302 INFO:teuthology.orchestra.run.smithi090.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42193.log.gz 2024-04-25T01:19:13.303 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40975.log 2024-04-25T01:19:13.303 INFO:teuthology.orchestra.run.smithi090.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40191.log.gz 2024-04-25T01:19:13.304 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40891.log 2024-04-25T01:19:13.304 INFO:teuthology.orchestra.run.smithi090.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45009.log.gz 2024-04-25T01:19:13.306 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40975.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40975.log.gz 2024-04-25T01:19:13.306 INFO:teuthology.orchestra.run.smithi090.stderr:gzip/var/log/ceph/ceph-client.admin.40891.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.46164.log 2024-04-25T01:19:13.307 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40891.log.gz 2024-04-25T01:19:13.307 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51987.log 2024-04-25T01:19:13.309 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44085.log 2024-04-25T01:19:13.309 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46164.log.gz 2024-04-25T01:19:13.310 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43542.log 2024-04-25T01:19:13.310 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51987.log.gz 2024-04-25T01:19:13.312 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38966.log 2024-04-25T01:19:13.312 INFO:teuthology.orchestra.run.smithi090.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44085.log.gz 2024-04-25T01:19:13.313 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43542.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43542.log.gz 2024-04-25T01:19:13.313 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45918.log 2024-04-25T01:19:13.314 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38966.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38966.log.gz 2024-04-25T01:19:13.315 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41314.log 2024-04-25T01:19:13.316 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41986.log 2024-04-25T01:19:13.316 INFO:teuthology.orchestra.run.smithi090.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45918.log.gz 2024-04-25T01:19:13.318 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41314.log: gzip -5 --verbose -- 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41314.log.gz 2024-04-25T01:19:13.318 INFO:teuthology.orchestra.run.smithi090.stderr: /var/log/ceph/ceph-client.admin.40585.log 2024-04-25T01:19:13.318 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41986.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41986.log.gz 2024-04-25T01:19:13.319 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46002.log 2024-04-25T01:19:13.320 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42487.log 2024-04-25T01:19:13.320 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40585.log.gz 2024-04-25T01:19:13.322 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45576.log 2024-04-25T01:19:13.322 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46002.log.gz 2024-04-25T01:19:13.323 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42487.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42487.log.gz 2024-04-25T01:19:13.323 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39348.log 2024-04-25T01:19:13.324 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46044.log 2024-04-25T01:19:13.325 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45576.log.gz 2024-04-25T01:19:13.326 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39348.log: gzip -5 --verbose -- 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39348.log.gz 2024-04-25T01:19:13.326 INFO:teuthology.orchestra.run.smithi090.stderr: /var/log/ceph/ceph-client.admin.53623.log 2024-04-25T01:19:13.327 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46044.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46044.log.gz 2024-04-25T01:19:13.327 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45456.log 2024-04-25T01:19:13.329 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.53623.log: gzip 88.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.53623.log.gz --verbose 2024-04-25T01:19:13.329 INFO:teuthology.orchestra.run.smithi090.stderr: -- /var/log/ceph/ceph-client.admin.40359.log 2024-04-25T01:19:13.330 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49719.log 2024-04-25T01:19:13.330 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45456.log.gz 2024-04-25T01:19:13.333 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40359.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40359.log.gz 2024-04-25T01:19:13.333 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45960.log 2024-04-25T01:19:13.333 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49719.log.gz 2024-04-25T01:19:13.334 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49952.log 2024-04-25T01:19:13.335 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52071.log 2024-04-25T01:19:13.335 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45960.log.gz 2024-04-25T01:19:13.337 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49952.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.49952.log.gz 2024-04-25T01:19:13.338 INFO:teuthology.orchestra.run.smithi090.stderr:gzip/var/log/ceph/ceph-client.admin.52071.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43071.log 2024-04-25T01:19:13.338 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52071.log.gz 2024-04-25T01:19:13.338 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48125.log 2024-04-25T01:19:13.341 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50204.log 2024-04-25T01:19:13.341 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43071.log.gz 2024-04-25T01:19:13.342 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48125.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48125.log.gz 2024-04-25T01:19:13.342 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45330.log 2024-04-25T01:19:13.344 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40417.log 2024-04-25T01:19:13.344 INFO:teuthology.orchestra.run.smithi090.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.50204.log.gz 2024-04-25T01:19:13.345 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40317.log 2024-04-25T01:19:13.345 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45330.log.gz 2024-04-25T01:19:13.346 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40417.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40417.log.gz 2024-04-25T01:19:13.346 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51694.log 2024-04-25T01:19:13.348 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44295.log 2024-04-25T01:19:13.348 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40317.log.gz 2024-04-25T01:19:13.349 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44883.log 2024-04-25T01:19:13.349 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51694.log.gz 2024-04-25T01:19:13.350 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53119.log 2024-04-25T01:19:13.350 INFO:teuthology.orchestra.run.smithi090.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44295.log.gz 2024-04-25T01:19:13.352 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47827.log 2024-04-25T01:19:13.352 INFO:teuthology.orchestra.run.smithi090.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44883.log.gz 2024-04-25T01:19:13.353 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.53119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47394.log 2024-04-25T01:19:13.353 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53119.log.gz 2024-04-25T01:19:13.355 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.47827.log: gzip -5 --verbose 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47827.log.gz 2024-04-25T01:19:13.355 INFO:teuthology.orchestra.run.smithi090.stderr: -- /var/log/ceph/ceph-client.admin.40275.log 2024-04-25T01:19:13.355 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.47394.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47394.log.gz 2024-04-25T01:19:13.356 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47646.log 2024-04-25T01:19:13.357 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45618.log 2024-04-25T01:19:13.357 INFO:teuthology.orchestra.run.smithi090.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40275.log.gz 2024-04-25T01:19:13.359 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.47646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46584.log 2024-04-25T01:19:13.359 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47646.log.gz 2024-04-25T01:19:13.360 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52993.log 2024-04-25T01:19:13.360 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45618.log.gz 2024-04-25T01:19:13.361 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46584.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46584.log.gz 2024-04-25T01:19:13.362 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40233.log 2024-04-25T01:19:13.363 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42529.log 2024-04-25T01:19:13.363 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52993.log.gz 2024-04-25T01:19:13.364 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40233.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40233.log.gz 2024-04-25T01:19:13.366 INFO:teuthology.orchestra.run.smithi090.stderr:gzip/var/log/ceph/ceph-client.admin.42529.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41995.log 2024-04-25T01:19:13.366 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42529.log.gz 2024-04-25T01:19:13.366 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41996.log 2024-04-25T01:19:13.368 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51610.log 2024-04-25T01:19:13.369 INFO:teuthology.orchestra.run.smithi090.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41995.log.gz 2024-04-25T01:19:13.370 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41996.log: gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41996.log.gz 2024-04-25T01:19:13.370 INFO:teuthology.orchestra.run.smithi090.stderr: -5 --verbose --/var/log/ceph/ceph-client.admin.51610.log: /var/log/ceph/ceph-client.admin.38559.log 2024-04-25T01:19:13.371 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51610.log.gz 2024-04-25T01:19:13.371 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48824.log 2024-04-25T01:19:13.373 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41323.log 2024-04-25T01:19:13.373 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38559.log.gz 2024-04-25T01:19:13.374 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48824.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48824.log.gz 2024-04-25T01:19:13.375 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38700.log 2024-04-25T01:19:13.376 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41323.log: gzip -5 88.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.41323.log.gz /var/log/ceph/ceph-client.admin.50162.log 2024-04-25T01:19:13.376 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-25T01:19:13.377 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49491.log 2024-04-25T01:19:13.377 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38700.log.gz 2024-04-25T01:19:13.379 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51526.log 2024-04-25T01:19:13.379 INFO:teuthology.orchestra.run.smithi090.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.50162.log.gz 2024-04-25T01:19:13.380 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49491.log: gzip -5 --verbose -- 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49491.log.gz 2024-04-25T01:19:13.380 INFO:teuthology.orchestra.run.smithi090.stderr: /var/log/ceph/ceph-client.admin.41644.log 2024-04-25T01:19:13.381 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51526.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51526.log.gz 2024-04-25T01:19:13.382 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40933.log 2024-04-25T01:19:13.383 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44967.log 2024-04-25T01:19:13.383 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41644.log.gz 2024-04-25T01:19:13.384 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42277.log 2024-04-25T01:19:13.385 INFO:teuthology.orchestra.run.smithi090.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40933.log.gz 2024-04-25T01:19:13.386 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52867.log 2024-04-25T01:19:13.386 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44967.log.gz 2024-04-25T01:19:13.388 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42277.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42277.log.gz 2024-04-25T01:19:13.388 INFO:teuthology.orchestra.run.smithi090.stderr:gzip/var/log/ceph/ceph-client.admin.52867.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42571.log 2024-04-25T01:19:13.389 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52867.log.gz 2024-04-25T01:19:13.389 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50439.log 2024-04-25T01:19:13.391 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48902.log 2024-04-25T01:19:13.391 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42571.log.gz 2024-04-25T01:19:13.393 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50439.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.50439.log.gz 2024-04-25T01:19:13.393 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48902.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48902.log.gz 2024-04-25T01:19:13.394 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39498.log 2024-04-25T01:19:13.395 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45876.log 2024-04-25T01:19:13.397 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53539.log 2024-04-25T01:19:13.397 INFO:teuthology.orchestra.run.smithi090.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39498.log.gz 2024-04-25T01:19:13.398 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45876.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45876.log.gz 2024-04-25T01:19:13.398 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42445.log 2024-04-25T01:19:13.399 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.53539.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53539.log.gz 2024-04-25T01:19:13.400 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51222.log 2024-04-25T01:19:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38759.log 2024-04-25T01:19:13.401 INFO:teuthology.orchestra.run.smithi090.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42445.log.gz 2024-04-25T01:19:13.402 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51222.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51222.log.gz 2024-04-25T01:19:13.403 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.38759.log: -- /var/log/ceph/ceph-client.0.43409.log 2024-04-25T01:19:13.404 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38759.log.gz 2024-04-25T01:19:13.404 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46752.log 2024-04-25T01:19:13.406 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.0.43409.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42319.log 2024-04-25T01:19:13.406 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.0.43409.log.gz 2024-04-25T01:19:13.407 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-04-25T01:19:13.408 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47267.log 2024-04-25T01:19:13.408 INFO:teuthology.orchestra.run.smithi090.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46752.log.gz 2024-04-25T01:19:13.409 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47999.log 2024-04-25T01:19:13.409 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42319.log.gz 2024-04-25T01:19:13.410 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.47267.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47267.log.gz 2024-04-25T01:19:13.411 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42697.log 2024-04-25T01:19:13.411 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.47999.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47999.log.gz 2024-04-25T01:19:13.412 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45660.log 2024-04-25T01:19:13.413 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49279.log 2024-04-25T01:19:13.414 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42697.log.gz 2024-04-25T01:19:13.415 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45660.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45660.log.gz 2024-04-25T01:19:13.415 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50613.log 2024-04-25T01:19:13.417 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49279.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49279.log.gz 2024-04-25T01:19:13.417 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40149.log 2024-04-25T01:19:13.418 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50613.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.50613.log.gz 2024-04-25T01:19:13.418 INFO:teuthology.orchestra.run.smithi090.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48964.log 2024-04-25T01:19:13.419 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51904.log 2024-04-25T01:19:13.420 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40149.log.gz 2024-04-25T01:19:13.421 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48964.log: gzip -5 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48964.log.gz 2024-04-25T01:19:13.421 INFO:teuthology.orchestra.run.smithi090.stderr: --verbose -- /var/log/ceph/ceph-client.admin.52509.log 2024-04-25T01:19:13.422 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51904.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51904.log.gz 2024-04-25T01:19:13.422 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46290.log 2024-04-25T01:19:13.424 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39180.log 2024-04-25T01:19:13.424 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52509.log.gz 2024-04-25T01:19:13.426 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44673.log 2024-04-25T01:19:13.426 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46290.log.gz 2024-04-25T01:19:13.426 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41654.log 2024-04-25T01:19:13.427 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39180.log.gz 2024-04-25T01:19:13.428 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51484.log 2024-04-25T01:19:13.428 INFO:teuthology.orchestra.run.smithi090.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44673.log.gz 2024-04-25T01:19:13.430 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41654.log: gzip -5 --verbose -- 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41654.log.gz 2024-04-25T01:19:13.430 INFO:teuthology.orchestra.run.smithi090.stderr: /var/log/ceph/ceph-client.admin.45786.log 2024-04-25T01:19:13.430 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51484.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51484.log.gz 2024-04-25T01:19:13.431 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27208.log 2024-04-25T01:19:13.432 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41650.log 2024-04-25T01:19:13.432 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45786.log.gz 2024-04-25T01:19:13.434 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph.tmp-client.admin.27208.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27208.log.gz 2024-04-25T01:19:13.434 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45177.log 2024-04-25T01:19:13.435 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49533.log 2024-04-25T01:19:13.435 INFO:teuthology.orchestra.run.smithi090.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41650.log.gz 2024-04-25T01:19:13.436 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45177.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45177.log.gz 2024-04-25T01:19:13.437 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49761.log 2024-04-25T01:19:13.438 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52551.log 2024-04-25T01:19:13.438 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49533.log.gz 2024-04-25T01:19:13.439 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52029.log 2024-04-25T01:19:13.440 INFO:teuthology.orchestra.run.smithi090.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49761.log.gz 2024-04-25T01:19:13.441 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42739.log 2024-04-25T01:19:13.441 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52551.log.gz 2024-04-25T01:19:13.442 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42403.log 2024-04-25T01:19:13.442 INFO:teuthology.orchestra.run.smithi090.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52029.log.gz 2024-04-25T01:19:13.443 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42739.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42739.log.gz 2024-04-25T01:19:13.446 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49237.log 2024-04-25T01:19:13.446 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42403.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42403.log.gz 2024-04-25T01:19:13.446 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47869.log 2024-04-25T01:19:13.446 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51054.log 2024-04-25T01:19:13.447 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49237.log.gz 2024-04-25T01:19:13.448 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.47869.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47869.log.gz 2024-04-25T01:19:13.448 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33279.log 2024-04-25T01:19:13.449 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51054.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51054.log.gz 2024-04-25T01:19:13.450 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47604.log 2024-04-25T01:19:13.451 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38922.log 2024-04-25T01:19:13.451 INFO:teuthology.orchestra.run.smithi090.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.33279.log.gz 2024-04-25T01:19:13.452 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.47604.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47604.log.gz 2024-04-25T01:19:13.454 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38922.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38922.log.gz 2024-04-25T01:19:13.454 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38812.log 2024-04-25T01:19:13.455 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-04-25T01:19:13.457 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45702.log 2024-04-25T01:19:13.457 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38812.log.gz 2024-04-25T01:19:13.460 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.45702.log: 91.4% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 88.7% 2024-04-25T01:19:13.460 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45702.log.gz 2024-04-25T01:19:13.460 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46710.log 2024-04-25T01:19:13.461 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41990.log 2024-04-25T01:19:13.463 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44463.log 2024-04-25T01:19:13.463 INFO:teuthology.orchestra.run.smithi090.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46710.log.gz 2024-04-25T01:19:13.465 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41990.log: gzip -5 --verbose 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41990.log.gz 2024-04-25T01:19:13.465 INFO:teuthology.orchestra.run.smithi090.stderr: -- /var/log/ceph/ceph-client.2.43850.log 2024-04-25T01:19:13.465 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44463.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44463.log.gz 2024-04-25T01:19:13.466 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39306.log 2024-04-25T01:19:13.467 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.2.43850.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.43850.log.gz 2024-04-25T01:19:13.467 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46374.log 2024-04-25T01:19:13.469 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53245.log 2024-04-25T01:19:13.469 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39306.log.gz 2024-04-25T01:19:13.470 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46374.log: 87.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.46374.log.gz 2024-04-25T01:19:13.470 INFO:teuthology.orchestra.run.smithi090.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44421.log 2024-04-25T01:19:13.472 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.53245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50655.log 2024-04-25T01:19:13.472 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53245.log.gz 2024-04-25T01:19:13.473 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40705.log 2024-04-25T01:19:13.473 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44421.log.gz 2024-04-25T01:19:13.474 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50655.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50655.log.gz 2024-04-25T01:19:13.475 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.40705.log: -- /var/log/ceph/ceph-client.admin.51736.log 2024-04-25T01:19:13.475 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40705.log.gz 2024-04-25T01:19:13.476 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44589.log 2024-04-25T01:19:13.478 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47144.log 2024-04-25T01:19:13.478 INFO:teuthology.orchestra.run.smithi090.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51736.log.gz 2024-04-25T01:19:13.479 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44589.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44589.log.gz 2024-04-25T01:19:13.480 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44127.log 2024-04-25T01:19:13.480 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.47144.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.47144.log.gz 2024-04-25T01:19:13.481 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40849.log 2024-04-25T01:19:13.482 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44715.log 2024-04-25T01:19:13.482 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44127.log.gz 2024-04-25T01:19:13.484 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40849.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40849.log.gz 2024-04-25T01:19:13.485 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5/var/log/ceph/ceph-client.admin.44715.log: --verbose -- /var/log/ceph/ceph-client.admin.51652.log 2024-04-25T01:19:13.485 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44715.log.gz 2024-04-25T01:19:13.486 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51442.log 2024-04-25T01:19:13.487 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50271.log 2024-04-25T01:19:13.488 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51652.log.gz 2024-04-25T01:19:13.489 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51442.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51442.log.gz 2024-04-25T01:19:13.489 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50313.log 2024-04-25T01:19:13.490 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50271.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50271.log.gz 2024-04-25T01:19:13.491 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41658.log 2024-04-25T01:19:13.492 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.43846.log 88.0% 2024-04-25T01:19:13.492 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50313.log.gz 2024-04-25T01:19:13.493 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41658.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41658.log.gz 2024-04-25T01:19:13.494 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38854.log 2024-04-25T01:19:13.494 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.2.43846.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.43846.log.gz 2024-04-25T01:19:13.495 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48467.log 2024-04-25T01:19:13.496 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49363.log 2024-04-25T01:19:13.496 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38854.log.gz 2024-04-25T01:19:13.497 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.0.48467.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.48467.log.gz 2024-04-25T01:19:13.497 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49068.log 2024-04-25T01:19:13.499 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42235.log 2024-04-25T01:19:13.499 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49363.log.gz 2024-04-25T01:19:13.500 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51820.log 2024-04-25T01:19:13.501 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42235.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.49068.log.gz 2024-04-25T01:19:13.502 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42235.log.gz 2024-04-25T01:19:13.502 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45093.log 2024-04-25T01:19:13.503 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51820.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51820.log.gz 2024-04-25T01:19:13.503 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40789.log 2024-04-25T01:19:13.505 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46626.log 2024-04-25T01:19:13.505 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45093.log.gz 2024-04-25T01:19:13.507 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40789.log: gzip -5 --verbose 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40789.log.gz 2024-04-25T01:19:13.507 INFO:teuthology.orchestra.run.smithi090.stderr: -- /var/log/ceph/ceph-client.admin.38463.log 2024-04-25T01:19:13.507 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46626.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46626.log.gz 2024-04-25T01:19:13.508 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42655.log 2024-04-25T01:19:13.509 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45135.log 2024-04-25T01:19:13.509 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38463.log.gz 2024-04-25T01:19:13.511 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40543.log 2024-04-25T01:19:13.511 INFO:teuthology.orchestra.run.smithi090.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42655.log.gz 2024-04-25T01:19:13.512 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44337.log 2024-04-25T01:19:13.512 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45135.log.gz 2024-04-25T01:19:13.513 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44025.log 2024-04-25T01:19:13.513 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40543.log.gz 2024-04-25T01:19:13.515 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44337.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44337.log.gz 2024-04-25T01:19:13.516 INFO:teuthology.orchestra.run.smithi090.stderr:gzip/var/log/ceph/ceph-client.admin.44025.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.44211.log 2024-04-25T01:19:13.516 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44025.log.gz 2024-04-25T01:19:13.517 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.50843.log 2024-04-25T01:19:13.519 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41660.log 2024-04-25T01:19:13.519 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44211.log.gz 2024-04-25T01:19:13.520 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.0.50843.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.50843.log.gz 2024-04-25T01:19:13.520 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51568.log 2024-04-25T01:19:13.521 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40747.log 2024-04-25T01:19:13.521 INFO:teuthology.orchestra.run.smithi090.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41660.log.gz 2024-04-25T01:19:13.523 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41997.log 2024-04-25T01:19:13.523 INFO:teuthology.orchestra.run.smithi090.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51568.log.gz 2024-04-25T01:19:13.524 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40747.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40747.log.gz 2024-04-25T01:19:13.525 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41997.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41997.log.gz 2024-04-25T01:19:13.670 INFO:teuthology.orchestra.run.smithi090.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-25T01:19:13.838 INFO:teuthology.orchestra.run.smithi103.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-25T01:19:13.844 INFO:teuthology.orchestra.run.smithi103.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-25T01:19:13.904 INFO:teuthology.orchestra.run.smithi103.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-25T01:19:13.928 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-25T01:19:13.931 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-04-25T01:19:13.931 INFO:teuthology.orchestra.run.smithi103.stderr:real 0m0.853s 2024-04-25T01:19:13.932 INFO:teuthology.orchestra.run.smithi103.stderr:user 0m3.101s 2024-04-25T01:19:13.932 INFO:teuthology.orchestra.run.smithi103.stderr:sys 0m0.381s 2024-04-25T01:19:13.945 INFO:teuthology.orchestra.run.smithi090.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-25T01:19:14.034 INFO:teuthology.orchestra.run.smithi090.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-25T01:19:14.073 INFO:teuthology.orchestra.run.smithi090.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-25T01:19:14.106 INFO:teuthology.orchestra.run.smithi090.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-25T01:19:14.109 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-25T01:19:14.109 INFO:teuthology.orchestra.run.smithi090.stderr:real 0m1.035s 2024-04-25T01:19:14.109 INFO:teuthology.orchestra.run.smithi090.stderr:user 0m3.641s 2024-04-25T01:19:14.109 INFO:teuthology.orchestra.run.smithi090.stderr:sys 0m1.147s 2024-04-25T01:19:14.110 INFO:tasks.ceph:Archiving logs... 2024-04-25T01:19:14.110 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672309/remote/smithi090/log 2024-04-25T01:19:14.111 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T01:19:14.872 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672309/remote/smithi103/log 2024-04-25T01:19:14.873 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T01:19:15.375 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T01:19:15.385 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T01:19:15.385 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T01:19:15.393 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T01:19:15.540 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T01:19:15.551 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T01:19:15.551 DEBUG:teuthology.orchestra.run.smithi090:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:19:15.559 DEBUG:teuthology.orchestra.run.smithi103:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:19:15.607 INFO:teuthology.orchestra.run.smithi090.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:19:15.613 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:19:15.617 INFO:teuthology.orchestra.run.smithi090.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:19:15.618 INFO:teuthology.orchestra.run.smithi090.stdout:=============================================================================== 2024-04-25T01:19:15.618 INFO:teuthology.orchestra.run.smithi090.stdout:^- hv01.front.sepia.ceph.com 2 6 377 32 +1113us[+1113us] +/- 587ms 2024-04-25T01:19:15.618 INFO:teuthology.orchestra.run.smithi090.stdout:^* hv02.front.sepia.ceph.com 3 6 377 32 -149us[ -185us] +/- 62ms 2024-04-25T01:19:15.618 INFO:teuthology.orchestra.run.smithi090.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 32 +144us[ +144us] +/- 71ms 2024-04-25T01:19:15.618 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:19:15.622 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:19:15.622 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-04-25T01:19:15.622 INFO:teuthology.orchestra.run.smithi103.stdout:^- hv01.front.sepia.ceph.com 2 6 377 32 +1135us[+1135us] +/- 587ms 2024-04-25T01:19:15.622 INFO:teuthology.orchestra.run.smithi103.stdout:^* hv02.front.sepia.ceph.com 3 6 377 32 -147us[ -146us] +/- 62ms 2024-04-25T01:19:15.622 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 +187us[ +187us] +/- 71ms 2024-04-25T01:19:15.622 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:19:15.623 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T01:19:15.633 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T01:19:15.634 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T01:19:15.642 DEBUG:teuthology.orchestra.run.smithi090:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T01:19:15.842 DEBUG:teuthology.orchestra.run.smithi103:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T01:19:16.019 DEBUG:teuthology.orchestra.run.smithi090:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:19:16.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:19:16.113 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:19:16.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:19:16.204 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T01:19:16.214 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T01:19:16.226 INFO:teuthology.task.internal:Duration was 669.249743 seconds 2024-04-25T01:19:16.226 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T01:19:16.239 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T01:19:16.239 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T01:19:16.247 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T01:19:16.373 INFO:teuthology.orchestra.run.smithi090.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:19:16.391 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:19:16.702 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T01:19:16.703 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi090.front.sepia.ceph.com 2024-04-25T01:19:16.703 DEBUG:teuthology.orchestra.run.smithi090:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T01:19:16.775 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi103.front.sepia.ceph.com 2024-04-25T01:19:16.775 DEBUG:teuthology.orchestra.run.smithi103:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T01:19:16.859 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T01:19:16.859 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T01:19:16.867 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T01:19:16.951 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T01:19:16.951 DEBUG:teuthology.orchestra.run.smithi090:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T01:19:16.957 DEBUG:teuthology.orchestra.run.smithi103:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T01:19:17.130 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T01:19:17.140 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T01:19:17.140 DEBUG:teuthology.orchestra.run.smithi090:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T01:19:17.224 DEBUG:teuthology.orchestra.run.smithi103:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T01:19:17.312 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T01:19:17.322 DEBUG:teuthology.orchestra.run.smithi090:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T01:19:17.331 DEBUG:teuthology.orchestra.run.smithi103:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T01:19:17.423 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = core 2024-04-25T01:19:17.430 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = core 2024-04-25T01:19:17.471 DEBUG:teuthology.orchestra.run.smithi090:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T01:19:17.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:19:17.520 DEBUG:teuthology.orchestra.run.smithi103:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T01:19:17.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:19:17.575 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T01:19:17.585 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T01:19:17.585 DEBUG:teuthology.misc:Transferring archived files from smithi090:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672309/remote/smithi090 2024-04-25T01:19:17.586 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T01:19:17.683 DEBUG:teuthology.misc:Transferring archived files from smithi103:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672309/remote/smithi103 2024-04-25T01:19:17.683 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T01:19:17.780 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T01:19:17.780 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T01:19:17.788 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T01:19:17.843 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T01:19:17.854 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T01:19:17.855 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T01:19:17.884 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T01:19:17.884 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T01:19:17.892 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T01:19:17.940 INFO:teuthology.orchestra.run.smithi090.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 01:19 /home/ubuntu/cephtest 2024-04-25T01:19:17.951 INFO:teuthology.orchestra.run.smithi103.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 01:19 /home/ubuntu/cephtest 2024-04-25T01:19:17.954 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T01:19:17.965 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T01:19:17.987 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 806ec36b21129f113ce356c007dd59694d340b37 client.1-kernel-sha1: 806ec36b21129f113ce356c007dd59694d340b37 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/alternate-pool} duration: 669.2497425079346 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-04-25T01:19:17.988 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T01:19:18.075 INFO:teuthology.run:pass